When traffic is not optimized between two sites, the following steps can be taken to determine the source of the problem:
Check the health of the appliances.
Setup a TCP session which is expected to be optimized.
Determine if the client-side Steelhead appliance sees this TCP session.
Determine if the server-side Steelhead appliance sees this TCP session.
Determine the cause of the failure of setup of inner channel.
A Steelhead appliance will not optimize any new TCP sessions whilst the
device is in Admission Control, like a TCP Connection Limit admission
control or Memory based admission control. Use the command
show info
to confirm this and the command
show alarms triggered
to determine which alarms are triggered.
A Steelhead appliance will not optimize any new TCP sessions if the packet goes via an interface which is in by-pass mode. Use tcpdump to confirm that the traffic from the client to the server goes via the Steelhead appliance.
A Steelhead appliance in a Connection Forwarding cluster will not
optimize any new TCP sessions if the Connection Forwarding peer is
unreachable and Allow Failure in the Connection Forwarding configuration
is not enabled. Use the command
show in-path neighbour
to confirm this.
A Steelhead appliance will not optimize any new TCP sessions if the
combination of the client IP address and server IP address is in
the Asymmetric Routing table. Use the command
show in-path asym-route-tab
to and
in-path asym-route-tab flush
to deal with this.
Only once confirmed that both Steelhead appliances are in a healthy state, further troubleshooting can be started.
First step is to determine the details of the TCP traffic that doesn't get optimized. The information required is the source-subnets where the clients are located (source-address) and the IP address of the server (dest-address) and the TCP port of the service (dest-port).
In this example the IP address of the client will be 10.0.1.1, the IP address of the server will be 192.168.1.1 and the TCP port is 543.
The second step is to get access to a client. This would be easiest if there is access to a client via a remote access or remote desktop program:
Access to Unix based clients can be achieved via SSH or Telnet. Once logged in on the client, use the Telnet command to setup a TCP session to the server.
Access to Microsoft Windows based client can be achieved via the
Microsoft Remote Desktop application or screen sharing applications
programs like VNC or Goto Assist. Once access has been obtained,
use the Shell command
telnet.exe
to setup a TCP session to the server.
Note that Windows 7 does not install the telnet program by default, it can be installed via Control Panel -> Programs and Features -> Enable Windows Features.
If no access can be obtained via real clients, consider accessing the local LAN switch, if it supports remote shells, or via an SSH session towards the local Steelhead appliance, if the primary interface of the Steelhead appliance is connected to the LAN switch. There use the telnet command to setup a TCP session to the server.
The syntax of the Telnet command is as follows:
Figure 7.2. Usage of the telnet command
$ telnet 192.168.1.1 543 Trying 192.168.1.1... Connected to 192.168.1.1. Escape character is '^]'.
The full usage of this command is described in the chapter Telnet section in the The Command Line and Tools chapter.
The next step is to see if and how the client-side Steelhead appliance sees this TCP session.
This can be found via the GUI under Reports -> Networking -> Current Connections. Under connection type, select All and under Filter enter the IP address of the client and search for the line with the IP addresses of the client and the server.
On the CLI this can be done with the command
show connections all filter 10.0.1.1
.
Figure 7.4. Output of the "show connections all" command.
SH # show connections all T Source Destination App Rdn Since -------------------------------------------------------------------------------- O 10.0.1.1 37780 192.168.1.1 543 TCP 0% 2012/07/29 01:54:35 PI 192.168.1.1 42342 10.0.1.1 22 2012/07/28 14:06:56 s PU 10.0.1.1 51274 192.168.1.1 443 2012/07/29 01:43:22 c PU 10.0.1.1 51490 192.168.1.1 80 2012/07/29 01:54:08 c -------------------------------------------------------------------------------- Established Optimized (O): 1 Half-Opened Optimized (H): 0 Half-Closed Optimized (C): 0 Pass Through (P): 3 PI = Passthrough Intentional PU = Passthrough Unintentional
If the client-side Steelhead appliance does not show this TCP session, then the TCP session followed a path between the client and the server which is not covered by this Steelhead appliance.
If the TCP session shows up as 192.168.1.1:543 10.0.1.1:1025, then the first packet seen by the client-side Steelhead appliance is the TCP SYN-ACK, not the TCP SYN packet. In that case the path from the client to the server bypasses this Steelhead appliance and the path from the server to the client does go through this Steelhead appliance.
If the TCP session shows up as 10.0.1.1:1025 192.168.1.1:543, then check the details of the TCP session. This can be done in the GUI by clicking on the magnifying glass in front of the TCP session.
On the CLI this can be done with the command
show connections all filter 10.0.1.1 full
.
Figure 7.6. Output of the "show connections all full" command.
T Source Destination App Rdn Since -------------------------------------------------------------------------------- O 10.0.1.1 37780 192.168.1.1 543 TCP 0% 2012/07/29 01:54:35 Peer: 192.168.1.6 7800 Flags: cfe,sdr,lz,te=0,pe=0 Outer Local: 10.0.1.6 7801 Inner: 19421 Outer Remote: 10.0.1.6 37780 WAN/LAN: 0KB / 0KB PI 192.168.1.1 42342 10.0.1.1 22 2012/07/28 14:06:56 s PU 10.0.1.1 51274 192.168.1.1 443 2012/07/29 01:43:22 c PU 10.0.1.1 51490 192.168.1.1 80 2012/07/29 01:54:08 c
There are two kinds of pass-through TCP sessions:
Intentional pass-through sessions, caused by in-path rules.
Un-intentional pass-through session, caused by the failure to find or connect to another Steelhead appliance.
If the TCP session is intentional passed-through, then checking the in-path rules is a good first step.
The pass-through reasons contain the next clue on why the client-side Steelhead appliance couldn't setup an optimized TCP session:
If the pass-through reason is SYN on the WAN side, then the naked SYN packet for this TCP session showed up at the WAN interface instead of at the LAN interface: The LAN and WAN cables are swapped.
If the pass-through reason is No more Steelheads, then the SYN+ packet leaving the Steelhead appliance did not go through the server-side Steelhead appliance. Use the traceroute command to determine the path the IP packets take.
If the pass-through reason is Asymmetric Routing, then the server-side Steelhead appliance saw the SYN+ packet and send the SYN/ACK+, but the client-side Steelhead appliance did not see the SYN/ACK+ but the result of the client receiving the SYN/ACK+.
Check the Asymmetric Routing table to find out the reason for the pass-through of the TCP session.
If the pass-through reason is Connection is currently paused, then the optimization service is in admission control and not optimizing new TCP sessions.
There is no space in the TCP Options field for the auto-discovery probe. The maximum length of the TCP header is 64 bytes with 20 bytes used by fixed fields, leaving only 44 bytes for TCP Options. The standard TCP options like MSS size, Selective ACK, TCP Timestamps) leave enough space for the auto-discovery probe, but other extensions like auto-discovery probes from other vendors doesn't leave enough spare for the auto-discovery probe to be added.
Figure 7.7. No room for new TCP options for the auto-discovery phase
CSH [intercept.WARN] no room for option: 122.102.98.114:49809 -> 172.21.3.101:443
A full list of pass-through reasons can be found in the Steelhead Management Console and Users Guide and KB article S15377.
Next step is to confirm that the client-side Steelhead appliance is putting the auto-discovery probe on to the TCP SYN packet. For this two SSH sessions to the CLI are needed, one to trace traffic on the LAN side and one to trace traffic on the WAN side. If the TCP SYN packet comes in on a specific LAN interface, it will be sent out via the corresponding WAN interface.
In this example, the client-side Steelhead appliance does properly attach a TCP auto-discovery probe on the new TCP SYN packet.
Figure 7.8. Running tcpdump on the LAN side
CSH # tcpdump -ni lan0_0 '(host 192.168.1.1 and host 10.0.1.1 and port 543) or (vlan and ( \ host 192.168.1.1 and host 10.0.1.1 and port 543))' tcpdump: WARNING: wan0_0: no IPv4 address assigned tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on lan0_0, link-type EN10MB (Ethernet), capture size 300 bytes 09:51:50.122355 IP 10.0.1.1.57198 > 192.168.1.1.543: Flags [S], seq 3205639252, win 65535, \ options [mss 1460,nop,wscale 3,nop,nop,TS val 346474876 ecr 0,sackOK,eol], length 0 09:51:50.256198 IP 192.168.1.1.543 > 10.0.1.1.57198: Flags [S.], seq 3950051338, ack 32056 \ 39253, win 5792, options [mss 1460,sackOK,TS val 1644000 ecr 346474876,nop,wscale 2], \ length 0 09:51:50.259818 IP 10.0.1.1.57198 > 192.168.1.1.543: Flags [.], seq 1, ack 1, win 65535, o \ ptions [nop,nop,TS val 346475011 ecr 1644000], length 0
Figure 7.9. Running tcpdump on the WAN side
CSH # tcpdump -ni wan0_0 '(host 192.168.1.1 and host 10.0.1.1 and port 543) or (vlan and ( \ host 192.168.1.1 and host 10.0.1.1 and port 543))' tcpdump: WARNING: wan0_0: no IPv4 address assigned tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on wan0_0, link-type EN10MB (Ethernet), capture size 300 bytes 09:51:50.122430 IP 10.0.1.1.57198 > 192.168.1.1.543: Flags [S], seq 3205639252, win 65535, \ options [mss 1460,nop,wscale 3,nop,nop,TS val 346474876 ecr 0,sackOK,nop,nop,rvbd-pro \ be AD CSH:10.0.1.6 01010a0001060005,rvbd-probe EAD 0c01,nop,eol], length 0 09:51:50.256076 IP 192.168.1.1.543 > 10.0.1.1.57198: Flags [S.], seq 20020520, ack 3205639 \ 253, win 65535, options [mss 1460,nop,wscale 3,nop,nop,TS val 346474876 ecr 0,sackOK,n \ op,nop,rvbd-probe EAD 0c01,nop,nop,nop,eol], length 0 09:51:50.256109 IP 192.168.1.1.543 > 10.0.1.1.57198: Flags [S.], seq 20020520, ack 3205639 \ 253, win 65535, options [mss 1460,nop,wscale 3,TS val 346474876 ecr 0,sackOK,rvbd-prob \ e AD CSH:10.0.1.6 SSH:192.168.1.6:7800 11110a000106c0a801061e78,rvbd-probe EAD 0e3d,no \ p,eol], length 0
The TCP SYN packets with the rvbd-probe AD TCP option, or the SYN+, is seen here and shows that the client-side Steelhead has processed the new TCP session properly.
In this example, the client-side Steelhead appliance does not attach a TCP auto-discovery probe on the new TCP SYN packet.
Figure 7.10. Running tcpdump on the LAN side
CSH # tcpdump -ni lan0_0 '(host 192.168.1.1 and host 10.0.1.1 and port 543) or (vlan and ( \ host 192.168.1.1 and host 10.0.1.1 and port 543))' tcpdump: WARNING: wan0_0: no IPv4 address assigned tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on lan0_0, link-type EN10MB (Ethernet), capture size 300 bytes 09:55:09.968491 IP 10.0.1.1.57269 > 192.168.1.1.543: Flags [S], seq 587559488, win 65535, \ options [mss 1460,nop,wscale 3,nop,nop,TS val 346673686 ecr 0,sackOK,eol], length 0 09:55:10.102059 IP 192.168.1.1.543 > 10.0.1.1.57269: Flags [S.], seq 214939753, ack 587559 \ 489, win 65535, options [mss 1460,nop,wscale 3,sackOK,TS val 2495570782 ecr 346673686] \ , length 0 09:55:10.105859 IP 10.0.1.1.57269 > 192.168.1.1.543: Flags [.], seq 1, ack 1, win 65535, o \ ptions [nop,nop,TS val 346673823 ecr 2495570782], length 0
Figure 7.11. Running tcpdump on the WAN side
CSH # tcpdump -ni wan0_0 '(host 192.168.1.1 and host 10.0.1.1 and port 543) or (vlan and ( \ host 192.168.1.1 and host 10.0.1.1 and port 543))' tcpdump: WARNING: wan0_0: no IPv4 address assigned tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on wan0_0, link-type EN10MB (Ethernet), capture size 300 bytes 09:55:09.968576 IP 10.0.1.1.57269 > 192.168.1.1.543: Flags [S], seq 587559488, win 65535, \ options [mss 1460,nop,wscale 3,nop,nop,TS val 346673686 ecr 0,sackOK,eol], length 0 09:55:10.101988 IP 192.168.1.1.543 > 10.0.1.1.57269: Flags [S.], seq 214939753, ack 587559 \ 489, win 65535, options [mss 1460,nop,wscale 3,sackOK,TS val 2495570782 ecr 346673686] \ , length 0 09:55:10.105920 IP 10.0.1.1.57269 > 192.168.1.1.543: Flags [.], seq 1, ack 1, win 65535, o \ ptions [nop,nop,TS val 346673823 ecr 2495570782], length 0
If there is no auto-discovery probe seen on the outgoing SYN packet, then the optimization service might not be enabled on that particular in-path interface, the appliance might be in admission control or there is asymmetric routing happening.
The next step is to see what the server-side Steelhead appliance sees about this TCP session.
The first step would be to check in the GUI on the Current Connections page for this TCP session, just like on the client-side Steelhead appliance.
If the pass-through reason is SYN on the WAN side, then the auto-discovery TCP option has been stripped on the path between the client-side Steelhead appliance and the server-side Steelhead appliance.
If the pass-through type is Intentional pass-through, then the auto-discovery probe is ignored because of a peering-rule.
Next step is to confirm that the server-side Steelhead appliance does see an auto-discovery probe in the SYN+ packet.
Figure 7.12. Running tcpdump on the client-side WAN side, seeing the SYN+ packet
CSH # tcpdump -ni wan0_0 '(host 192.168.1.1 and host 10.0.1.1 and port 543) or (vlan and ( \ host 192.168.1.1 and host 10.0.1.1 and port 543))' tcpdump: WARNING: wan0_0: no IPv4 address assigned tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on wan0_0, link-type EN10MB (Ethernet), capture size 300 bytes 10:05:30.767917 IP 10.0.1.1.57489 > 192.168.1.1.543: Flags [S], seq 1098146093, win 65535, \ options [mss 1460,nop,wscale 3,nop,nop,TS val 347290479 ecr 0,sackOK,nop,nop,rvbd-pro \ be AD CSH:10.0.1.6 01010a0001060005,rvbd-probe EAD 0c01,nop,eol], length 0 10:05:30.900270 IP 192.168.1.1.543 > 10.0.1.1.57489: Flags [S.], seq 2704856677, ack 10981 \ 46094, win 65535, options [mss 1460,nop,wscale 3,sackOK,TS val 1624055221 ecr 34729047 \ 9], length 0 10:05:30.903472 IP 10.0.1.1.57489 > 192.168.1.1.543: Flags [.], seq 1, ack 1, win 65535, o \ ptions [nop,nop,TS val 347290613 ecr 1624055221], length 0
Figure 7.13. Running tcpdump on the server-side WAN side without the SYN+ packet
CSH # tcpdump -ni wan0_0 '(host 192.168.1.1 and host 10.0.1.1 and port 543) or (vlan and ( \ host 192.168.1.1 and host 10.0.1.1 and port 543))' tcpdump: WARNING: wan0_0: no IPv4 address assigned tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on wan0_0, link-type EN10MB (Ethernet), capture size 300 bytes 09:47:29.245645 IP 10.0.1.1.57489 > 192.168.1.1.543: Flags [S], seq 1098146093, win 65535, \ options [mss 1460,nop,wscale 3,nop,nop,TS val 347290479 ecr 0,sackOK,nop,nop,nop,nop, \ nop,nop,nop,nop,nop,nop,nop,nop,nop,nop,nop,eol], length 0 09:47:29.245784 IP 192.168.1.1.543 > 10.0.1.1.57489: Flags [S.], seq 2704856677, ack 10981 \ 46094, win 65535, options [mss 1460,nop,wscale 3,sackOK,TS val 1624055221 ecr 34729047 \ 9], length 0 09:47:29.382200 IP 10.0.1.1.57489 > 192.168.1.1.543: Flags [.], seq 1, ack 1, win 65535, o \ ptions [nop,nop,TS val 347290613 ecr 1624055221], length 0
If a normal naked SYN packet gets seen on the server-side Steelhead appliance, although there was a SYN+ packet send by the client-side Steelhead appliance, then it is stripped somewhere in the path between the client-side Steelhead appliance and the server-side Steelhead appliance. Firewalls, intrusion detection and prevention systems (IDS/IPS) are well known for removing the probes.
If there is neither a SYN+ nor a naked SYN packet seen on the server-side Steelhead appliance, then the path of the traffic between the client and the server does not include the server-side Steelhead appliance. Use the traceroute program to determine the path of the traffic.
When the server-side Steelhead appliances sees a SYN+ packet and it is willing to peer with it, it will return a SYN/ACK+ packet. This packet should arrive at the client-side Steelhead appliance. If it doesn't get seen at the client-side Steelhead appliance, then the path the traffic takes from the server-side Steelhead appliance to the client does not go via the client-side Steelhead appliance.
If both the Steelhead appliances see the auto-discovery probes, then the inner channel will be setup.
Once the two Steelhead appliance have detected each other via the auto-discovery probe in the TCP Options, the client-side Steelhead will setup an inner channel to the server-side Steelhead appliance.
The behaviour in setup of an inner channel is different when the two appliances do or don't have a Out-of-Band Splice setup between them.
This can be determined via the list of Connected Appliances (available
with the CLI command
show peers
and in the GUI at Reports -> Optimization -> Connected Appliances).
If the peer in-path IP address shows up as Online, then an OOB
Splice has been established already.
When no Out-of-Band Splice exists, the optimization service will setup a new OOB TCP session between the IP address of the in-path interface on the client-side Steelhead appliance and the IP address of the in-path interface on the server-side Steelhead appliance on TCP port 7800.
On the client-side Steelhead appliance a TCP SYN packet will be sent, it should arrive on the server-side Steelhead appliance and be replied by a TCP SYN/ACK packet. The TCP SYN/ACK packet should arrive on the client-side Steelhead appliance and it should be replied with by a TCP ACK packet.
Note that the path the SYN+ packet from the client takes can be different from the path that the TCP SYN packet from the inner channel. This also means that the behaviour of various networking equipment, like routing decisions, the traffic shaper behaviour and the firewall policy, can be different.
Once the TCP session for the OOB splice is successfully setup, the Steelhead appliances will perform the OOB handshake during which they will exchange details and capabilities and the OOB Splice is completed. After that, unless one of the Steelhead appliances goes into Admission Control and will terminate the OOB Splice, there will be no more data exchanged over the OOB Splice.
If the OOB Splice does not get established, the inner channel for the optimized TCP session will also not be setup and the TCP session will be passed-through. Use the traceroute command to see if there is a difference in the path the traffic takes from the client to the server and in the path it takes for traffic from the IP address of the in-path interface of the client-side Steelhead appliance to the IP address of the in-path interface of the server-side Steelhead appliance.
The next step is the setup of a Connection Pool of 20 TCP sessions which can be converted later to inner channels. These are normal TCP sessions from the IP address of the in-path interface of the client-side Steelhead appliance to the IP address of the in-path interface of the server-side Steelhead appliance on TCP port 7800, but no data is send over it until later. If the OOB Splice got setup, then the setup of these TCP sessions shouldn't be a problem.
The final step is the conversion of one of the TCP sessions of the Connection Pool into an inner channel by sending the inner channel handshake over the TCP session and once acknowledged with a TCP ACK, the inner channel is established.
When the Out-of-Band Splice already exists, there will also be the Connection Pool, a pool of pre-setup TCP sessions between the in-path interfaces of the client-side and server-side Steelhead appliances available to be converted into inner channels for optimized TCP sessions.
The Connection Pool exists as long as the OOB Splice exists. The OOB Splice is a TCP session with a very short TCP keep-alive interval and thus should be the first to know that the server-side Steelhead is not reachable.
Before they start being used, these TCP sessions in the Connection Pool might have already been there for a long time, lying idle until a new TCP session gets optimized. When needed, the inner channel handshake gets send over one of the TCP sessions in the Connection Pool and once acknowledged with a TCP ACK, the TCP session is converted into an inner channel.
If the TCP ACK expected after the inner channel handshake does not show up at the client-side Steelhead appliance, then the inner channel might have been timed out in a session-table in a firewall.
If the reply to the inner channel handshake is a TCP RST packet, then there is possibly a firewall in the middle which has expired the TCP session.
With the Port Transparency WAN Visibility mode, the inner channel consists of a TCP session from the IP address of the in-path interface of the client-side Steelhead appliance to the IP address of the in-path interface of the server-side Steelhead appliance, but with the TCP port numbers of the original TCP session.
Because of the different TCP port numbers, it is not possible to setup a Connection Pool in advance and the setup of the TCP session for the inner channel does need to happen for every optimized TCP session with the Port Transparency WAN Visibility mode.
After the auto-discovery phase, the client-side Steelhead appliance will send a TCP SYN packet with the Transparency TCP Option in it. This Transparency TCP Option has the IP addresses of the in-path interfaces of the Steelhead appliances in it and the TCP port numbers used on the Steelhead appliances to identify the inner channel.
Just with the Correct Addressing WAN Visibility, the path that the TCP SYN packet with the Transparency TCP option takes can be different from the path that the SYN+ packet takes. This also means that the behaviour of various networking equipment, like the routing decisions, the traffic shaper behaviour and the firewall policy, can be different.
If this TCP SYN packet with the Transparency TCP Option does not reach the server-side Steelhead appliance, check the path the TCP SYN packet takes with the traceroute command.
If the path is the same, then check if a normal TCP session without the Transparency TCP Options towards the IP address of the server-side Steelhead appliance but to the server TCP port does arrive on the server-side Steelhead appliance. If it does arrive there, then there is something in the path which dropped the packet because of the Transparency TCP Option.
After the TCP SYN packet with the Transparency TCP Option arrives at the server-side Steelhead appliance, it will send a TCP SYN/ACK with the Transparency TCP Option back to the client-side Steelhead appliance which will send the final TCP ACK with the Transparency TCP Option and the TCP session of the inner channel is setup.
Once the TCP session of the inner channel is setup, the client-side Steelhead appliance will send the inner channel handshake over the TCP session and once acknowledged with a TCP ACK, the inner channel is established.
With the Full Transparency WAN Visibility mode, the inner channel consists of a TCP session from the IP address of the original TCP session and with the TCP port numbers of the original TCP session.
Also here it is not possible to setup a Connection Pool in advance and the setup of the TCP session for the inner channel does also need to happen for every optimized TCP session with the Full Transparency WAN Visibility mode.
After the auto-discovery phase, the client-side Steelhead appliance will send a TCP SYN packet with the Transparency TCP Option in it, just like with Port Transparency.
Unlike with the Correct Addressing and the Port Transparency WAN Visibility, the path that the TCP SYN packet with the Transparency TCP option takes is the same as the SYN+ packet takes. This means that firewall networking equipment might complain now: The previous TCP SYN packet had different TCP sequence numbers.
If this TCP SYN packet with the Full Transparency TCP Option does not reach the server-side Steelhead appliance, change the in-path rule from the Full Transparency WAN Visibility to the Full Transparency With Firewall Reset WAN Visibility. This will cause the Steelhead appliance to send a TCP RST before sending the TCP SYN packet with the Full Transparency TCP Options, attempting to clear any firewall state tables.
After the TCP SYN packet with the Transparency TCP Option arrives at the server-side Steelhead appliance, it will send a TCP SYN/ACK with the Transparency TCP Option back to the client-side Steelhead appliance which will send the final TCP ACK with the Transparency TCP Option and the TCP session of the inner channel is setup.
Once the TCP session of the inner channel is setup, the client-side Steelhead appliance will send the inner channel handshake over the TCP session and once acknowledged with a TCP ACK, the inner channel is established.