Appendix B. Troubleshooting workflow for network performance related issues

Table of Contents

B.1. Step One: Does the TCP session get optimized?
B.2. Step Two: Does TCP optimization and data reduction work?
B.3. Step Three: Does latency optimization work

This appendix describes the steps to be taken to isolate network performance related issues.

B.1. Step One: Does the TCP session get optimized?

If the optimized TCP session gets setup properly, it will be displayed in the list of Current Connections. If the TCP session doesn't show there, this chapter needs to be followed.

Most of the troubleshooting for this step can be done via the GUI or by checking the output of the tcpdump tool.

The setup of an optimized TCP session consists of three phases: The auto-discovery phase, the OOB channel and the setup of the inner and outer channels phase.

B.1.1. Phase A: Does the auto-discovery phase complete?

During the auto-discovery phase, the client-side Steelhead (CSH) should intercept the naked SYN from the client on the LAN interface, attach the auto-discovery probe and send the SYN+ out on the corresponding WAN interface.

Why was the naked SYN not seen?

  • The traffic towards the server does not go via the CSH.

    • Are there other paths from the client's LAN switch to the WAN router.

    • Check the MAC tables on the various devices to confirm that the traffic towards the CSH in-path IP addresses and the gateway behind the CSH are going out via the same port.

    • Are there any other CSHs there?

  • In case of a virtual in-path WCCP/PBR deployment, the WCCP/PBR router does not forward the traffic towards the CSH.

    • Check the redirection access-lists on the WCCP/PBR routers to confirm that the client/server IP addresses are matching.

    • Check that the incoming interface for the client traffic does have the redirection statements configured.

    • Check that the WCCP routers and the CSH can see each other.

Why did the auto-discovery probe not get attached?

  • The optimization service is not running.

    • It can have crashed, is turned off, didn't want to start.

    • There is a connection-forwarding failure or fail-over mode.

  • The naked SYN showed up on the WAN interface for in-path deployments.

    • LAN/WAN cable switched.

  • The in-path interface is not enabled for optimization.

    • Check the settings under Configure -> Optimization -> General Settings.

  • A pass-through in-path rule prevents it.

    • Check the in-path rules.

  • The CSH is in connection-based admission control.

    • Confirm that there has been no admission control alarms.

  • The client/server IP addresses are in the asymmetric routing table.

    • Check the asymmetric routing table and flush it if necessary.

  • There is no space in the TCP header to store the auto-discovery probe.

    • Check the kernel logs this with the command support show kernel-logs.

The SYN+ should arrive on the server-side Steelhead (SSH).

Why does the SYN+ not get seen?

  • Routing issues between CSH and SSH.

    • Run traceroute and confirm that the traffic goes through the SSH.

  • A naked SYN gets seen!

    • Auto-discovery probe gets stripped by firewall.

    • Compare TCP headers.

    • Third SYN from the client? A Firewall doesn't like the auto-discovery probe and drops the SYN+.

  • Traffic passed a NAT gateway which changes source IP address and maybe also stripped off the auto-discovery probe.

    • Try fixed-target rules

In case of Enhanced Auto-Discovery, the SSH will send a SYN/ACK++ to the CSH and forwards the SYN+ via the corresponding interface and waits for a SYN/ACK.

Why does the SYN/ACK++ not get seen on the SSH?

  • The in-path interface gateway is set to the LAN side which doesn't send it back via this in-path interface.

    • Check in-path routing table.

    • Check the LAN side interface.

  • In-path interface cannot resolve the gateway IP address.

    • Inspect ARP table.

    • Try to ping the default gateway from the in-path interface IP address.

  • The optimization service is not running.

    • It can have crashed, is turned off, didn't want to start.

    • There is a connection-forwarding failure or fail-over mode.

  • The in-path interface is not enabled for optimization.

    • Check the settings under Configure -> Optimization -> General Settings.

  • A pass-through peering rule prevents it.

    • Check the peering rules.

  • The SSH is in connection-based admission control.

    • Confirm that there has been no admission control alarms.

Why does the SYN/ACK++ not get seen by the CSH?

  • Routing issues between the CSH and SSH.

    • Run traceroute and confirm that the traffic goes through the CSH.

  • A naked SYN/ACK arrives.

    • Auto-discovery probe gets stripped by a firewall

    • Compare TCP headers.

  • Asymmetric routing bypasses the CSH.

    • You should see a TCP RST from the client.

Why does the SYN+ not get seen leaving the SSH?

  • No idea. Does this happen?

Why does the SYN/ACK not get seen?

  • The server doesn't answer.

    • The server is off.

    • The traffic is blocked by firewall.

  • The traffic from the server to the client does not go via the SSH.

    • This is a case of server-side asymmetry.

The SSH will send a SYN/ACK+ towards the CSH.

Why does the SYN/ACK+ not get send?

  • The in-path interface gateway is set to the LAN side which doesn't send it back via the WAN interface.

    • Check in-path routing table.

    • Check the LAN interface.

  • In-path interface cannot resolve the gateway IP address.

    • Inspect ARP table.

    • Try to ping the default gateway from the in-path interface IP address.

Why does the SYN/ACK+ not get received by the CSH?

  • Asymmetric routing bypasses the CSH.

    • You should see a TCP RST from the client on the LAN side.

  • In case of Enhanced Auto-Discovery, a firewall blocks the SYN/ACK+ because it has already seen it as the SYN/ACK++.

  • A naked SYN/ACK arrives

    • Auto-discovery probe gets stripped by a firewall.

    • Compare TCP headers.

At the end of the auto-discovery phase, the CSH will not send a final ACK of the TCP handshake.

B.1.2. Phase B: Do the OOB channel and the Connection Pool get setup?

After the auto-discovery phase, the CSH will setup an OOB channel to the SSH. This is a TCP session from the CSH in-path interface to the SSH in-path interface on TCP port 7800.

Why does the OOB TCP session not get setup?

  • Routing issues between the CSH and SSH.

    • Run ping and traceroute to see check the paths.

    • Telnet from CSH in-path IP address to SSH in-path IP address on TCP port 7800.

    • Run tcpdump to see which part doesn't work.

  • Timeout in the setup of the OOB channel.

    • Traffic towards SSH is blocked.

After the OOB channel is setup, the CSH will setup the Connection Pool to the CSH. This is a set of 20 TCP sessions from the CSH in-path interface to the SSH in-path interface on TCP port 7800.

Why do the Connection pool TCP sessions not get setup?

  • The CSH in-path rules state that the inner channel WAN visibility is Port of Full Transparency.

  • Routing issues between the CSH and SSH.

    • Run ping and traceroute to see check the paths.

    • Telnet from CSH in-path IP address to SSH in-path IP address on TCP port 7800.

    • Run tcpdump to see which part doesn't work.

  • Timeout of the setup of the TCP Connections.

    • Traffic towards SSH is blocked.

B.1.3. Phase C: Does the inner channel get established?

After the auto-discovery phase, the CSH will setup an Inner Channel.

For the Correct Addressing WAN visibility, this will happen by the conversion of a TCP session in the Connection Pool into an inner channel.

Why does the conversion from TCP session of the Connection Pool into an inner channel fail?

  • Setup packet doesn't arrive at SSH.

    • The TCP sessions could have timed out by a firewall in the network. Reduce TCP Keep-Alive interval for the inner channels.

For the Port Transparency WAN visibility, this will be a normal TCP session from the CSH in-path interface IP address to the SSH in-path interface IP address, but with the TCP port of original naked SYN.

Why does the inner channel not get setup?

  • Routing issues between the CSH and SSH.

    • Run ping and traceroute to see check the paths.

    • Take traces to see which part doesn't get setup.

  • Traffic towards SSH is blocked.

    • Timeout of the setup of the inner channel.

  • A naked SYN arrives at the SSH.

    • Transparency options get stripped by a firewall.

For the Full Transparency WAN visibility, this will be a TCP session from the original client IP address to the original server IP address, but with the TCP port of original naked SYN.

Why does the inner channel not get setup?

  • A firewall doesn't allow the "old" SYN with different TCP sequence numbers.

    • Use Full Transparency with Firewall Reset to reset the state tables on the firewall.

Once the TCP session is setup, the CSH will send the splice setup message and the inner channel is setup.