The final result is the same, the client and the server stop communicating with each other, but the cause is different:
An optimized TCP session hangs when traffic in one direction is blocked, an optimized TCP session gets reset when the client or the server gets a TCP RST packet.
A TCP session gets reset when it receives a TCP RST packet. TCP RST packets are generated when:
A TCP SYN packet arrives at the server but there is no service listening on the destination TCP port. The client will report this as Connection refused.
A TCP SYN/ACK packet arrives at the client but the client doesn't have a socket open on that TCP port. This can happen the program which has opened the socket has already closed it before the TCP SYN/ACK returns.
A TCP packet arrives at a client or server but that one doesn't know about this TCP session anymore. This can happen when a long idle TCP session comes back alive and the machine on the other side has been restarted.
A TCP packet arrives at a client or server but the TCP sequence numbers are invalid. This can happen in WAN optimization environments where the SYN/ACK+ bypasses both the client-side and server-side Steelhead appliances.
A firewall sees traffic for a TCP session which is not in its TCP session table. This can happen when the TCP session comes back online after it has been idle for a long time and the TCP session has timed-out on the firewall.
A firewall sees incorrect TCP sequence behaviour. This happens when traffic for a second TCP session with the same IP addresses and TCP port numbers gets seen. This can happen when using the Full Transparency WAN Visibility.
A firewall detects a time-out on a state change in the TCP three-way handshake and resets the TCP session towards the client and the server to alert it that the TCP session could not be established. This can happen during the auto-discovery process when a firewall in the middle only sees the SYN+ and SYN/ACK+ and not the final TCP ACK.
An Intrusion Prevention System detects a violation of the protocol on top of TCP and aborts the TCP session. This can happen when the Port Transparency or the Full Transparency WAN Visibility is used in a network with an IPS device between the Steelhead appliances.
For an optimized TCP session, which consists of three TCP sessions, the inner channel between the two Steelhead appliances fails and therefore the two outer channels also get reset.
A hanging TCP session happens when traffic being sent out from one side is blocked in the network and the TCP Window of the sender is filling up and stops sending further packets.
This can happen when:
A device in the path is buffering the traffic while it waits for a network path to become available. This could happen when a VPN tunnel gets restarted or when a non-IP routing device is buffering the packets.
A device in the path is blackholing the traffic.
A device in the path cannot forward the IP packet because of MTU size issues and the ICMP Fragmentation Needed packets send by the blocking device do not arrive at the sender.
An Intrusion Prevention System has detected a protocol violation on top of TCP and discards the packet. This can happen when the Port Transparency or the Full Transparency WAN Visibility is used in a network with an IPS between the Steelhead appliances.
This issue can be detected by tracing the TCP session at the sender and see that despite sending various TCP packets, there is no TCP acknowledgement coming from the other side.