As discussed earlier, there are several kinds of admission control. This chapter only focuses on connection-based admission control, where the number of optimized TCP sessions going through the Steelhead appliance is larger than the licensed amount.
The first symptom will be the alarm on the Steelhead appliance, delivered to you via email or an SNMP trap. In the reports section, the graph of the Connection History will show a flat line at the time the Steelhead appliance was in connection-based admission control:
At midnight, the number of optimized TCP sessions was below 15 000. At around 08:00, it has risen to 15 000 and the optimization service went into admission control. At 16:00, it has dropped back to below 15 000 and the optimization service came out of admission control.
There are several reasons for entering admission control:
Sizing issues, caused by natural growth of the network or by a change in the protocols spoken.
Malicious traffic, caused by clients which are consuming large amount of TCP sessions without any good reason.
Failure to tear down TCP sessions, caused by clients or servers not properly taking down TCP sessions.
In either of these two examples, the capacity of the current model is not sufficient anymore and should either be upgraded using a higher license or upgraded to a larger appliance if a license upgrade is not possible.
Natural growth is related to an increase of the number of clients behind the Steelhead appliance or to an increase in network-based applications.
A change in the protocols spoken is related to a migration from one protocol to another, like from POP3 to MAPI: POP3 uses short-lived TCP sessions so for 100 clients you can live with 20 concurrent TCP sessions for this, while MAPI uses at least 3 long-lived TCP sessions so for 100 clients you suddenly need about 300 concurrent TCP sessions already!
Use the Current Connections overview to see if there are obvious hosts to look at.
The behaviour of a TCP port scanner is that it sends out a large amount of TCP SYN packets and waits for any answers. When a Steelhead appliance sees these TCP SYN packets, it will try to setup optimized TCP sessions and can run into Connection-based admission control.
The way to find them is to run tcpdump and filter for packets with
the TCP SYN flag set with the command
tcpdump -ni lan0_0 -c 100 'tcp[tcpflags] & tcp-syn != 0'
.
The best practice is to create an in-path rule on the local Steelhead appliance to not optimize any traffic from the host which is doing the port scanning.
Once the offending host has been determined, the next step will be to identify the application which is sending the traffic.
On Linux and Unix systems, this can be done with the command
lsof
:
Figure 7.18. Use "lsof" to determine which application is owning a TCP session
$ lsof -n | grep 192.168.1.1:543 nc 5637 edwin 3u IPv4 56433930 TCP 192.168.1.1:543->10.0.1.1:53036 (ES \ TABLISHED)
In this case the application nc which is using the TCP session between 10.0.1.1:53036 and 192.168.1.1:543.
On Microsoft Windows, this can be done with the output of
netstat -no
to determine the Process ID:
Figure 7.19. Use "netstat -no" to determine the Process ID of the application owning a TCP session
C:\Users\edwin.MAVETJU>netstat -no Active Connections Proto Local Address Foreign Address State PID [...] TCP 10.17.6.60:51749 173.194.79.125:5222 ESTABLISHED 1908 [...]
And the Windows Task Manager, add the PID to the output via View -> Select Columns, to determine the application:
In this case the application "pidgin.exe" which is using the TCP session between 10.0.1.1:51749 and 173.194.79.125:5222.
A TCP session starts with a SYN, SYN/ACK and ACK handshake and is terminated with a RST if things go wrong or with a FIN where both side have successfully received all the traffic.
Once the client or the server has sent a FIN, the TCP session is in a half-closed state. Once both the client and the server have sent a FIN, the TCP session is closed.
As long as an optimized TCP session is in the half-closed state, it counts towards the TCP session usage. If the FIN by the other side is never send, the TCP session on the Steelhead appliance will stay open indefinitely. In the Current Connection report, you will then see an abnormal large amount of half-closed TCP sessions. For example for a 250H model, it will show 20 optimized TCP sessions and 200 half-closed TCP sessions.
According to the RFC about TCP, the half-closed timer for TCP
sessions should expire after 2 times MSL (Maximum Segment Lifetime,
most common is 60 seconds). So to overcome this half-closed state
issue, the following CLI command can be entered to close the
half-closed TCP sessions after certain period:
protocol connection lan half-closed kill-timeout <number of seconds>
.
If a half-closed TCP session is about to be terminated due to this time-out, it will show up in the logs as:
Figure 7.21. A half-closed TCP is about to be terminated
SH sport[431]: [io/outer/cons.INFO] 5 {192.168.0.1:1025 10.0.1.1:80} Kicking off half-clos \ ed connection