7.6. Connection-based Admission Control

As discussed earlier, there are several kinds of admission control. This chapter only focuses on connection-based admission control, where the number of optimized TCP sessions going through the Steelhead appliance is larger than the licensed amount.

The first symptom will be the alarm on the Steelhead appliance, delivered to you via email or an SNMP trap. In the reports section, the graph of the Connection History will show a flat line at the time the Steelhead appliance was in connection-based admission control:

Figure 7.17. Connection-based admission control for a 5520 model

Connection-based admission control for a 5520 model

At midnight, the number of optimized TCP sessions was below 15 000. At around 08:00, it has risen to 15 000 and the optimization service went into admission control. At 16:00, it has dropped back to below 15 000 and the optimization service came out of admission control.

There are several reasons for entering admission control:

7.6.1. Sizing issues

In either of these two examples, the capacity of the current model is not sufficient anymore and should either be upgraded using a higher license or upgraded to a larger appliance if a license upgrade is not possible.

7.6.1.1. Natural growth

Natural growth is related to an increase of the number of clients behind the Steelhead appliance or to an increase in network-based applications.

7.6.1.2. Change in protocols spoken

A change in the protocols spoken is related to a migration from one protocol to another, like from POP3 to MAPI: POP3 uses short-lived TCP sessions so for 100 clients you can live with 20 concurrent TCP sessions for this, while MAPI uses at least 3 long-lived TCP sessions so for 100 clients you suddenly need about 300 concurrent TCP sessions already!

7.6.2. Malicious traffic

Use the Current Connections overview to see if there are obvious hosts to look at.

7.6.2.1. Port scanners

The behaviour of a TCP port scanner is that it sends out a large amount of TCP SYN packets and waits for any answers. When a Steelhead appliance sees these TCP SYN packets, it will try to setup optimized TCP sessions and can run into Connection-based admission control.

The way to find them is to run tcpdump and filter for packets with the TCP SYN flag set with the command tcpdump -ni lan0_0 -c 100 'tcp[tcpflags] & tcp-syn != 0'.

The best practice is to create an in-path rule on the local Steelhead appliance to not optimize any traffic from the host which is doing the port scanning.

7.6.2.2. How to determine the application

Once the offending host has been determined, the next step will be to identify the application which is sending the traffic.

On Linux and Unix systems, this can be done with the command lsof:

Figure 7.18. Use "lsof" to determine which application is owning a TCP session

$ lsof -n | grep 192.168.1.1:543
nc       5637 edwin     3u     IPv4   56433930    TCP  192.168.1.1:543->10.0.1.1:53036 (ES \
    TABLISHED)

In this case the application nc which is using the TCP session between 10.0.1.1:53036 and 192.168.1.1:543.

On Microsoft Windows, this can be done with the output of netstat -no to determine the Process ID:

Figure 7.19. Use "netstat -no" to determine the Process ID of the application owning a TCP session

C:\Users\edwin.MAVETJU>netstat -no

Active Connections

  Proto  Local Address          Foreign Address        State           PID
  [...]
  TCP    10.17.6.60:51749       173.194.79.125:5222    ESTABLISHED     1908
  [...]

And the Windows Task Manager, add the PID to the output via View -> Select Columns, to determine the application:

Figure 7.20. Windows Task Manager with Process ID

Windows Task Manager with Process ID

In this case the application "pidgin.exe" which is using the TCP session between 10.0.1.1:51749 and 173.194.79.125:5222.

7.6.3. Failure to tear down TCP sessions

A TCP session starts with a SYN, SYN/ACK and ACK handshake and is terminated with a RST if things go wrong or with a FIN where both side have successfully received all the traffic.

Once the client or the server has sent a FIN, the TCP session is in a half-closed state. Once both the client and the server have sent a FIN, the TCP session is closed.

As long as an optimized TCP session is in the half-closed state, it counts towards the TCP session usage. If the FIN by the other side is never send, the TCP session on the Steelhead appliance will stay open indefinitely. In the Current Connection report, you will then see an abnormal large amount of half-closed TCP sessions. For example for a 250H model, it will show 20 optimized TCP sessions and 200 half-closed TCP sessions.

According to the RFC about TCP, the half-closed timer for TCP sessions should expire after 2 times MSL (Maximum Segment Lifetime, most common is 60 seconds). So to overcome this half-closed state issue, the following CLI command can be entered to close the half-closed TCP sessions after certain period: protocol connection lan half-closed kill-timeout <number of seconds>.

If a half-closed TCP session is about to be terminated due to this time-out, it will show up in the logs as:

Figure 7.21. A half-closed TCP is about to be terminated

SH sport[431]: [io/outer/cons.INFO] 5 {192.168.0.1:1025 10.0.1.1:80} Kicking off half-clos \
    ed connection