5.3. Admission Control

Admission Control is a state of the optimization service where it will no longer intercept any new TCP sessions because of licensing limitations or internal problems.

When in admission control, except for MAPI based admission control, the Steelhead appliance will drop all the Out-of-Band splices and the TCP sessions of the Connection Pools to the connected Steelhead appliances. When the Steelhead appliance comes out of admission control, the OOB Splice and the Connection Pool will be setup again once a new optimized TCP session towards the remote Steelhead appliance will be setup.

The various values for admission control can be seen in the CLI with the command show admission:

Figure 5.20. Output of the command "show admission" for a model 560L

SH # show admission 
Enable Admission Control Override Settings: no

  Override Settings:
    Connection Enable:   250
    Connection Cutoff:   260
    Memory Enable:       1300 MB
    Memory Cutoff:       1400 MB
    Low Memory Ratio:    96%

  Current Settings:
    Connection Enable:   250
    Connection Cutoff:   260
    Memory Enable:       1300 MB
    Memory Cutoff:       1400 MB
    Low Memory Ratio:    96%

  Current State:
    Connections:         2
    Memory:              995 MB

Or in the system dump with the following command:

Figure 5.21. Determining the admission control values via the system dump

sh-4.1# grep /rbt/sport/admission/config/ active-brief.txt 
/rbt/sport/admission/config/cpu_util/enable = false (bool)
/rbt/sport/admission/config/cutoff_conn = 260 (uint32)
/rbt/sport/admission/config/cutoff_mem = 1400 (uint32)
/rbt/sport/admission/config/enable_conn = 250 (uint32)
/rbt/sport/admission/config/enable_mem = 1300 (uint32)
/rbt/sport/admission/config/low_mem_ratio = 96 (uint32)
/rbt/sport/admission/config/mapi/enable = false (bool)
/rbt/sport/admission/config/mapi/threshold = 85 (uint32)
/rbt/sport/admission/config/override = false (bool)
/rbt/sport/admission/config/tcp_mem/enable = true (bool)
/rbt/sport/admission/config/unlock/licensed = false (bool)
/rbt/sport/admission/config/vm/dirty_background_ratio = 5 (uint8)
/rbt/sport/admission/config/vm/dirty_expire_centisecs = 800 (uint32)
/rbt/sport/admission/config/vm/dirty_ratio = 6 (uint8)
/rbt/sport/admission/config/vm/dirty_writeback_centisecs = 300 (uint32)

5.3.1. Connection-based admission control

Every Steelhead has a licensed maximum number of concurrent TCP connections, for example, the 250H model which has a maximum of concurrent 200 TCP sessions.

In the admission control algorithm, this value is used to define when the interception of new TCP sessions should be enabled again. Interception is disabled when this value plus some spare is reached. This spare amount is to dampen the flapping of the admission control status. Interception is enabled again when the number of optimized TCP drops below this value.

Figure 5.22. Connection-based admission control for a 250H model

Connection-based admission control for a 250H model

The flat yellow line between 08:00 and 14:00 shows that the device was in connection-based admission control the entire time. Check the Current Connections report to determine which hosts uses most and if their traffic is legit.

5.3.1.1. Determining via the GUI

If the issue is still happening the data in the list of Current Connections can be used to find out if there is a host with an extraordinary large amount of optimized TCP sessions in the list.

5.3.1.2. Determining via the system dump

If a system dump is available, the file connection_table can be used to determine the host with the most TCP sessions:

Figure 5.23. Check the list of TCP sessions to find the host which uses the most

[~/systemdump-SH] edwin@t43>awk '{ print $1 }' < connection_table | sed -e 's/:.*//' | sor \
    t | uniq -c | sort -nr | head
 120   10.0.1.6
 112   10.0.1.1
   5   10.0.1.5
   3   10.0.1.8

The IP address 192.168.1.6 should be ignored since it is the IP address of an in-path interface. The host with IP address 192.168.1.1 is using 112 TCP sessions.

The next steps would be to find the TCP sessions for that host:

Figure 5.24. Get the list of TCP sessions for one specific host

[~/systemdump-SH] edwin@t43>grep 192.168.1.1 connection_table
10.0.1.1:6802 EAL_DEV_LAN 192.168.1.1:80 EAL_DEV_LOCAL CONN_ESTABLISHED 1338909287 0 NONE  \
    0 0 inpath0_0
10.0.1.1:6814 EAL_DEV_LAN 192.168.1.1:80 EAL_DEV_LOCAL CONN_CLOSED 1338909304 0 NONE 0 0 i \
    npath0_0
10.0.1.1:6832 EAL_DEV_LAN 192.168.1.1:80 EAL_DEV_LOCAL CONN_CLOSED 1338909308 0 NONE 0 0 i \
    npath0_0
10.0.1.1:6846 EAL_DEV_LAN 192.168.1.1:80 EAL_DEV_LOCAL CONN_ESTABLISHED 1338909329 0 NONE  \
    0 0 inpath0_0
10.0.1.1:6819 EAL_DEV_LAN 192.168.1.1:80 EAL_DEV_LOCAL CONN_CLOSED 1338909306 0 NONE 0 0 i \
    npath0_0
[...]

So for this host most of the traffic went to the host with IP address 192.168.1.1 on TCP port 80.

5.3.2. MAPI connection-based admission control

On the network level, a MAPI session uses multiple TCP sessions to communicate towards the Exchange server. All these TCP sessions should be optimized and go through the same Steelhead appliances, otherwise the latency optimization will not work properly and MAPI connections can be interrupted.

To prevent MAPI sessions from becoming partly optimized, the Steelhead appliance can be configured to stop performing latency optimization for new MAPI sessions when the Steelhead appliance is using 85% of the maximum number of concurrent TCP sessions. Non-MAPI traffic will still be optimized, new TCP sessions from current MAPI sessions will still be optimized, only new TCP sessions from new MAPI sessions will not be optimized anymore.

Figure 5.25. Output of the command "show admission internal"

SH (config) # admission control mapi threshold 85
SH (config) # show admission internal
    TCP Memory Enabled:      yes
    CPU Utilization Enabled: no
    MAPI Enabled:            no
    MAPI Threshold:          85

5.3.3. Memory-based admission control

During startup, the optimization process will allocate a large chunk of memory for its own internal memory pool. When new optimized TCP sessions get setup, memory from this pool will be allocated for its data-structures and buffers.

There are two situations where this memory-pool can run out of space and the optimization service will go in memory-based admission control:

  • The optimization service is servicing so many TCP connections that it uses all the available memory. When the Steelhead appliance constantly enters and leaves memory-based admission control, this is most likely what is happening. If it happens for the first time, the first step would be to create a system dump and open a case with Riverbed TAC.

    The solution would be to reduce the amount of TCP sessions to be optimized via:

    • In-path rules to disable optimization for certain TCP sessions.

    • Additional hardware in a virtual in-path deployment cluster.

    • The increase the capacity of the Steelhead appliance, either by a license upgrade or by a model upgrade.

  • The optimization service is allocating memory and never releasing it, this is called a memory-leak. If this happens, the first step would be to create a system dump and open a case with Riverbed TAC, but do not restart the optimization service yet. Unless the cause can be determined from the system dump provided, Riverbed TAC might want to obtain a process dump of the optimization service.

    Once Riverbed TAC has all the required information, the immediate solution would be to restart the optimization service, the long term solution would be a software upgrade to a RiOS version where the memory-leak is resolved in.

5.3.4. TCP Memory Pressure

TCP Memory Pressure admission control happens when the optimization service is running out of memory allocated for network buffers.

Figure 5.26. Optimizaiton service going into TCP Memory Pressure based admission control

sport[9672]: [admission_control.NOTICE] - {- -} TCP Memory Pressure.
sport[9672]: [admission_control.WARN] - {- -} Pausing intercept: TCP Memory Pressure;
sport[9672]: [admission_control.NOTICE] - {- -} Memory Usage: 3298 Current Connections: 16 \
    55 TCP Memory Usage: 34869;

This can happen because of a slow client or server or because of a slow remote Steelhead appliance. The amount of memory used for every TCP session can be found in the file sysinfo.txt in the system of the device:

Figure 5.27. Non-zero receive queues and non-zero send queues

Output of 'netstat -a --numeric-hosts':"
[...]
Proto Recv-Q Send-Q Local Address           Foreign Address     State 
tcp        0 108228 192.168.1.6:23456       192.168.1.1:543     ESTABLISHED
tcp   208322      0 192.168.1.6:7801        10.0.1.6:1040       ESTABLISHED

For the non-zero send queue, most likely there will be a large amount of TCP sessions towards a single host. Determining this host and finding out why it is so slow would be the next steps.

For the non-zero receive queue, a case with Riverbed TAC should be opened since this is a slowness towards the optimization service.

5.3.5. Optimized WAN bandwidth limitation

Technically speaking this is not an admission control related issue, the optimized WAN bandwidth limitation is a licensing related feature which will limit the bandwidth of the outgoing optimized traffic. This is not affecting pass-through traffic.

Different models have different optimized WAN bandwidth limitations. For example the 250M model has a limit of 1 Mbps, while the 250H model has a limit of 2 Mbps. Not all the models have capped optimized WAN bandwidth limitations, for example the 2050 model has a 45 Mbps optimized WAN bandwidth speed which is not capped but up to which the model is capable of running.

This limitation is implemented by a 200 milliseconds separated burst of traffic. This trace was taken for a transfer of a 1 Mb file via a 250M model via an optimized TCP session with only TCP optimization and no data reduction:

Figure 5.28. Wireshark I/O graph for a 1 Mbps optimized WAN bandwidth limit

Wireshark I/O graph for a 1 Mbps optimized WAN bandwidth limit

Figure 5.29. Wireshark TCP graph for a 1 Mbps optimized WAN bandwidth limit

Wireshark TCP graph for a 1 Mbps optimized WAN bandwidth limit

It clearly shows the 200 ms interval during which no traffic is send.

Figure 5.30. Optimized Throughput graph for a 1 Mbps optimized WAN bandwidth limit

Optimized Throughput graph for a 1 Mbps optimized WAN bandwidth limit

The Optimized Throughput graph is showing to be pegged at 1 Mbps.