8.10. HTTP Latency Optimization

HTTP latency optimization works in various ways:

HTTP latency optimization doesn't work when:

8.10.1. How HTTP Latency Optimization work

8.10.1.1. HTTP Meta-Data Caching

A HTTP Reply header can contain the expected life-time of an object:

Figure 8.37. An HTTP Reply header

HTTP/1.0 200 OK
Content-Type: image/png
Last-Modified: Fri, 15 Jul 2011 23:21:33 GMT
Date: Mon, 14 Nov 2011 21:41:50 GMT
Expires: Mon, 15 Nov 2011 21:41:50 GMT
Cache-Control: public, max-age=86400
X-Content-Type-Options: nosniff
Server: example.mavetju.org
Content-Length: 35636
X-XSS-Protection: 1; mode=block

The following meta-data can be seen on this response:

  • The modification time of the object is Fri, 15 Jul 2011 23:21:33 GMT.

  • The date on the server is Mon, 14 Nov 2011 21:41:50 GMT.

  • The object can be used in public caches until Mon, 15 Nov 2011 21:41:50 GMT. This is an old notation style.

  • The object can be used in public caches for 86400 seconds (or 1 day). This is the new notation style.

There are several ways for a web-browser to get an object:

  • From its browser cache. From the last time the object was requested, the browser was told that it could use it for 86400 seconds.

  • After the 86400 seconds, the object in the browser-cache has become stale and the web-browser will ask the web-server to send the object again if it has been changed since its Last-Modified date. If it has been changed, the web server will send the new object. If it has not been changed, it will respond with a HTTP 304 response and a new Cache-Control line which states that the stale object can be used for another 86400 seconds. This is a low-bandwidth browser-cache mechanism for refreshing objects.

    This answer can either go all the way to the web-server or can be answered by a local web-cache or a server-side web-cache.

    Figure 8.38. A 304 response code

    GET /foo.png HTTP/1.1
    Host: example.mavetju.org
    User-Agent: Mozilla/5.0 (X11; FreeBSD i386; rv:22.0) Gecko/20100101 Firefox/22.0
    Referer: http://example.mavetju.org/
    Connection: keep-alive
    If-Modified-Since: Mon, 10 Mar 2008 18:47:44 GMT
    
    HTTP/1.1 304 Not Modified
    Date: Sun, 29 Sep 2013 17:06:08 GMT
    Server: Apache/2.2.3 (FreeBSD)
    Connection: close
    Expires: Sun, 29 Sep 2013 21:08:31 GMT
    Cache-Control: public, max-age=86400
    

  • The web-browser can ask the web-server to send the object forcing it to come from the web-server without allowing it to be served from any caches.

Figure 8.39. Network with Web-browsers, web-caches and web-servers

browser -- SH -- WAN -- SH -- cache -- WAN -- SH -- server cache -- server

When the web-browser asks for an object and the Steelhead appliance optimizes the HTTP session, it can keep track of the meta-data. If the client asks for an object refresh via an If-Modified-Since and the Steelhead appliance has already learned from a request via another client that the object hasn't changed and a new expiration time has been given, the Steelhead appliance can answer the web-browser that the object is still valid and provide a new expiration time. That doesn't save only bandwidth but also reduces the round trip time.

8.10.1.2. HTTP Object Prefetching

If the object requested by the web-server is of the type text/html, it can contain embedded objects like images, style sheet and JavaScript files.

If the Steelhead appliance parses the returned text/html object and fetches the embedded objects from the web-server before the web-browser has a chance to request them, then the Steelhead appliance can serve the objects to the web-browser locally instead of the web-browser having to go all the way to the web-server for it.

8.10.2. Troubleshooting HTTP latency optimization

8.10.2.1. Response of the first request takes too long

Unlike proxy servers where the TCP session is terminated on the proxy server, an optimized TCP session is setup to the remote server first before the latency optimization kicks in. Thus the first request will take longer to setup, especially if the Full Transparency WAN visibility is used for the inner channel.

8.10.2.2. Invalid URL or referrer field

The most common issue is that a requested URL or given referrer contains a space. While technically this is a violation of the syntax, the space should be replaced by a %20, a lot of scripts or web-browsers send it, while web-servers and proxies have been taught to deal with the issue.

Figure 8.40. An invalid URL error messages due to spaces in the URL

CSH sport[24250]: [http/client.INFO] 3277788 {10.0.1.1:24750 192.168.1.1:80} Request parse \
     error, code = HTTP_ERR_HEADER_LINE GET /foo /bar.png HTTP:/1.1\013\010Referrer: http: \
    //example.mavetju.org/\013\010Host: example.mavetju.org\013\010

By default, the Steelhead appliances will throw an error and not perform HTTP latency on this TCP session anymore, but with the command protocol http space-in-uri enable it will allow this and process the request anyway.

Figure 8.41. Output of the command "show protocol http"

CSH (config) # protocol http space-in-uri enable
You must restart the optimization service for your changes to take effect.
CSH (config) # show protocol http internal
[..]
Allow Parsing Space in URI:          yes

8.10.2.3. What kind of HTTP latency optimization is performed?

When performing the HTTP latency optimization, the client-side Steelhead will add a line in the HTTP response header:

Figure 8.42. X-RBT-Optimized header

X-RBT-Optimized-By: CSH (RiOS 6.1.5b) PT

The first part of the string is the name of the client-side Steelhead performing the latency optimization, the middle part contains the RiOS version running on the Steelhead appliance and the third part contains the optimization scheme code.

Table 8.1. HTTP Optimization Scheme Code

ULURL LearningThe optimization service learns the order of URLs being requested for a certain base URL and will request the next ones once a base URL gets requested
PPParse-and-PrefetchThe optimization service is parsing the return HTML code for objects it can request in advance
PTObject Prefetch Table (OPT)Response is served from the local Steelhead HTTP cache
MCMetadata Response 
RAReuse-Auth 
FNForce-NTLM 
SAStrip-Auth-hdr 
GRGratuitous-401The client-side Steelhead appliance replied with a 401 or 403
SCStrip-CompressThe header field "Accept-Encoding: gzip, deflate" has been removed
ICInsert-Cookie 
IKInsert-Keep-aliveDespite that the client didn't request it, the optimization service inserted a "Connection: Keep-Alive" in the request
IFInflight-Cache (Stream Splitting) 

8.10.2.4. What kind of internal decisions for HTTP optimization are made?

Further information about the internals of the HTTP optimization can be enabled with the command protocol http x-debug enable.

Figure 8.43. Enabling the HTTP debugging feature

CSH (config) # protocol http x-debug enable
CSH (config) # show protocol http internal
Insert Pre-Fetch X-Header:                      yes 
Insert X-RBT-Debug header:                      yes 
[...]

In the HTTP response, this will add a new line: X-RBT-Debug.

Figure 8.44. The X-RBT-Debug line

X-RBT-Debug: 0x81,SPIP,FPSF

The first field, 0x81, is the optimization scheme and its value represents a bitmask which explains which features are enabled for the request:

Table 8.2. Optimization scheme bit mask

0x0001URL Learning
0x0002Parse and Prefetch
0x0004Object Prefetch Table
0x0008Reuse NTLM Authentication
0x0010Force Negotation NTLM
0x0020Strip authentication header
0x0040Gratuitous 401
0x0080Strip Compression
0x0100Insert Cookie
0x0200Insert keep-alive
0x010000Frontpage Extensions
0x020000WebDAV support
0x040000FSS HTTP

So for the value of 0x81 the features used are Strip Compression and URL Learning.

This value is also seen in the logs for that TCP session:

Figure 8.45. HTTP Optimization scheme

CSH sport[16611]: [http/client.INFO] 6629561 {10.0.1.1:65310 192.168.1.1:80} codec flow co \
    ntrol is disabled 
CSH sport[16611]: [http/client.INFO] 6629561 {10.0.1.1:65310 192.168.1.1:80} opt_scheme_ u \
    pdated to 129 from auto config table 

In this example, the 129 decimal is 0x81 hexadecimal, relates to the same features as seen earlier.

The next fields are the actions and decisions of the HTTP optimization code. The four letters are part of a code:

The first letter is the status:

  • S from Success

  • F from Failure

  • I from Information

The second letter is the category:

  • C from Cache

  • P from Prefetch

  • M from Miscellaneous

The third and the fourth letter are the action and specific reason codes.

So the code SPIP would be a success from the prefetching functionality and the code FPSF would be a failure for the prefetching functionality.

Table 8.3. Cache related category

FCAAFailed to cache because the content is authenticated but cache_without_auth is off
FCAEFailed to cache because the extension is not present in the cache extension list
FCAIInvalid request, e.g., absence of host header
FCALThe body is too big to cache
FCASThe remaining lifetime is too short to cache
FCAUNot allowed to cache, e.g., cache-control header
FCS2The current cache entry only has the header
FCSAFailed to serve due to the absence of Accept-Encoding header
FCSCThe Steelhead appliance is not configured to serve full body cache
FCSDFailed to serve from cache because of a different encoding
FCSEFailed to serve from cache because the extension is not present in the cache extension list
FCSIInvalid request, e.g., absence of host header
FCSMMiscellaneous reason
FCSNThe object is not present in cache
FCSUThe object is not allowed to cache, e.g., cache-control header
FCSXFailed to serve cached object because it has been expired
SCANThe object is added to cache
SCARThe object has been refreshed in the cache

Table 8.4. Prefetch related category

FPIAFailed to initiate prefetching because authentication is required. OR, no response has been received through this connection so the Steelhead appliance is not yet aware if authentication is needed or not
FPIDFailed to add the URL to the APT database
FPIEDidn't initiate prefetching because inner channel is being terminated
FPIHFailed to initiate prefetching because a header is missing (most likely cookie header)
FPIMReached the maximum prefetch limit
FPLFFailed to learn the object because APT is full
FPLHFailed to learn the object due to the absence of the header
FPLMFailed to learn the object because the Steelhead appliance is occupying its MAX MEM
FPSAAuthentication is required
FPSEAPT request was EOF'ed
FPSFAPT was not found for the object
FPSNAPT request was not sent
FPSRReceived error status code for APT request
IPGPAPT is generated via PnP
IPGUAPT is generated via URL Learning
IPLPThe object has been purged from UL APT tree
IPSCThe prefetched object is served through callback, i.e., the request was halted for the APT response to come.
SPIPThis object triggered prefetching
SPLNThe object was learned by UL
SPPPThe page is parsed for PnP

Table 8.5. Miscellaneous category

FM4SReceived a 401 despite that strip authentication took the strip header off
FMGSFailed to store the 401 response because the body is shorter than specified
IMAAExpecting an authenticaton header in the request but it was absent
IMACStopped storing the body for processing because the Steelhead appliance is in a LOW MEM state
IMBRThis is a base request
IMIRThe Steelhead appliance inserted a referer header
SMGNThe 401/407 response is newly added
SMGRThe 401/407 response cache entry is refreshed