6.4. Log files

There are various system logs included in the system dump of the Steelhead appliance:

6.4.1. System logs

The messages files (messages, messages.1.gz, messages.2.gz etc) are the system logs which can also be seen on the CLI with the command show log and show log files N.

The first line in the system logs is the version of RiOS running on the Steelhead appliance at the time the system logs get rotated.

By default the logging level of the Steelhead appliances is set to notice level, but when investigating problems info level logging might be required.

Figure 6.7. Beginning of the file "messages"

BUILD_PROD_VERSION:               rbt_sh 6.5.0a #84_26 2011-04-04 20:34:46 i386 root@misko \
    lc:svn://svn/mgmt/branches/canary_84_fix_branch
May 28 00:00:00 CSH syslogd 1.4.1: restart.
May 28 00:00:21 CSH sport[6693]: [smbsfe.NOTICE] 15446 {10.0.1.1:53286 192.168.1.1:445} Cl \
    ient and Server negotiated SMB2 protocol SMB2.Turning OFF CIFS optimizations. However  \
    SDR optimizations will continue.
May 28 00:00:53 CSH sport[6693]: [connect_pool.NOTICE] - {- -} Destroying pool to peer: 10 \
    .45.5.3
May 28 00:00:53 CSH sport[6693]: [splice/oob.NOTICE] 10 {- -} Lost OOB Splice between ladd \
    r=10.33.0.91:7800 and raddr=10.45.5.3:7800

The format of the lines logged by the optimization service is:

Figure 6.8. Format of the log messages by the optimization service

<date> <time> <hostname> sport[process id] <functionality.loglevel> <splice-id> {<client I \
    P address:client TCP port> <server IP address:server TCP port>} <message>


The date only contains the month name and the date, not the year. The date and time do not contain the time zone configured on the Steelhead appliance. The loglevel, or severity, is the importance of the message shown, it can be info, notice, warn(ing), err(or) and crit(ical). The splice-id is a sequence number of the inner channel being setup. if no splice-id can be determined it will be replaced by a "-" (dash) and the IP address and TCP ports will be replaced by a dash too.

The functionality can be the part of the "sport" optimization service:

  • splice/probe: Messages related to the auto-discovery of optimized TCP sessions.

  • io/inner/prod: Messages related to the setup of an optimized TCP session.

  • splice/oob: The Out-of-Band Splice between two Steelhead appliances.

  • connect_pool: The list of TCP sessions part of the Connection Pool.

  • mapi/*: MAPI latency optimization related messages.

  • smbcfe: CIFS client-side latency optimization related messages.

  • smbsfe: CIFS server-side latency optimization related messages.

  • http/*: HTTP latency optimization related messages.

Or from other processes on the Steelhead appliances:

  • kernel / intercept: Messages in related to the intercept kernel-module which tracks the setup of new optimized TCP sessions.

  • mgmtd: Messages related to the management daemon.

  • pm: Messages related to the process manager.

6.4.2. GUI logs

The GUI logs are logged in the files web_access.log and web_error.log, which are the normal Apache-style logging.

The file web_access.log contains the IP address the request came from, the date and time, the URL requested, the return result and the size of the payload returned.

Figure 6.9. Beginning of the file "web_access.log"

10.0.1.1 - - [30/May/2011:16:20:02 +1000] "GET /rollup-product.css?v=1001518176 HTTP/1.1"  \
    304 -
10.0.1.1 - - [30/May/2011:16:20:03 +1000] "POST /mgmt/xmldata?p=inpathRules HTTP/1.1" 200  \
    489
10.0.1.1 - - [30/May/2011:16:20:07 +1000] "GET /images/aet_edit_close.png HTTP/1.1" 200 17 \
    3
10.0.1.1 - - [30/May/2011:16:20:18 +1000] "POST /mgmt/xmldata?p=dynamicStatus HTTP/1.1" 20 \
    0 98

The file web_error.log contains the date and time and the reported issue.

Figure 6.10. Beginning of the file "web_error.log"

[Mon May 30 00:00:00 2011] [notice] Apache configured -- resuming normal operations
[Mon May 30 16:20:00 2011] [notice] Graceful restart requested, doing restart
[Mon May 30 16:20:00 2011] [notice] Apache configured -- resuming normal operations

6.4.3. Kernel log

The kernel log of the Steelhead appliance is recorded in the file dmesg. This kernel log is a circular buffer which gets populated from the moment the Steelhead appliance gets booted. If the system dump is taken soon after the reboot, it will contain the whole boot log.

Figure 6.11. Part of the file "dmesg"

Linux version 2.6.9-34.EL-rbt-7989SMP (root@miskolc.lab.nbttech.com) (gcc version 3.4.6 20 \
    060404 (Red Hat 3.4.6-10)) #2 SMP Fri Feb 4 18:54:45 PST 2011
BIOS-provided physical RAM map:
 BIOS-e820: 0000000000000000 - 000000000009fc00 (usable)
 BIOS-e820: 000000000009fc00 - 00000000000a0000 (reserved)
 BIOS-e820: 00000000000e0000 - 0000000000100000 (reserved)
 BIOS-e820: 0000000000100000 - 00000000dffd0000 (usable)

The file dmesg-boot, available since RiOS version 8.5, contains a copy of the dmesg file just after the startup of the Steelhead appliance.

6.4.4. CLI history log

The file cli_history_root contains the commands entered on the CLI by the admin user. For example, this is the history of the upgrade of the Steelhead appliance:

Figure 6.12. Part of the file "cli_history_root"

 20110524155032 0
show run
 20110524155047 0
no interface primary shutdown 
 20110524155500 0
image fetch *
 20110524160139 0
image install sh-i386-6.5.0a 
 20110524160316 0
image install sh-i386-6.5.0a 1
 20110524160347 0
image boot 1
 20110524160349 0
wr m
 20110524160349 0
reload

The numbers are the year-month-day-hour-minute-second format.

There are lines which could contain a username or password. That part of the line has been removed, for example in the image fetch command.

There is a similar file for the monitor user called cli_history_monitor.

6.4.5. Optimization service memory log

Under normal operations, the Steelhead appliance is logging under the Notice priority. But a lot of interesting information is logged in the optimization service under the Info level.

The optimization service keeps the last 16 kilobytes of the messages logged under Info level and higher. When the optimization service logs an error with a priority of Error or higher (Error, Critical or Emergency), it will write this to the file /var/log/memlog. This way important information of the optimization service in case of an error in the optimization service can be caught even if specific Info level logging hasn't been enabled.

Figure 6.13. First ten lines of the file memlog.1

[Sep 14 15:26:25 19591 659750 /splice/client INFO] {57.200.61.29:1993 57.200.2.20:8014} St \
    art flowing, lport 38856, rport 7800, OPOL_NORMAL, NAGLE_ALWAYS, PREOPT_NONE, LATOPT_A \
    UTO, TRANSPORT_ID_NONE, TPTOPT_NONE(0x0), TRPY_FULL
[Sep 14 15:26:25 19591 -1 /http/cache INFO] {- -} decode decode DONE
[Sep 14 15:26:25 19591 659750 /splice/client INFO] {57.200.61.29:1993 57.200.2.20:8014} fi \
    ni client 57.200.61.29:1993 server 57.200.2.20:8014 cfe 57.200.157.219:38856 sfe 172.1 \
    9.18.152:7800 app TCP
[Sep 14 15:26:25 19591 -1 /http/cache INFO] {- -} decode decode DONE
[Sep 14 15:26:25 19591 659751 /splice/client INFO] {57.200.61.29:1994 57.200.2.20:8014} in \
    it client 57.200.61.29:1994 server 57.200.2.20:8014 cfe 57.200.157.219:7801 sfe 172.19 \
    .18.152:7800 client connected: yes
[Sep 14 15:26:25 19591 659751 /splice/client INFO] {57.200.61.29:1994 57.200.2.20:8014} tr \
    py: TRPY_FULL, csum 0 local: 57.200.61.29:1994 remote: 57.200.2.20:8014
[Sep 14 15:26:25 19591 -1 /http/cache INFO] {- -} decode decode DONE
[Sep 14 15:26:25 19591 -1 /http/cache INFO] {- -} decode decode DONE
[Sep 14 15:26:25 19591 -1 /http/cache INFO] {- -} decode decode DONE
[Sep 14 15:26:25 19591 659751 /splice/client INFO] {57.200.61.29:1994 57.200.2.20:8014} Sp \
    lice client side initializing: No protocol port = 8014 protocol id = TCP(0) transport  \
    = TRANSPORT_ID_NONE

6.4.6. Log file management

6.4.6.1. Period based log file rotation

By default the system logs are rotated once every day at midnight:

Figure 6.14. Output of the "show logging" command, daily rotation

SH # show logging
Local logging level: notice
Default remote logging level: notice
Number of archived log files to keep: 10
Log rotation frequency: daily

However, every twenty minutes the Steelhead appliance will check if the file size of the current log file is more than 1 Gb. If it is more than 1 Gb, it will force a rotation of it.

6.4.6.2. Size based log file rotation

The log file rotation can be set to a size based rotation scheme, for example when the 20 minute check interval is too small.

Figure 6.15. Output of the "show logging" command, rotation by size

SH (config) # logging files rotation criteria size 1000
SH (config) # show logging
Local logging level: notice
Default remote logging level: notice
Number of archived log files to keep: 10
Log rotation size threshold: 1000 megabytes
SH (config) # logging files rotation criteria frequency daily

6.4.6.3. Manual log file rotation

To force a log file rotation, use the command logging files rotation force.

To manually remove a set of log files, use the command logging files delete oldest.

Figure 6.16. Manually removing log files

SH # logging files delete oldest ?
<cr>             Delete the single oldest log file
<number>         Select the number of oldest log files to delete
SH # logging files delete oldest 3
SH #

You can download previous system logs via the GUI under Reports -> Diagnostics -> System Logs Download.