As described earlier, the Steelhead appliance and the optimization service have several hardware watchdogs and watchdog processes:
The hardware watchdog, which will reboot the appliance if the user-land process wdt has not been communicated with the hardware watchdog for 30 seconds. This prevents the Steelhead appliance to hang indefinitely and gives it a chance to recover.
This process communicates with the hardware watchdog every six seconds. If it is too late, the following line will be printed:
Figure 5.36. The wdt process is too slow
SH wdt[11472]: [wdt.WARNING]: [lan1_0] watchdog poll late: 6265 with interval 6000
The most likely cause is a problem with the hard disks.
The network watchdog, which will put the by-pass card into fail-to-wire or fail-to-block mode if the optimization service hasn't communicated with the network watchdog for 7 seconds. This will remove the Steelhead appliance from the path in the network and the traffic flows again.
The optimization service communicates with the network watchdog every second. If it is too late, the following lines will be printed:
Figure 5.37. The optimization service is too slow
SH sport[9367]: [ErWatchdog.NOTICE] - {- -} ether-relay watchdog poll late: 3491 period 10 \ 00 SH sport[9367]: [null_watchdog.WARN] - {- -} null watchdog poll late: 3127 period 1000
The most likely cause is a problem inside the optimization service and a case with Riverbed TAC should be opened.
The software watchdog, a watchdog in the optimization service which checks if the threads inside the optimization service are still responsive enough or that they are blocked.
Figure 5.38. The optimization service has hung threads
SH sport[5348]: [eventthread/watch/worker/4.WARN] - {- -} watcher: One or more threads not \ responding after at least 20s; unhealthy threads follow SH sport[5348]: [eventthread/watch/worker/4.WARN] - {- -} watcher: EventThread(main)[LW \ P 5348] 0x1c1a000 is not healthy
The most likely cause is a problem inside the optimization service and a case with Riverbed TAC should be opened.