5.10. Time related issues - The NTP service.

Having all your network devices operate on a synchronized time is essential for reporting and troubleshooting. Further, to achieve Active Directory integration, the clock on the Steelhead appliance needs to be in sync with the clock on the Domain Controllers.

By default the Steelhead appliance has the NTP service enabled against a couple of NTP clocks on the public Internet. If the Steelhead appliances do not have access to the public Internet and the network does not have an internal NTP server, then the Steelhead appliances should sync to the clock on the Domain Controllers.

5.10.1. Configuration of the NTP service

In the configuration of the NTP service you specify a number of NTP servers.

Note that using the hostnames from the riverbed.pool.ntp.org pool of NTP servers returns geographically local IP addresses which might not match the IP addresses seen on other return statuses.

Figure 5.52. The standard NTP service configuration

SH # show running
[...]
no ntp disable
   ntp server ftp.riverbed.com enable
   ntp server ftp.riverbed.com version "4"
   ntp server 0.riverbed.pool.ntp.org enable
   ntp server 0.riverbed.pool.ntp.org version "4"
   ntp server 1.riverbed.pool.ntp.org enable
   ntp server 1.riverbed.pool.ntp.org version "4"
   ntp server 2.riverbed.pool.ntp.org enable
   ntp server 2.riverbed.pool.ntp.org version "4"
   ntp server 3.riverbed.pool.ntp.org enable
   ntp server 3.riverbed.pool.ntp.org version "4"
[...]

SH # show ntp
NTP enabled: yes
No NTP peers configured.
NTP server: 208.70.196.25 (version 4)   Enabled: yes
NTP server: 121.0.0.42 (version 4)      Enabled: yes
NTP server: 202.60.94.11 (version 4)    Enabled: yes
NTP server: 203.23.237.200 (version 4)  Enabled: yes
NTP server: 203.192.179.98 (version 4)  Enabled: yes

5.10.2. Determining the status of the NTP service

In RiOS version 6.5 and later the status of the NTP service can be seen with the command show ntp status:

Figure 5.53. Status of the NTP service with the command "show ntp status"

SH # show ntp status
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 ntp.tourism.wa. 130.95.179.80    2 u   13   64    1  1702.52  667.275   0.002
 ns.creativecont .STEP.          16 u  985   64    0    0.000    0.000   0.002
+cachens2.onqnet 209.81.9.7       2 u    6   64    1  772.591  218.753   0.002
+ds001.hostingsh 203.12.160.2     3 u  173   64  374  105.205   40.096  35.771
+wireless.org.au 17.83.253.7      3 u  173   64  374  111.738   12.724  35.467
*ftp.riverbed.co 203.161.12.165   3 u  172   64  374  114.232   -6.295  56.603

The NTP service determines the best NTP server based on the stratum (number of hops towards a reference clock), the delay (the network delay towards the server), the offset between the received time of the NTP server and the clock on the local machine and the network jitter.

A stratum of 16 means that the NTP server is unsynchronized.

Instead of IP addresses, the refid can also contain the name of the clock source, like .GPS. and .CDMA..

The character in front of the remote clock explains how the NTP service feels about it: A * means that this NTP server is the chosen peer to sync from, a + means that this NTP server is a backup in case the chosen peer becomes unreachable.

5.10.3. Troubleshooting

5.10.3.1. Expected behaviour

Use the tcpdump tool to see if the NTP service is talking to the NTP servers and if they answer back. NTP traffic is UDP based and on both port 123 for the client and the server.

Figure 5.54. Tcpdump trace for NTP traffic

SH # tcpdump -ni primary port ntp 
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on primary, link-type EN10MB (Ethernet), capture size 96 bytes
08:43:03.149487 IP 119.12.133.239.123 > 202.158.218.239.123: NTPv4, Client, length 48
08:43:05.721339 IP 202.158.218.239.123 > 119.12.133.239.123: NTPv4, Server, length 48
08:43:14.632504 IP 119.12.133.239.123 > 192.189.54.17.123: NTPv4, Client, length 48
08:43:14.785459 IP 192.189.54.17.123 > 119.12.133.239.123: NTPv4, Server, length 48
08:43:15.631914 IP 119.12.133.239.123 > 119.148.74.66.123: NTPv4, Client, length 48
08:43:16.632304 IP 119.12.133.239.123 > 119.63.208.27.123: NTPv4, Client, length 48
08:43:16.632433 IP 119.12.133.239.123 > 192.189.54.17.123: NTPv4, Client, length 48
08:43:17.632670 IP 119.12.133.239.123 > 119.148.74.66.123: NTPv4, Client, length 48
08:43:18.440873 IP 119.63.208.27.123 > 119.12.133.239.123: NTPv4, Server, length 48
08:43:18.441874 IP 192.189.54.17.123 > 119.12.133.239.123: NTPv4, Server, length 48
08:43:18.441971 IP 119.148.74.66.123 > 119.12.133.239.123: NTPv4, Server, length 49

Both client and server traffic is seen here, which means that the NTP server is reachable and the Steelhead appliance is allowed to talk to its NTP service.

5.10.3.2. No answer from the NTP service

If the NTP server doesn't exist anymore or is configured to not respond to the queries, no answers will be returned:

Figure 5.55. Tcpdump trace for NTP traffic

SH # tcpdump -ni primary port ntp 
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on primary, link-type EN10MB (Ethernet), capture size 96 bytes
08:43:03.149487 IP 119.12.133.239.123 > 202.158.218.239.123: NTPv4, Client, length 48
08:43:18.494871 IP 119.12.133.239.123 > 202.158.218.239.123: NTPv4, Client, length 48
08:43:33.948712 IP 119.12.133.239.123 > 202.158.218.239.123: NTPv4, Client, length 48
08:43:48.487123 IP 119.12.133.239.123 > 202.158.218.239.123: NTPv4, Client, length 48

Only client traffic is seen here, which means that the NTP server is either unreachable or that the Steelhead appliance is not allowed to talk to this NTP server.

5.10.3.3. Time offset is too big

If the offset between the reference clock and the system clock is too big, more than 30 minutes, then the NTP service will refuse to synchronize and exit.

To overcome this problem, either disable the NTP service, set the clock of the Steelhead appliance with the command clock set '<yyyy/mm/dd hh:mm:ss>' to a value close to the reference clock and then enable the NTP service again. Or disable the NTP service, issue the command ntpdate <NTP server> and restart the NTP service.

Figure 5.56. Manual adjusting the date/time with the command "ntpdate"

SH (config) # show clock
Time: 12:12:20
Date: 2012/12/12
Zone: Australia Sydney
No active ntp peers
SH (config) # show ntp active-peers
No active ntp peers
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 static-96-226-1 139.78.135.14    2 u    2   64    1  209.560  3923152   0.000
 xen1.rack911.co 209.51.161.238   2 u    2   64    1  177.707  3923152   0.000
 lithium.constan 128.4.1.1        2 u    8   64    1  224.869  3923152   0.000
 240.140.8.72.in 64.147.116.229   2 u    7   64    1  172.970  3923152   0.000

     remote       conf  auth  key  
===================================
 static-96-226-1  yes   none  none 
 xen1.rack911.co  yes   none  none 
 lithium.constan  yes   none  none 
 240.140.8.72.in  yes   none  none 

SH (config) # no ntp enable 
SH (config) # ntpdate 0.riverbed.pool.ntp.org
26 Jan 21:59:13 ntpdate[12761]: step time server 208.68.36.196 offset 3923152.813162 sec
SH (config) # show clock
Time: 21:59:16
Date: 2013/01/26
Zone: Australia Sydney
No active ntp peers
SH (config) # ntp enable 
SH (config) # show ntp active-peers
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
-shed.galexander 204.163.51.41    3 u    9   64   37  168.203   -3.754   0.433
+xen1.rack911.co 209.51.161.238   2 u    4   64   37  177.716    5.831   0.301
+ec2-50-16-231-1 209.51.161.238   2 u    9   64   37  218.954    3.933   0.577
*lttleman.deekay 130.207.244.240  2 u    9   64   37  223.387    6.205   1.380

     remote       conf  auth  key  
===================================
 shed.galexander  yes   none  none 
 xen1.rack911.co  yes   none  none 
 ec2-50-16-231-1  yes   none  none 
 lttleman.deekay  yes   none  none 

At the beginning the offset is nearly 4 million seconds, or 45 days. After running the command ntpdate 0.riverbed.pool.ntp.org the clock is correct again and after a couple of minutes the NTP service shows up as synchronized again.