3.3. Tcpdump, the network packet capture tool

Being able to see the traffic that goes in and out a network card is great value to determine what occurs on network level. Tcpdump, the standard network packet capture tool, is included on the Steelhead appliances. It uses the pcap library for the capturing of the packets on the NIC.

The place where tcpdump captures the data from the network stack is just before the sending or just after the receiving of the data to and from the network card. This makes sure that the capture happens with the data as closely as it is on the network cable as possible.

Tcpdump supports both IPv4 and IPv6 addresses.

3.3.1. Tcpdump options

  • The option -n will prevent DNS name resolution of the IP addresses and services detected in the captured packet.

  • The option -s specifies the number of bytes to read from the captured packets.

  • The option -i will select the interface to capture on.

  • The option -c specifies the limit on the number of packets captured.

  • The option -w specifies the filename to write the capture to.

  • The option -v will print how many packets are captured and written to disk.

  • The option -le will print the link layer information of the captured packet.

  • The option -X (capital X) will also print contents of the captured packet in hex and character dump format.

3.3.2. Usage hints for tcpdump

  • Always use the -n option to disable name resolution otherwise it will slow down the display of packets and possible dropping of captured packets.

  • Don't capture on the in-path interfaces, always capture on the corresponding LAN and WAN interfaces.

  • Don't capture on the primary interface, always capture on the prihw interface.

  • Use -c to limit the number of packets captured to prevent them from scrolling of the screen.

  • When writing captured packets to disk, use the -v option to see the number of packets written.

  • Use correct filters to capture only the packets you are interested in.

  • The default filter is based on the standard Ethernet frame headers. Use the vlan primitive to switch to the 802.1Q VLAN Ethernet frame format.

  • Use a snaplength of 1600 for a full payload capture.

3.3.3. Auto-discovery and Full transparency / Port Transparency options

The normal tcpdump program will not be able to decode the Riverbed TCP Options and will display them as a hex-string:

Figure 3.6. The Riverbed specific TCP options not decoded in tcpdump before RiOS 6.1.4 and 6.5.2

11:12:47.668647 IP 10.0.1.1.52175 > 192.168.1.1.80: Flags [S], seq 4101063156, win 65535,  \
    options [mss 1460,nop,wscale 3,nop,nop,TS val 693134564 ecr 0,sackOK,nop,nop,opt-76:01 \
    010a0001060005,opt-76:0c01,nop,eol], length 0
11:12:47.801418 IP 192.168.1.1.80 > 10.0.1.1.52175: Flags [S.], seq 20020520, ack 41010631 \
    57, win 65535, options [mss 1460,nop,wscale 3,nop,nop,TS val 693134564 ecr 0,sackOK,no \
    p,nop,opt-76:0c01,nop,nop,nop,eol], length 0
11:12:47.801454 IP 192.168.1.1.80 > 10.0.1.1.52175: Flags [S.], seq 20020520, ack 41010631 \
    57, win 65535, options [mss 1460,nop,wscale 3,TS val 693134564 ecr 0,sackOK,opt-76:111 \
    10a000106c0a801061e78,opt-76:0e3d,nop,eol], length 0

With the release of RiOS 6.1.4 and 6.5.2, the tcpdump program included on the Steelhead appliances will decode the auto-discovery and the WAN Visibility Transparency TCP options:

Figure 3.7. The Riverbed specific TCP options for auto-discovery decoded in tcpdump after RiOS 6.1.4 and 6.5.2

11:12:47.668647 IP 10.0.1.1.52175 > 192.168.1.1.80: Flags [S], seq 4101063156, win 65535,  \
    options [mss 1460,nop,wscale 3,nop,nop,TS val 693134564 ecr 0,sackOK,nop,nop,rvbd-prob \
    e AD CSH:10.0.1.6 01010a0001060005,rvbd-probe EAD 0c01,nop,eol], length 0
11:12:47.801418 IP 192.168.1.1.80 > 10.0.1.1.52175: Flags [S.], seq 20020520, ack 41010631 \
    57, win 65535, options [mss 1460,nop,wscale 3,nop,nop,TS val 693134564 ecr 0,sackOK,no \
    p,nop,rvbd-probe EAD 0c01,nop,nop,nop,eol], length 0
11:12:47.801454 IP 192.168.1.1.80 > 10.0.1.1.52175: Flags [S.], seq 20020520, ack 41010631 \
    57, win 65535, options [mss 1460,nop,wscale 3,TS val 693134564 ecr 0,sackOK,rvbd-probe \
     AD CSH:10.0.1.6 SSH:192.168.1.6:7800 11110a000106c0a801061e78,rvbd-probe EAD 0e3d,nop \
    ,eol], length 0

For the Auto-Discovery process, it will show the IP addresses of the CSH (Client-side Steelhead appliance) and the SSH (Server-side Steelhead appliance).

  • The keyword rvbd-probe tells that the option is TCP option 76 for the auto-discovery process.

  • The keyword AD tells that the option is part of the auto-discovery process.

  • The keyword EAD tells that the option is part of the enhanced auto-discovery process.

  • The keyword CSH gives the IP address of the client-side Steelhead appliance.

  • The keyword SSH gives the IP address and TCP port of the server-side Steelhead appliance.

For the WAN Visibility Transparency feature, it will show the IP addresses and TCP port numbers of the inner channel.

Figure 3.8. The Riverbed specific TCP options for WAN Visibility Transparency feature decoded in tcpdump after RiOS 6.1.4 and 6.5.2

12:02:03.274738 IP 10.0.1.1.3789 > 192.168.1.1.80: Flags [P.], seq 1:11, ack 7, win 4592,  \
    options [nop,nop,TS val 958790087 ecr 1176840218,rvbd-trans Transp sSH:10.0.1.6:42455  \
    dSH:192.168.1.1:7800 00000a000106c0a80106a5d71e78], length 10
12:02:03.288821 IP 192.168.1.1.80 > 10.0.1.1.3789: Flags [.], seq 7, ack 11, win 23, optio \
    ns [nop,nop,TS val 1176840247 ecr 958790087,rvbd-trans Transp sSH:192.168.1.6:7800 dSH \
    :10.0.1.6:42455 0000c0a801060a0001061e78a5d7], length 0

  • The keyword rvbd-trans tells that the option is TCP option 78 for the WAN Visibility Full Transparency feature.

  • The keyword Transp tells that the standard transparency options are used.

  • The keyword sSH gives the IP address and TCP port of the sending Steelhead appliance.

  • The keyword dSH gives the IP address and TCP port of the destination Steelhead appliance.

With the release of Wireshark version 1.6.0, it can decode the auto-discovery and the transparency TCP options.

Figure 3.9. The Riverbed specific TCP options decoded in tshark

$ tshark -O tcp -V -nr autodiscovery.cap
Frame 1: 94 bytes on wire (752 bits), 94 bytes captured (752 bits)
Ethernet II, Src: d4:9a:20:c2:52:0e (d4:9a:20:c2:52:0e), Dst: 00:0d:b9:17:28:de (00:0d:b9: \
    17:28:de)
Internet Protocol Version 4, Src: 10.0.1.1 (10.0.1.1), Dst: 192.168.1.1 (192.168.1.1)
Transmission Control Protocol, Src Port: 52175 (52175), Dst Port: 80 (80), Seq: 0, Len: 0
    Source port: 52175 (52175)
    Destination port: 80 (80)
    [Stream index: 0]
    Sequence number: 0    (relative sequence number)
    Header length: 60 bytes
    Flags: 0x02 (SYN)
    Window size value: 65535
    [Calculated window size: 65535]
    Checksum: 0xe76d [validation disabled]
    Options: (40 bytes)
        Maximum segment size: 1460 bytes
        No-Operation (NOP)
        Window scale: 3 (multiply by 8)
        No-Operation (NOP)
        No-Operation (NOP)
        Timestamps: TSval 693134564, TSecr 0
        TCP SACK Permitted Option: True
        No-Operation (NOP)
        No-Operation (NOP)
        Riverbed Probe: Probe Query, CSH IP: 10.0.1.6
            Length: 10
            0000 .... = Type: 0
            .... 0001 = Version: 1
            Reserved
            CSH IP: 10.0.1.6 (10.0.1.6)
            Application Version: 5
        Riverbed Probe: Probe Query Info
            Length: 4
            0000 110. = Type: 6
            .... ...0 = Version: 2
            Probe Flags: 0x01
                .... .0.. = Not CFE: Not set
                .... ...1 = Last Notify: Set
        No-Operation (NOP)
        End of Option List (EOL)

Frame 2: 86 bytes on wire (688 bits), 86 bytes captured (688 bits)
Ethernet II, Src: 00:0d:b9:17:28:de (00:0d:b9:17:28:de), Dst: d4:9a:20:c2:52:0e (d4:9a:20: \
    c2:52:0e)
Internet Protocol Version 4, Src: 192.168.1.1 (192.168.1.1), Dst: 10.0.1.1 (10.0.1.1)
Transmission Control Protocol, Src Port: 80 (80), Dst Port: 52175 (52175), Seq: 0, Ack: 1, \
     Len: 0
    Source port: 80 (80)
    Destination port: 52175 (52175)
    [Stream index: 0]
    Sequence number: 0    (relative sequence number)
    Acknowledgement number: 1    (relative ack number)
    Header length: 52 bytes
    Flags: 0x12 (SYN, ACK)
    Window size value: 65535
    [Calculated window size: 65535]
    Checksum: 0xe020 [validation disabled]
    Options: (32 bytes)
        Maximum segment size: 1460 bytes
        No-Operation (NOP)
        Window scale: 3 (multiply by 8)
        No-Operation (NOP)
        No-Operation (NOP)
        Timestamps: TSval 693134564, TSecr 0
        TCP SACK Permitted Option: True
        No-Operation (NOP)
        No-Operation (NOP)
        Riverbed Probe: Probe Query Info
            Length: 4
            0000 110. = Type: 6
            .... ...0 = Version: 2
            Probe Flags: 0x01
                .... .0.. = Not CFE: Not set
                .... ...1 = Last Notify: Set
        No-Operation (NOP)
        No-Operation (NOP)
        No-Operation (NOP)
        End of Option List (EOL)
    [SEQ/ACK analysis]
        [This is an ACK to the segment in frame: 1]
        [The RTT to ACK the segment was: 0.132771000 seconds]

Frame 3: 94 bytes on wire (752 bits), 94 bytes captured (752 bits)
Ethernet II, Src: 00:0d:b9:17:28:de (00:0d:b9:17:28:de), Dst: d4:9a:20:c2:52:0e (d4:9a:20: \
    c2:52:0e)
Internet Protocol Version 4, Src: 192.168.1.1 (192.168.1.1), Dst: 10.0.1.1 (10.0.1.1)
Transmission Control Protocol, Src Port: 80 (80), Dst Port: 52175 (52175), Seq: 0, Ack: 1, \
     Len: 0
    Source port: 80 (80)
    Destination port: 52175 (52175)
    [Stream index: 0]
    Sequence number: 0    (relative sequence number)
    Acknowledgement number: 1    (relative ack number)
    Header length: 60 bytes
    Flags: 0x12 (SYN, ACK)
    Window size value: 65535
    [Calculated window size: 65535]
    Checksum: 0x7893 [validation disabled]
    Options: (40 bytes)
        Maximum segment size: 1460 bytes
        No-Operation (NOP)
        Window scale: 3 (multiply by 8)
        Timestamps: TSval 693134564, TSecr 0
        TCP SACK Permitted Option: True
        Riverbed Probe: Probe Response, Server Steelhead: 192.168.1.6:7800
            Length: 14
            0001 .... = Type: 1
            .... 0001 = Version: 1
            Reserved
            CSH IP: 10.0.1.6 (10.0.1.6)
            SSH IP: 192.168.1.6 (192.168.1.6)
            SSH Port: 7800
        Riverbed Probe: Probe Response Info
            Length: 4
            0000 111. = Type: 7
            .... ...0 = Version: 2
            Probe Flags: 0x3d
                ...1 .... = Disable Probe Cache on CSH: Set
                .... ..0. = SSL Enabled: Not set
                .... ...1 = SSH outer to server established: Set
        No-Operation (NOP)
        End of Option List (EOL)

Figure 3.10. The Riverbed specific TCP options decoded in Wireshark

The Riverbed specific TCP options decoded in Wireshark

3.3.4. Pcap filters

There are several basic pcap primitives to be understood before tcpdump can be used effectively:

  • The primitive host <IP address> filters the packets to match this source or destination IP address.

  • The primitive net <IP subnet> filters the packets to match this source or destination IP subnet.

  • The primitive port <port number> filters the packets to match this source or destination TCP or UDP port numbers.

  • The primitive ip and ip6 filters explicitly for IPv4 and IPv6 addresses.

  • The primitive icmp filters for only ICMP packets, the primitive tcp filters for only TCP packets, the primitive udp filters for only UDP packets.

  • The primitive vlan is not so much a filter but more a change in the offset in the filter: From this part of the filter, the offset for the IP packet is indexed to the size of a 802.1Q VLAN tagged Ethernet frame.

The various primitives can be merged with a boolean "and" and "or". But keep in mind that, unlike boolean logic, the "and" and "or" have the same precedence. The precedence can be changed by using round brackets around a part of the filter.

For example, if the filter needs to capture all traffic from host H1 on port P1 or from host H2 on port P2, then host H1 and port P1 or host H2 and port P2 gets interpreted as ((((host H1) and port P1) or host H2) and port P2). However, what was needed was: (host H1 and port P1) or (host H2 and port P2).

Always use round brackets to prioritize parts of the filter!

3.3.5. Examples of tcpdump scenarios

3.3.5.1. Display the first ten packets from or to one host

The first example is capturing the first 10 packets to host 10.0.1.1.

Figure 3.11. Display the first ten packets from and to a certain host

SH # tcpdump -ni lan0_0 -c 10 'host 10.0.1.1'
tcpdump: WARNING: lan0_0: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on lan0_0, link-type EN10MB (Ethernet), capture size 96 bytes
12:07:07.298829 ARP, Request who-has 10.0.1.6 tell 10.0.1.1, length 46
12:07:07.298872 ARP, Reply 10.0.1.6 is-at 00:0e:b6:8c:74:9c, length 28
12:07:08.194021 IP 10.0.1.1.52902 > 202.83.176.1.53: 53837 [1au] A? www.mavetju.org. (44)
12:07:08.234151 IP 202.83.176.1.53 > 10.0.1.1.52902: 53837*- 1/2/3 A 192.168.1.1 (141)
12:07:09.464621 IP 10.0.1.1.42825 > 192.168.1.1.80: Flags [S], seq 3872240793, win 65535,  \
    options [mss 1460,nop,wscale 6,sackOK,TS val 5351449 ecr 0], length 0
12:07:09.598157 IP 192.168.1.1.80 > 10.0.1.1.42825: Flags [S.], seq 2971872340, ack 387224 \
    0794, win 5792, options [mss 1460,sackOK,TS val 285084142 ecr 5351449,nop,wscale 2], l \
    ength 0
12:07:09.602342 IP 10.0.1.1.42825 > 192.168.1.1.80: Flags [.], seq 1, ack 1, win 1040, opt \
    ions [nop,nop,TS val 5351588 ecr 285084142], length 0
12:07:09.602375 IP 10.0.1.1.42825 > 192.168.1.1.80: Flags [P.], seq 1:188, ack 1, win 1040 \
    , options [nop,nop,TS val 5351589 ecr 285084142], length 187
12:07:09.602393 IP 192.168.1.1.80 > 10.0.1.1.42825: Flags [.], seq 1, ack 188, win 1716, o \
    ptions [nop,nop,TS val 285084146 ecr 5351589], length 0
12:07:09.875867 IP 192.168.1.1.80 > 10.0.1.1.42825: Flags [P.], seq 1:1151, ack 188, win 1 \
    716, options [nop,nop,TS val 285084420 ecr 5351589], length 1150
10 packets captured
14 packets received by filter
0 packets dropped by kernel

In the example above, 14 packets were captured from the network and 10 matched the filter. Zero packets could not be processed in time by the pcap filter.

The packets are: an Ethernet broadcast ARP request and reply, a DNS request and answer and then a TCP SYN, SYN/ACK and ACK to setup the TCP session and some data towards a web server.

3.3.5.2. Display the first ten packets between two hosts

For troubleshooting, capturing traffic between two hosts is required. Here we filter between the hosts 10.0.1.1 and 192.168.1.1.

Figure 3.12. Display the first ten packets between hosts 10.0.1.1 and 192.168.1.1

SH # tcpdump -ni lan0_0 -c 10 'host 10.0.1.1 and host 192.168.1.1'
tcpdump: WARNING: lan0_0: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on lan0_0, link-type EN10MB (Ethernet), capture size 96 bytes
12:07:09.464621 IP 10.0.1.1.42825 > 192.168.1.1.80: Flags [S], seq 3872240793, win 65535,  \
    options [mss 1460,nop,wscale 6,sackOK,TS val 5351449 ecr 0], length 0
12:07:09.598157 IP 192.168.1.1.80 > 10.0.1.1.42825: Flags [S.], seq 2971872340, ack 387224 \
    0794, win 5792, options [mss 1460,sackOK,TS val 285084142 ecr 5351449,nop,wscale 2], l \
    ength 0
12:07:09.602342 IP 10.0.1.1.42825 > 192.168.1.1.80: Flags [.], seq 1, ack 1, win 1040, opt \
    ions [nop,nop,TS val 5351588 ecr 285084142], length 0
12:07:09.602375 IP 10.0.1.1.42825 > 192.168.1.1.80: Flags [P.], seq 1:188, ack 1, win 1040 \
    , options [nop,nop,TS val 5351589 ecr 285084142], length 187
12:07:09.602393 IP 192.168.1.1.80 > 10.0.1.1.42825: Flags [.], seq 1, ack 188, win 1716, o \
    ptions [nop,nop,TS val 285084146 ecr 5351589], length 0
12:07:09.875867 IP 192.168.1.1.80 > 10.0.1.1.42825: Flags [P.], seq 1:1151, ack 188, win 1 \
    716, options [nop,nop,TS val 285084420 ecr 5351589], length 1150
12:07:09.979743 IP 10.0.1.100.42825 > 192.168.1.6.80: Flags [.], seq 188, ack 1151, win 10 \
    40, options [nop,nop,TS val 5351967 ecr 285084420], length 0
12:07:10.881288 IP 10.0.1.100.42825 > 192.168.1.6.80: Flags [P.], seq 188:420, ack 1151, w \
    in 1040, options [nop,nop,TS val 5352869 ecr 285084420], length 232
12:07:10.881392 IP 192.168.1.6.80 > 10.0.1.100.42825: Flags [.], seq 1151, ack 420, win 17 \
    16, options [nop,nop,TS val 285085425 ecr 5352869], length 0
12:07:11.106812 IP 192.168.1.6.80 > 10.0.1.100.42825: Flags [P.], seq 1151:1663, ack 420,  \
    win 1716, options [nop,nop,TS val 285085651 ecr 5352869], length 512
10 packets captured
18 packets received by filter
0 packets dropped by kernel

The packets shown are the setup of a HTTP TCP session and the exchange of the data.

3.3.5.3. Display packets on a VLAN trunk

The difference in length of a normal IEEE 802.3 Ethernet frame and a IEEE 802.1q VLAN Ethernet frame is four bytes:

Figure 3.13. A normal 802.3 Ethernet frame

A normal 802.3 Ethernet frame

Figure 3.14. An 802.1Q VLAN Ethernet frame

An 802.1Q VLAN Ethernet frame

The implementation of pcap capture library is not capable to automatically determine the size difference of the two frame types and it will assume the size of a normal 802.3 Ethernet frame. This means that the offset for the IP header, and thus the higher level protocols on top of IP, for 802.1q frames is incorrect and the filtering of packets on the specified fields will fail. To overcome this, the primitive "vlan" can be used, however it will cause the rest of the filter to assume that the Ethernet frame size is the size of a 802.1q Ethernet frame.

To filter for host 10.0.1.1 for normal 802.3 Ethernet frames: host 10.0.1.1. To filter for host 10.0.1.1 for 802.1q Ethernet frames: (vlan and host 10.0.1.1). To filter for host 10.0.1.1 for normal 802.3 and 802.1q Ethernet frames: host 10.0.1.1 or (vlan and host 10.0.1.1).

The default output of tcpdump doesn't show the VLAN tag ID of the 802.1q Ethernet frames, to show then use the option -le to display them.

Figure 3.15. Display the first ten packets between host 10.0.1.1 and host 192.168.1.1

SH # tcpdump -le -ni lan0_0 -c 10 'host 10.0.1.1 and host 192.168.1.1 or (vlan and host 10 \
    .0.1.1 and host 192.168.1.1)''
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on lan0_0, link-type EN10MB (Ethernet), capture size 96 bytes
12:11:36.684544 00:21:70:3e:6b:7f > 00:0d:b9:17:28:de, ethertype 802.1Q (0x8100), length 9 \
    4: vlan 200, p 0, ethertype IPv4, IP 10.0.1.1.60035 > 192.168.1.1.https: S 1139103120: \
    1139103120(0) win 5840 <mss 1460,sackOK,timestamp 1864443320 0,nop,wscale 2>
12:11:36.685159 00:0d:b9:17:28:de > 00:21:70:3e:6b:7f, ethertype 802.1Q (0x8100), length 8 \
    2: vlan 200, p 0, ethertype IPv4, IP 192.168.1.1.https > 10.0.1.1.60035: S 2827780182: \
    2827780182(0) ack 1139103121 win 4380 <mss 1460,nop,wscale 0,nop,nop,timestamp 7927005 \
    29 1864443320,sackOK,eol>
12:11:36.685203 00:21:70:3e:6b:7f > 00:0d:b9:17:28:de, ethertype 802.1Q (0x8100), length 7 \
    0: vlan 200, p 0, ethertype IPv4, IP 10.0.1.1.60035 > 192.168.1.1.https: . ack 1 win 1 \
    460 <nop,nop,timestamp 1864443320 792700529>
12:11:37.025716 00:21:70:3e:6b:7f > 00:0d:b9:17:28:de, ethertype 802.1Q (0x8100), length 1 \
    36: vlan 200, p 0, ethertype IPv4, IP 10.0.1.1.60035 > 192.168.1.1.https: P 1:67(66) a \
    ck 1 win 1460 <nop,nop,timestamp 1864443661 792700529>
12:11:37.026275 00:0d:b9:17:28:de > 00:21:70:3e:6b:7f, ethertype 802.1Q (0x8100), length 1 \
    518: vlan 200, p 0, ethertype IPv4, IP 192.168.1.1.https > 10.0.1.1.60035: . 1:1449(14 \
    48) ack 67 win 4380 <nop,nop,timestamp 792700870 1864443661>
12:11:37.026284 00:0d:b9:17:28:de > 00:21:70:3e:6b:7f, ethertype 802.1Q (0x8100), length 1 \
    518: vlan 200, p 0, ethertype IPv4, IP 192.168.1.1.https > 10.0.1.1.60035: . 1449:2897 \
    (1448) ack 67 win 4380 <nop,nop,timestamp 792700870 1864443661>
12:11:37.026288 00:0d:b9:17:28:de > 00:21:70:3e:6b:7f, ethertype 802.1Q (0x8100), length 1 \
    518: vlan 200, p 0, ethertype IPv4, IP 192.168.1.1.https > 10.0.1.1.60035: P 2897:4345 \
    (1448) ack 67 win 4380 <nop,nop,timestamp 792700870 1864443661>
12:11:37.026314 00:21:70:3e:6b:7f > 00:0d:b9:17:28:de, ethertype 802.1Q (0x8100), length 7 \
    0: vlan 200, p 0, ethertype IPv4, IP 10.0.1.1.60035 > 192.168.1.1.https: . ack 1449 wi \
    n 2184 <nop,nop,timestamp 1864443661 792700870>
12:11:37.026327 00:21:70:3e:6b:7f > 00:0d:b9:17:28:de, ethertype 802.1Q (0x8100), length 7 \
    0: vlan 200, p 0, ethertype IPv4, IP 10.0.1.1.60035 > 192.168.1.1.https: . ack 2897 wi \
    n 2908 <nop,nop,timestamp 1864443661 792700870>
12:11:37.026336 00:21:70:3e:6b:7f > 00:0d:b9:17:28:de, ethertype 802.1Q (0x8100), length 7 \
    0: vlan 200, p 0, ethertype IPv4, IP 10.0.1.1.60035 > 192.168.1.1.https: . ack 4345 wi \
    n 3632 <nop,nop,timestamp 1864443661 792700870>
10 packets captured
19 packets received by filter
0 packets dropped by kernel

3.3.5.4. Display all ICMP traffic from and to a certain host

This can be used to determine the forward and backward path the traffic takes by pinging from the client.

Figure 3.16. Capture all traffic for all ICMP traffic from and to host 10.0.1.1

SH # tcpdump -ni lan0_0 -c 10 'host 10.0.1.1 and icmp'
tcpdump: WARNING: lan0_0: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on lan0_0, link-type EN10MB (Ethernet), capture size 96 bytes
10:41:36.717861 IP 10.0.1.1 > 192.168.1.1: ICMP echo request, id 2722, seq 0, length 64
10:41:36.851611 IP 192.168.1.1 > 10.0.1.1: ICMP echo reply, id 2722, seq 0, length 64
10:41:37.714418 IP 10.0.1.1 > 192.168.1.1: ICMP echo request, id 2722, seq 1, length 64
10:41:37.847110 IP 192.168.1.1 > 10.0.1.1: ICMP echo reply, id 2722, seq 1, length 64
10:41:44.120470 IP 10.0.1.1 > 192.168.1.1: ICMP echo request, id 2978, seq 0, length 64
10:41:44.253989 IP 192.168.1.1 > 10.0.1.1: ICMP echo reply, id 2978, seq 0, length 64
10:41:45.121247 IP 10.0.1.1 > 192.168.1.1: ICMP echo request, id 2978, seq 1, length 64
10:41:45.254498 IP 192.168.1.1 > 10.0.1.1: ICMP echo reply, id 2978, seq 1, length 64
10:41:46.122364 IP 10.0.1.1 > 192.168.1.1: ICMP echo request, id 2978, seq 2, length 64
10:41:46.255055 IP 192.168.1.1 > 10.0.1.1: ICMP echo reply, id 2978, seq 2, length 64
10:41:47.123360 IP 10.0.1.1 > 192.168.1.1: ICMP echo request, id 2978, seq 3, length 64
10:41:47.255448 IP 192.168.1.1 > 10.0.1.1: ICMP echo reply, id 2978, seq 3, length 64

3.3.5.5. Display the first ten packets from and to certain host on TCP port 443

This can be used to capture the setup of a TCP session.

Figure 3.17. Capture all traffic from and to host 10.0.1.1 on port 443

SH # tcpdump -ni lan0_0 -c 10 'host 10.0.1.1 and port 443'
tcpdump: WARNING: lan0_0: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on lan0_0, link-type EN10MB (Ethernet), capture size 300 bytes
10:42:56.184613 IP 10.0.1.1.57118 > 192.168.1.1.443: Flags [S], seq 1770219666, win 65535, \
     options [mss 1460,nop,wscale 1,nop,nop,TS val 2747475397 ecr 0,sackOK,eol], length 0
10:42:56.317961 IP 192.168.1.1.443 > 10.0.1.1.57118: Flags [S.], seq 2854931292, ack 17702 \
    19667, win 5792, options [mss 1460,sackOK,TS val 92454666 ecr 2747475397,nop,wscale 2] \
    , length 0
10:42:56.320460 IP 10.0.1.1.57118 > 192.168.1.1.443: Flags [.], seq 1, ack 1, win 33304, o \
    ptions [nop,nop,TS val 2747475537 ecr 92454666], length 0
10:42:56.320522 IP 10.0.1.1.57118 > 192.168.1.1.443: Flags [P.], seq 1:149, ack 1, win 333 \
    04, options [nop,nop,TS val 2747475537 ecr 92454666], length 148
10:42:56.454635 IP 192.168.1.1.443 > 10.0.1.1.57118: Flags [.], seq 1, ack 149, win 1448,  \
    options [nop,nop,TS val 92454803 ecr 2747475537], length 0
10:42:56.509572 IP 192.168.1.1.443 > 10.0.1.1.57118: Flags [.], seq 1:1449, ack 149, win 1 \
    448, options [nop,nop,TS val 92454858 ecr 2747475537], length 1448
10:42:56.514621 IP 192.168.1.1.443 > 10.0.1.1.57118: Flags [P.], seq 1449:2564, ack 149, w \
    in 1448, options [nop,nop,TS val 92454858 ecr 2747475537], length 1115
10:42:56.518535 IP 10.0.1.1.57118 > 192.168.1.1.443: Flags [.], seq 149, ack 2564, win 327 \
    46, options [nop,nop,TS val 2747475729 ecr 92454858], length 0
10:42:56.522466 IP 10.0.1.1.57118 > 192.168.1.1.443: Flags [P.], seq 149:347, ack 2564, wi \
    n 33304, options [nop,nop,TS val 2747475732 ecr 92454858], length 198
10:42:56.522477 IP 10.0.1.1.57118 > 192.168.1.1.443: Flags [P.], seq 347:384, ack 2564, wi \
    n 33304, options [nop,nop,TS val 2747475732 ecr 92454858], length 37


3.3.5.6. Display all Ethernet broadcast traffic

When the issue seems to be related to a mismatch of the VLAN tag, the best way to determine what goes wrong is to look at the Ethernet broadcast traffic.

Figure 3.18. Tcpdump filter for all Ethernet broadcast traffic

SH # tcpdump -le -ni lan0_0 -c 10 'ether broadcast or (vlan and ether broadcast)''
18:53:46.027193 00:0e:b6:42:f8:9c > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 4 \
    6: vlan 12, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.6, length 28
18:53:46.123323 00:0d:b9:17:28:de > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 4 \
    6: vlan 3, p 0, ethertype ARP, Request who-has 10.0.1.1 tell 10.0.1.9, length 28

In this example, the Ethernet frames of host 10.0.1.6 are on VLAN 12, while the Ethernet frames of host 10.0.1.9 are on VLAN 3.

If no traffic shows up, try to force it by:

  • Clearing the ARP table on the Steelhead appliance and on the router.

  • From the router, ping an unused IP address on the LAN. This will force the router to keep sending out ARP requests and then, from the Steelhead appliance, ping an unused IP address on the LAN.

3.3.6. Tcpdump related issues

3.3.6.1. TCP Checksum errors

When running tcpdump, packets created by the Steelhead appliance and put on the wire might show up with a TCP checksum mismatch. This is caused by the capability of the network card to generate the TCP header checksum and the IP header checksum.

Figure 3.19. Incorrect TCP checksum messages because of TCP checksum offloading

SH # tcpdump -lennv -s 1500 -i wan0_0
tcpdump: WARNING: wan0_0: no IPv4 address assigned
listening on wan0_0, link-type EN10MB (Ethernet), capture size 1500 bytes
19:47:09.324317 d4:9a:20:c2:52:0e > 00:0d:b9:17:28:de, ethertype IPv4 (0x0800), length 66: \
     (tos 0x0, ttl 127, id 6410, offset 0, flags [DF], proto TCP (6), length 52)
    10.0.1.1.58616 > 192.168.1.1.445: Flags [S], cksum 0x0e5a (correct), seq 3025755960, w \
    in 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
19:47:09.342483 00:0d:b9:17:28:de > d4:9a:20:c2:52:0e, ethertype IPv4 (0x0800), length 66: \
     (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.1.1.445 > 10.0.1.1.58616: Flags [S.], cksum 0x4695 (incorrect -> 0xdc12), seq  \
    753995387, ack 3025755961, win 5840, options [mss 1460,nop,nop,sackOK,nop,wscale 2], l \
    ength 0

The second packet has an incorrect calculated TCP checksum: cksum 0x4695 (incorrect -> 0xdc12). Since the MAC address identifies it as being created by this Steelhead appliance, it is safe to assume that the TCP checksum mismatch is caused by the TCP checksum offloading towards the network card and can be ignored.

3.3.6.2. TCP Segmentation Offloading

TCP Segmentation Offloading is a feature on the NICs which allows the NIC to present multiple TCP packets as one large one:

Figure 3.20. Wireshark showing larger than MSS size TCP packets

Wireshark showing larger than MSS size TCP packets

Despite that the MSS size is negotiated to 1460, the captured packets have a TCP length of 2920 and 4380 bytes in size. This is two times and three times the maximum size.

The impact of this is that a capture length of 1600 suddenly doesn't capture the whole packet and analysis by Wireshark and other tools have insufficient data to work with.

To disable this feature, use the command no interface inpathX_X gso enable. Use the command show interface inpathX_X gso to see the current settings:

Figure 3.21. Output of the command "show interface inpathX_X gso"

SH # show interfaces inpath0_0 gso
generic-segmentation-offload: on
TCP-segmentation-offload: on

3.3.6.3. WCCP / PBR captures

In a WCCP or PBR environment, traffic which is not supposed to be optimized can still be forwarded to the Steelhead appliance.

The traces of pass-through traffic will show double packets for pass-through traffic: First the incoming packet towards the Steelhead appliance, then the outgoing packet towards the default gateway. On Ethernet level the payload is the same, the only difference is the MAC addresses.

However, captures taken of this pass-through traffic will show that the MAC addresses of the two packets are the same: This is because the way the data is handled in the network buffers. Wireshark will show these packets as duplicate packets. This is a dissector issue for this redirected traffic, not an issue with the traffic.