Being able to see the traffic that goes in and out a network card is great value to determine what occurs on network level. Tcpdump, the standard network packet capture tool, is included on the Steelhead appliances. It uses the pcap library for the capturing of the packets on the NIC.
The place where tcpdump captures the data from the network stack is just before the sending or just after the receiving of the data to and from the network card. This makes sure that the capture happens with the data as closely as it is on the network cable as possible.
Tcpdump supports both IPv4 and IPv6 addresses.
The option
-n
will prevent DNS name resolution of the IP addresses and services
detected in the captured packet.
The option
-s
specifies the number of bytes to read from the captured packets.
The option
-i
will select the interface to capture on.
The option
-c
specifies the limit on the number of packets captured.
The option
-w
specifies the filename to write the capture to.
The option
-v
will print how many packets are captured and written to disk.
The option
-le
will print the link layer information of the captured packet.
The option
-X
(capital X) will also print contents of the captured packet in
hex and character dump format.
Always use the
-n
option to disable name resolution otherwise it will slow down the
display of packets and possible dropping of captured packets.
Don't capture on the in-path interfaces, always capture on the corresponding LAN and WAN interfaces.
Don't capture on the primary interface, always capture on the prihw interface.
Use
-c
to limit the number of packets captured to prevent them from
scrolling of the screen.
When writing captured packets to disk, use the
-v
option to see the number of packets written.
Use correct filters to capture only the packets you are interested in.
The default filter is based on the standard Ethernet frame headers.
Use the
vlan
primitive to switch to the 802.1Q VLAN Ethernet frame format.
Use a snaplength of 1600 for a full payload capture.
The normal tcpdump program will not be able to decode the Riverbed TCP Options and will display them as a hex-string:
Figure 3.6. The Riverbed specific TCP options not decoded in tcpdump before RiOS 6.1.4 and 6.5.2
11:12:47.668647 IP 10.0.1.1.52175 > 192.168.1.1.80: Flags [S], seq 4101063156, win 65535, \ options [mss 1460,nop,wscale 3,nop,nop,TS val 693134564 ecr 0,sackOK,nop,nop,opt-76:01 \ 010a0001060005,opt-76:0c01,nop,eol], length 0 11:12:47.801418 IP 192.168.1.1.80 > 10.0.1.1.52175: Flags [S.], seq 20020520, ack 41010631 \ 57, win 65535, options [mss 1460,nop,wscale 3,nop,nop,TS val 693134564 ecr 0,sackOK,no \ p,nop,opt-76:0c01,nop,nop,nop,eol], length 0 11:12:47.801454 IP 192.168.1.1.80 > 10.0.1.1.52175: Flags [S.], seq 20020520, ack 41010631 \ 57, win 65535, options [mss 1460,nop,wscale 3,TS val 693134564 ecr 0,sackOK,opt-76:111 \ 10a000106c0a801061e78,opt-76:0e3d,nop,eol], length 0
With the release of RiOS 6.1.4 and 6.5.2, the tcpdump program included on the Steelhead appliances will decode the auto-discovery and the WAN Visibility Transparency TCP options:
Figure 3.7. The Riverbed specific TCP options for auto-discovery decoded in tcpdump after RiOS 6.1.4 and 6.5.2
11:12:47.668647 IP 10.0.1.1.52175 > 192.168.1.1.80: Flags [S], seq 4101063156, win 65535, \ options [mss 1460,nop,wscale 3,nop,nop,TS val 693134564 ecr 0,sackOK,nop,nop,rvbd-prob \ e AD CSH:10.0.1.6 01010a0001060005,rvbd-probe EAD 0c01,nop,eol], length 0 11:12:47.801418 IP 192.168.1.1.80 > 10.0.1.1.52175: Flags [S.], seq 20020520, ack 41010631 \ 57, win 65535, options [mss 1460,nop,wscale 3,nop,nop,TS val 693134564 ecr 0,sackOK,no \ p,nop,rvbd-probe EAD 0c01,nop,nop,nop,eol], length 0 11:12:47.801454 IP 192.168.1.1.80 > 10.0.1.1.52175: Flags [S.], seq 20020520, ack 41010631 \ 57, win 65535, options [mss 1460,nop,wscale 3,TS val 693134564 ecr 0,sackOK,rvbd-probe \ AD CSH:10.0.1.6 SSH:192.168.1.6:7800 11110a000106c0a801061e78,rvbd-probe EAD 0e3d,nop \ ,eol], length 0
For the Auto-Discovery process, it will show the IP addresses of the CSH (Client-side Steelhead appliance) and the SSH (Server-side Steelhead appliance).
The keyword
rvbd-probe
tells that the option is TCP option 76 for the auto-discovery
process.
The keyword
AD
tells that the option is part of the auto-discovery process.
The keyword
EAD
tells that the option is part of the enhanced auto-discovery
process.
The keyword
CSH
gives the IP address of the client-side Steelhead appliance.
The keyword
SSH
gives the IP address and TCP port of the server-side Steelhead
appliance.
For the WAN Visibility Transparency feature, it will show the IP addresses and TCP port numbers of the inner channel.
Figure 3.8. The Riverbed specific TCP options for WAN Visibility Transparency feature decoded in tcpdump after RiOS 6.1.4 and 6.5.2
12:02:03.274738 IP 10.0.1.1.3789 > 192.168.1.1.80: Flags [P.], seq 1:11, ack 7, win 4592, \ options [nop,nop,TS val 958790087 ecr 1176840218,rvbd-trans Transp sSH:10.0.1.6:42455 \ dSH:192.168.1.1:7800 00000a000106c0a80106a5d71e78], length 10 12:02:03.288821 IP 192.168.1.1.80 > 10.0.1.1.3789: Flags [.], seq 7, ack 11, win 23, optio \ ns [nop,nop,TS val 1176840247 ecr 958790087,rvbd-trans Transp sSH:192.168.1.6:7800 dSH \ :10.0.1.6:42455 0000c0a801060a0001061e78a5d7], length 0
The keyword
rvbd-trans
tells that the option is TCP option 78 for the WAN Visibility
Full Transparency feature.
The keyword
Transp
tells that the standard transparency options are used.
The keyword
sSH
gives the IP address and TCP port of the sending Steelhead appliance.
The keyword
dSH
gives the IP address and TCP port of the destination Steelhead
appliance.
With the release of Wireshark version 1.6.0, it can decode the auto-discovery and the transparency TCP options.
Figure 3.9. The Riverbed specific TCP options decoded in tshark
$ tshark -O tcp -V -nr autodiscovery.cap Frame 1: 94 bytes on wire (752 bits), 94 bytes captured (752 bits) Ethernet II, Src: d4:9a:20:c2:52:0e (d4:9a:20:c2:52:0e), Dst: 00:0d:b9:17:28:de (00:0d:b9: \ 17:28:de) Internet Protocol Version 4, Src: 10.0.1.1 (10.0.1.1), Dst: 192.168.1.1 (192.168.1.1) Transmission Control Protocol, Src Port: 52175 (52175), Dst Port: 80 (80), Seq: 0, Len: 0 Source port: 52175 (52175) Destination port: 80 (80) [Stream index: 0] Sequence number: 0 (relative sequence number) Header length: 60 bytes Flags: 0x02 (SYN) Window size value: 65535 [Calculated window size: 65535] Checksum: 0xe76d [validation disabled] Options: (40 bytes) Maximum segment size: 1460 bytes No-Operation (NOP) Window scale: 3 (multiply by 8) No-Operation (NOP) No-Operation (NOP) Timestamps: TSval 693134564, TSecr 0 TCP SACK Permitted Option: True No-Operation (NOP) No-Operation (NOP) Riverbed Probe: Probe Query, CSH IP: 10.0.1.6 Length: 10 0000 .... = Type: 0 .... 0001 = Version: 1 Reserved CSH IP: 10.0.1.6 (10.0.1.6) Application Version: 5 Riverbed Probe: Probe Query Info Length: 4 0000 110. = Type: 6 .... ...0 = Version: 2 Probe Flags: 0x01 .... .0.. = Not CFE: Not set .... ...1 = Last Notify: Set No-Operation (NOP) End of Option List (EOL) Frame 2: 86 bytes on wire (688 bits), 86 bytes captured (688 bits) Ethernet II, Src: 00:0d:b9:17:28:de (00:0d:b9:17:28:de), Dst: d4:9a:20:c2:52:0e (d4:9a:20: \ c2:52:0e) Internet Protocol Version 4, Src: 192.168.1.1 (192.168.1.1), Dst: 10.0.1.1 (10.0.1.1) Transmission Control Protocol, Src Port: 80 (80), Dst Port: 52175 (52175), Seq: 0, Ack: 1, \ Len: 0 Source port: 80 (80) Destination port: 52175 (52175) [Stream index: 0] Sequence number: 0 (relative sequence number) Acknowledgement number: 1 (relative ack number) Header length: 52 bytes Flags: 0x12 (SYN, ACK) Window size value: 65535 [Calculated window size: 65535] Checksum: 0xe020 [validation disabled] Options: (32 bytes) Maximum segment size: 1460 bytes No-Operation (NOP) Window scale: 3 (multiply by 8) No-Operation (NOP) No-Operation (NOP) Timestamps: TSval 693134564, TSecr 0 TCP SACK Permitted Option: True No-Operation (NOP) No-Operation (NOP) Riverbed Probe: Probe Query Info Length: 4 0000 110. = Type: 6 .... ...0 = Version: 2 Probe Flags: 0x01 .... .0.. = Not CFE: Not set .... ...1 = Last Notify: Set No-Operation (NOP) No-Operation (NOP) No-Operation (NOP) End of Option List (EOL) [SEQ/ACK analysis] [This is an ACK to the segment in frame: 1] [The RTT to ACK the segment was: 0.132771000 seconds] Frame 3: 94 bytes on wire (752 bits), 94 bytes captured (752 bits) Ethernet II, Src: 00:0d:b9:17:28:de (00:0d:b9:17:28:de), Dst: d4:9a:20:c2:52:0e (d4:9a:20: \ c2:52:0e) Internet Protocol Version 4, Src: 192.168.1.1 (192.168.1.1), Dst: 10.0.1.1 (10.0.1.1) Transmission Control Protocol, Src Port: 80 (80), Dst Port: 52175 (52175), Seq: 0, Ack: 1, \ Len: 0 Source port: 80 (80) Destination port: 52175 (52175) [Stream index: 0] Sequence number: 0 (relative sequence number) Acknowledgement number: 1 (relative ack number) Header length: 60 bytes Flags: 0x12 (SYN, ACK) Window size value: 65535 [Calculated window size: 65535] Checksum: 0x7893 [validation disabled] Options: (40 bytes) Maximum segment size: 1460 bytes No-Operation (NOP) Window scale: 3 (multiply by 8) Timestamps: TSval 693134564, TSecr 0 TCP SACK Permitted Option: True Riverbed Probe: Probe Response, Server Steelhead: 192.168.1.6:7800 Length: 14 0001 .... = Type: 1 .... 0001 = Version: 1 Reserved CSH IP: 10.0.1.6 (10.0.1.6) SSH IP: 192.168.1.6 (192.168.1.6) SSH Port: 7800 Riverbed Probe: Probe Response Info Length: 4 0000 111. = Type: 7 .... ...0 = Version: 2 Probe Flags: 0x3d ...1 .... = Disable Probe Cache on CSH: Set .... ..0. = SSL Enabled: Not set .... ...1 = SSH outer to server established: Set No-Operation (NOP) End of Option List (EOL)
There are several basic pcap primitives to be understood before tcpdump can be used effectively:
The primitive
host <IP address>
filters the packets to match this source or destination IP address.
The primitive
net <IP subnet>
filters the packets to match this source or destination IP subnet.
The primitive
port <port number>
filters the packets to match this source or destination TCP or
UDP port numbers.
The primitive
ip
and
ip6
filters explicitly for IPv4 and IPv6 addresses.
The primitive
icmp
filters for only ICMP packets, the primitive
tcp
filters for only TCP packets, the primitive
udp
filters for only UDP packets.
The primitive
vlan
is not so much a filter but more a change in the offset in the
filter: From this part of the filter, the offset for the IP packet
is indexed to the size of a 802.1Q VLAN tagged Ethernet frame.
The various primitives can be merged with a boolean "and" and "or". But keep in mind that, unlike boolean logic, the "and" and "or" have the same precedence. The precedence can be changed by using round brackets around a part of the filter.
For example, if the filter needs to capture all traffic from host
H1 on port P1 or from host H2 on port P2, then
host H1 and port P1 or host H2 and port P2
gets interpreted as
((((host H1) and port P1) or host H2) and port P2)
.
However, what was needed was:
(host H1 and port P1) or (host H2 and port P2)
.
Always use round brackets to prioritize parts of the filter!
The first example is capturing the first 10 packets to host 10.0.1.1.
Figure 3.11. Display the first ten packets from and to a certain host
SH # tcpdump -ni lan0_0 -c 10 'host 10.0.1.1' tcpdump: WARNING: lan0_0: no IPv4 address assigned tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on lan0_0, link-type EN10MB (Ethernet), capture size 96 bytes 12:07:07.298829 ARP, Request who-has 10.0.1.6 tell 10.0.1.1, length 46 12:07:07.298872 ARP, Reply 10.0.1.6 is-at 00:0e:b6:8c:74:9c, length 28 12:07:08.194021 IP 10.0.1.1.52902 > 202.83.176.1.53: 53837 [1au] A? www.mavetju.org. (44) 12:07:08.234151 IP 202.83.176.1.53 > 10.0.1.1.52902: 53837*- 1/2/3 A 192.168.1.1 (141) 12:07:09.464621 IP 10.0.1.1.42825 > 192.168.1.1.80: Flags [S], seq 3872240793, win 65535, \ options [mss 1460,nop,wscale 6,sackOK,TS val 5351449 ecr 0], length 0 12:07:09.598157 IP 192.168.1.1.80 > 10.0.1.1.42825: Flags [S.], seq 2971872340, ack 387224 \ 0794, win 5792, options [mss 1460,sackOK,TS val 285084142 ecr 5351449,nop,wscale 2], l \ ength 0 12:07:09.602342 IP 10.0.1.1.42825 > 192.168.1.1.80: Flags [.], seq 1, ack 1, win 1040, opt \ ions [nop,nop,TS val 5351588 ecr 285084142], length 0 12:07:09.602375 IP 10.0.1.1.42825 > 192.168.1.1.80: Flags [P.], seq 1:188, ack 1, win 1040 \ , options [nop,nop,TS val 5351589 ecr 285084142], length 187 12:07:09.602393 IP 192.168.1.1.80 > 10.0.1.1.42825: Flags [.], seq 1, ack 188, win 1716, o \ ptions [nop,nop,TS val 285084146 ecr 5351589], length 0 12:07:09.875867 IP 192.168.1.1.80 > 10.0.1.1.42825: Flags [P.], seq 1:1151, ack 188, win 1 \ 716, options [nop,nop,TS val 285084420 ecr 5351589], length 1150 10 packets captured 14 packets received by filter 0 packets dropped by kernel
In the example above, 14 packets were captured from the network and 10 matched the filter. Zero packets could not be processed in time by the pcap filter.
The packets are: an Ethernet broadcast ARP request and reply, a DNS request and answer and then a TCP SYN, SYN/ACK and ACK to setup the TCP session and some data towards a web server.
For troubleshooting, capturing traffic between two hosts is required. Here we filter between the hosts 10.0.1.1 and 192.168.1.1.
Figure 3.12. Display the first ten packets between hosts 10.0.1.1 and 192.168.1.1
SH # tcpdump -ni lan0_0 -c 10 'host 10.0.1.1 and host 192.168.1.1' tcpdump: WARNING: lan0_0: no IPv4 address assigned tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on lan0_0, link-type EN10MB (Ethernet), capture size 96 bytes 12:07:09.464621 IP 10.0.1.1.42825 > 192.168.1.1.80: Flags [S], seq 3872240793, win 65535, \ options [mss 1460,nop,wscale 6,sackOK,TS val 5351449 ecr 0], length 0 12:07:09.598157 IP 192.168.1.1.80 > 10.0.1.1.42825: Flags [S.], seq 2971872340, ack 387224 \ 0794, win 5792, options [mss 1460,sackOK,TS val 285084142 ecr 5351449,nop,wscale 2], l \ ength 0 12:07:09.602342 IP 10.0.1.1.42825 > 192.168.1.1.80: Flags [.], seq 1, ack 1, win 1040, opt \ ions [nop,nop,TS val 5351588 ecr 285084142], length 0 12:07:09.602375 IP 10.0.1.1.42825 > 192.168.1.1.80: Flags [P.], seq 1:188, ack 1, win 1040 \ , options [nop,nop,TS val 5351589 ecr 285084142], length 187 12:07:09.602393 IP 192.168.1.1.80 > 10.0.1.1.42825: Flags [.], seq 1, ack 188, win 1716, o \ ptions [nop,nop,TS val 285084146 ecr 5351589], length 0 12:07:09.875867 IP 192.168.1.1.80 > 10.0.1.1.42825: Flags [P.], seq 1:1151, ack 188, win 1 \ 716, options [nop,nop,TS val 285084420 ecr 5351589], length 1150 12:07:09.979743 IP 10.0.1.100.42825 > 192.168.1.6.80: Flags [.], seq 188, ack 1151, win 10 \ 40, options [nop,nop,TS val 5351967 ecr 285084420], length 0 12:07:10.881288 IP 10.0.1.100.42825 > 192.168.1.6.80: Flags [P.], seq 188:420, ack 1151, w \ in 1040, options [nop,nop,TS val 5352869 ecr 285084420], length 232 12:07:10.881392 IP 192.168.1.6.80 > 10.0.1.100.42825: Flags [.], seq 1151, ack 420, win 17 \ 16, options [nop,nop,TS val 285085425 ecr 5352869], length 0 12:07:11.106812 IP 192.168.1.6.80 > 10.0.1.100.42825: Flags [P.], seq 1151:1663, ack 420, \ win 1716, options [nop,nop,TS val 285085651 ecr 5352869], length 512 10 packets captured 18 packets received by filter 0 packets dropped by kernel
The packets shown are the setup of a HTTP TCP session and the exchange of the data.
The difference in length of a normal IEEE 802.3 Ethernet frame and a IEEE 802.1q VLAN Ethernet frame is four bytes:
The implementation of pcap capture library is not capable to automatically determine the size difference of the two frame types and it will assume the size of a normal 802.3 Ethernet frame. This means that the offset for the IP header, and thus the higher level protocols on top of IP, for 802.1q frames is incorrect and the filtering of packets on the specified fields will fail. To overcome this, the primitive "vlan" can be used, however it will cause the rest of the filter to assume that the Ethernet frame size is the size of a 802.1q Ethernet frame.
To filter for host 10.0.1.1 for normal 802.3 Ethernet frames:
host 10.0.1.1
.
To filter for host 10.0.1.1 for 802.1q Ethernet frames:
(vlan and host 10.0.1.1)
.
To filter for host 10.0.1.1 for normal 802.3 and 802.1q Ethernet frames:
host 10.0.1.1 or (vlan and host 10.0.1.1)
.
The default output of tcpdump doesn't show the VLAN tag ID of the 802.1q
Ethernet frames, to show then use the option
-le
to display them.
Figure 3.15. Display the first ten packets between host 10.0.1.1 and host 192.168.1.1
SH # tcpdump -le -ni lan0_0 -c 10 'host 10.0.1.1 and host 192.168.1.1 or (vlan and host 10 \ .0.1.1 and host 192.168.1.1)'' tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on lan0_0, link-type EN10MB (Ethernet), capture size 96 bytes 12:11:36.684544 00:21:70:3e:6b:7f > 00:0d:b9:17:28:de, ethertype 802.1Q (0x8100), length 9 \ 4: vlan 200, p 0, ethertype IPv4, IP 10.0.1.1.60035 > 192.168.1.1.https: S 1139103120: \ 1139103120(0) win 5840 <mss 1460,sackOK,timestamp 1864443320 0,nop,wscale 2> 12:11:36.685159 00:0d:b9:17:28:de > 00:21:70:3e:6b:7f, ethertype 802.1Q (0x8100), length 8 \ 2: vlan 200, p 0, ethertype IPv4, IP 192.168.1.1.https > 10.0.1.1.60035: S 2827780182: \ 2827780182(0) ack 1139103121 win 4380 <mss 1460,nop,wscale 0,nop,nop,timestamp 7927005 \ 29 1864443320,sackOK,eol> 12:11:36.685203 00:21:70:3e:6b:7f > 00:0d:b9:17:28:de, ethertype 802.1Q (0x8100), length 7 \ 0: vlan 200, p 0, ethertype IPv4, IP 10.0.1.1.60035 > 192.168.1.1.https: . ack 1 win 1 \ 460 <nop,nop,timestamp 1864443320 792700529> 12:11:37.025716 00:21:70:3e:6b:7f > 00:0d:b9:17:28:de, ethertype 802.1Q (0x8100), length 1 \ 36: vlan 200, p 0, ethertype IPv4, IP 10.0.1.1.60035 > 192.168.1.1.https: P 1:67(66) a \ ck 1 win 1460 <nop,nop,timestamp 1864443661 792700529> 12:11:37.026275 00:0d:b9:17:28:de > 00:21:70:3e:6b:7f, ethertype 802.1Q (0x8100), length 1 \ 518: vlan 200, p 0, ethertype IPv4, IP 192.168.1.1.https > 10.0.1.1.60035: . 1:1449(14 \ 48) ack 67 win 4380 <nop,nop,timestamp 792700870 1864443661> 12:11:37.026284 00:0d:b9:17:28:de > 00:21:70:3e:6b:7f, ethertype 802.1Q (0x8100), length 1 \ 518: vlan 200, p 0, ethertype IPv4, IP 192.168.1.1.https > 10.0.1.1.60035: . 1449:2897 \ (1448) ack 67 win 4380 <nop,nop,timestamp 792700870 1864443661> 12:11:37.026288 00:0d:b9:17:28:de > 00:21:70:3e:6b:7f, ethertype 802.1Q (0x8100), length 1 \ 518: vlan 200, p 0, ethertype IPv4, IP 192.168.1.1.https > 10.0.1.1.60035: P 2897:4345 \ (1448) ack 67 win 4380 <nop,nop,timestamp 792700870 1864443661> 12:11:37.026314 00:21:70:3e:6b:7f > 00:0d:b9:17:28:de, ethertype 802.1Q (0x8100), length 7 \ 0: vlan 200, p 0, ethertype IPv4, IP 10.0.1.1.60035 > 192.168.1.1.https: . ack 1449 wi \ n 2184 <nop,nop,timestamp 1864443661 792700870> 12:11:37.026327 00:21:70:3e:6b:7f > 00:0d:b9:17:28:de, ethertype 802.1Q (0x8100), length 7 \ 0: vlan 200, p 0, ethertype IPv4, IP 10.0.1.1.60035 > 192.168.1.1.https: . ack 2897 wi \ n 2908 <nop,nop,timestamp 1864443661 792700870> 12:11:37.026336 00:21:70:3e:6b:7f > 00:0d:b9:17:28:de, ethertype 802.1Q (0x8100), length 7 \ 0: vlan 200, p 0, ethertype IPv4, IP 10.0.1.1.60035 > 192.168.1.1.https: . ack 4345 wi \ n 3632 <nop,nop,timestamp 1864443661 792700870> 10 packets captured 19 packets received by filter 0 packets dropped by kernel
This can be used to determine the forward and backward path the traffic takes by pinging from the client.
Figure 3.16. Capture all traffic for all ICMP traffic from and to host 10.0.1.1
SH # tcpdump -ni lan0_0 -c 10 'host 10.0.1.1 and icmp' tcpdump: WARNING: lan0_0: no IPv4 address assigned tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on lan0_0, link-type EN10MB (Ethernet), capture size 96 bytes 10:41:36.717861 IP 10.0.1.1 > 192.168.1.1: ICMP echo request, id 2722, seq 0, length 64 10:41:36.851611 IP 192.168.1.1 > 10.0.1.1: ICMP echo reply, id 2722, seq 0, length 64 10:41:37.714418 IP 10.0.1.1 > 192.168.1.1: ICMP echo request, id 2722, seq 1, length 64 10:41:37.847110 IP 192.168.1.1 > 10.0.1.1: ICMP echo reply, id 2722, seq 1, length 64 10:41:44.120470 IP 10.0.1.1 > 192.168.1.1: ICMP echo request, id 2978, seq 0, length 64 10:41:44.253989 IP 192.168.1.1 > 10.0.1.1: ICMP echo reply, id 2978, seq 0, length 64 10:41:45.121247 IP 10.0.1.1 > 192.168.1.1: ICMP echo request, id 2978, seq 1, length 64 10:41:45.254498 IP 192.168.1.1 > 10.0.1.1: ICMP echo reply, id 2978, seq 1, length 64 10:41:46.122364 IP 10.0.1.1 > 192.168.1.1: ICMP echo request, id 2978, seq 2, length 64 10:41:46.255055 IP 192.168.1.1 > 10.0.1.1: ICMP echo reply, id 2978, seq 2, length 64 10:41:47.123360 IP 10.0.1.1 > 192.168.1.1: ICMP echo request, id 2978, seq 3, length 64 10:41:47.255448 IP 192.168.1.1 > 10.0.1.1: ICMP echo reply, id 2978, seq 3, length 64
This can be used to capture the setup of a TCP session.
Figure 3.17. Capture all traffic from and to host 10.0.1.1 on port 443
SH # tcpdump -ni lan0_0 -c 10 'host 10.0.1.1 and port 443' tcpdump: WARNING: lan0_0: no IPv4 address assigned tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on lan0_0, link-type EN10MB (Ethernet), capture size 300 bytes 10:42:56.184613 IP 10.0.1.1.57118 > 192.168.1.1.443: Flags [S], seq 1770219666, win 65535, \ options [mss 1460,nop,wscale 1,nop,nop,TS val 2747475397 ecr 0,sackOK,eol], length 0 10:42:56.317961 IP 192.168.1.1.443 > 10.0.1.1.57118: Flags [S.], seq 2854931292, ack 17702 \ 19667, win 5792, options [mss 1460,sackOK,TS val 92454666 ecr 2747475397,nop,wscale 2] \ , length 0 10:42:56.320460 IP 10.0.1.1.57118 > 192.168.1.1.443: Flags [.], seq 1, ack 1, win 33304, o \ ptions [nop,nop,TS val 2747475537 ecr 92454666], length 0 10:42:56.320522 IP 10.0.1.1.57118 > 192.168.1.1.443: Flags [P.], seq 1:149, ack 1, win 333 \ 04, options [nop,nop,TS val 2747475537 ecr 92454666], length 148 10:42:56.454635 IP 192.168.1.1.443 > 10.0.1.1.57118: Flags [.], seq 1, ack 149, win 1448, \ options [nop,nop,TS val 92454803 ecr 2747475537], length 0 10:42:56.509572 IP 192.168.1.1.443 > 10.0.1.1.57118: Flags [.], seq 1:1449, ack 149, win 1 \ 448, options [nop,nop,TS val 92454858 ecr 2747475537], length 1448 10:42:56.514621 IP 192.168.1.1.443 > 10.0.1.1.57118: Flags [P.], seq 1449:2564, ack 149, w \ in 1448, options [nop,nop,TS val 92454858 ecr 2747475537], length 1115 10:42:56.518535 IP 10.0.1.1.57118 > 192.168.1.1.443: Flags [.], seq 149, ack 2564, win 327 \ 46, options [nop,nop,TS val 2747475729 ecr 92454858], length 0 10:42:56.522466 IP 10.0.1.1.57118 > 192.168.1.1.443: Flags [P.], seq 149:347, ack 2564, wi \ n 33304, options [nop,nop,TS val 2747475732 ecr 92454858], length 198 10:42:56.522477 IP 10.0.1.1.57118 > 192.168.1.1.443: Flags [P.], seq 347:384, ack 2564, wi \ n 33304, options [nop,nop,TS val 2747475732 ecr 92454858], length 37
When the issue seems to be related to a mismatch of the VLAN tag, the best way to determine what goes wrong is to look at the Ethernet broadcast traffic.
Figure 3.18. Tcpdump filter for all Ethernet broadcast traffic
SH # tcpdump -le -ni lan0_0 -c 10 'ether broadcast or (vlan and ether broadcast)'' 18:53:46.027193 00:0e:b6:42:f8:9c > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 4 \ 6: vlan 12, p 0, ethertype ARP, Request who-has 10.0.1.9 tell 10.0.1.6, length 28 18:53:46.123323 00:0d:b9:17:28:de > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 4 \ 6: vlan 3, p 0, ethertype ARP, Request who-has 10.0.1.1 tell 10.0.1.9, length 28
In this example, the Ethernet frames of host 10.0.1.6 are on VLAN 12, while the Ethernet frames of host 10.0.1.9 are on VLAN 3.
If no traffic shows up, try to force it by:
Clearing the ARP table on the Steelhead appliance and on the router.
From the router, ping an unused IP address on the LAN. This will force the router to keep sending out ARP requests and then, from the Steelhead appliance, ping an unused IP address on the LAN.
When running tcpdump, packets created by the Steelhead appliance and put on the wire might show up with a TCP checksum mismatch. This is caused by the capability of the network card to generate the TCP header checksum and the IP header checksum.
Figure 3.19. Incorrect TCP checksum messages because of TCP checksum offloading
SH # tcpdump -lennv -s 1500 -i wan0_0 tcpdump: WARNING: wan0_0: no IPv4 address assigned listening on wan0_0, link-type EN10MB (Ethernet), capture size 1500 bytes 19:47:09.324317 d4:9a:20:c2:52:0e > 00:0d:b9:17:28:de, ethertype IPv4 (0x0800), length 66: \ (tos 0x0, ttl 127, id 6410, offset 0, flags [DF], proto TCP (6), length 52) 10.0.1.1.58616 > 192.168.1.1.445: Flags [S], cksum 0x0e5a (correct), seq 3025755960, w \ in 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0 19:47:09.342483 00:0d:b9:17:28:de > d4:9a:20:c2:52:0e, ethertype IPv4 (0x0800), length 66: \ (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 52) 192.168.1.1.445 > 10.0.1.1.58616: Flags [S.], cksum 0x4695 (incorrect -> 0xdc12), seq \ 753995387, ack 3025755961, win 5840, options [mss 1460,nop,nop,sackOK,nop,wscale 2], l \ ength 0
The second packet has an incorrect calculated TCP checksum:
cksum 0x4695 (incorrect -> 0xdc12)
.
Since the MAC address identifies it as being created by this Steelhead
appliance, it is safe to assume that the TCP checksum mismatch is
caused by the TCP checksum offloading towards the network card and
can be ignored.
TCP Segmentation Offloading is a feature on the NICs which allows the NIC to present multiple TCP packets as one large one:
Despite that the MSS size is negotiated to 1460, the captured packets have a TCP length of 2920 and 4380 bytes in size. This is two times and three times the maximum size.
The impact of this is that a capture length of 1600 suddenly doesn't capture the whole packet and analysis by Wireshark and other tools have insufficient data to work with.
To disable this feature, use the command
no interface inpathX_X gso enable
.
Use the command
show interface inpathX_X gso
to see the current settings:
Figure 3.21. Output of the command "show interface inpathX_X gso"
SH # show interfaces inpath0_0 gso generic-segmentation-offload: on TCP-segmentation-offload: on
In a WCCP or PBR environment, traffic which is not supposed to be optimized can still be forwarded to the Steelhead appliance.
The traces of pass-through traffic will show double packets for pass-through traffic: First the incoming packet towards the Steelhead appliance, then the outgoing packet towards the default gateway. On Ethernet level the payload is the same, the only difference is the MAC addresses.
However, captures taken of this pass-through traffic will show that the MAC addresses of the two packets are the same: This is because the way the data is handled in the network buffers. Wireshark will show these packets as duplicate packets. This is a dissector issue for this redirected traffic, not an issue with the traffic.