AppResponse: How does AR11 perform packet deduplication?

Categories:
AppResponse 11
Solution Number:
S32113
Last Modified:
2021-11-02
Issue
How does AR11 identify duplicate packets and perform deduplication?
 
Solution
Deduplication is enabled on a per-ViFG basis. Navigate to Administration > General Traffic Settings > Capture Jobs/Interfaces. Then the "Virtual Interface Groups" tab.



Edit a ViFG entry to enable deduplication:



Deduplication involves identifying packets which are duplicates of each other, and only recording one copy to packet storage. The duplicate packets must be seen by AppResponse within 10ms of each other.

The following fields are considered for deduplication.

For TCP
  • IP protocol
  • Src/Dst IP
  • IP ID *
  • IP Fragmentation Offset
  • Src/Dst Port number
  • TCP sequence number
  • TCP checksum
  • VIFG ID
For UDP
  • IP protocol
  • Src/Dst IP
  • IP ID *
  • IP Fragmentation Offset
  • Src/Dst Port number
  • UDP checksum
  • UDP length
  • VIFG ID
* IP packets typically have non-zero IP ID values that increment over the life of a TCP/UDP session. Some applications may set the IP ID value to 0 for all packets. The following configuration options found at the "Monitoring Interfaces" tab control whether packets with IP ID=0 are deduplicated. When not checked, packets with IP ID=0 are not deduplicated, even if all other fields under consideration are identical between a pair of packets.

Note that the Packet Broker selection is unrelated to deduplication.




Scenarios:

When duplicate packets exist, but deduplication does not occur, a symptom will be that AR 11, NetProfiler, or Portal report at least twice the expected throughput for monitored traffic.

  1. Deduplication applies to the Virtual Interface Group, which is a logical collection of physical monitoring interfaces or VLANs. So deduplication applies equally to ViFGs grouped by "Monitoring Interfaces" or "VLAN IDs" in the "Virtual Interface Groups" tab.
  2. If a packet that traverses two different VLANs is grouped in the same VIFG, it will be deduplicated. The packet's VLAN ID is not taken into consideration when identifying duplicates. To avoid deduplication of packets in different VLANS, with deduplication enabled, group the VLANs in different ViFGs.
  3. Due to TCP sequence number randomization, some packets may have their TCP Seq/Ack numbers change as they traverse a firewall. Although these fields are not examined directly for deduplication, altering Seq/Ack changes the TCP checksum. So deduplication will not occur if packets are captured before and after the firewall, as well as being grouped in the same ViFG.
Environment
deduplciation, duplicates, mifg, tcp sequence
Attachments
NOTICE: Riverbed® product names have changed. Please refer to the Product List for a complete list of product names.
Can't find an answer? Create a case