Mastering tcpdump on Linux: Packet Capture, Traffic Filtering, and Network Troubleshooting

Table of Contents

Quick start (5 min)

Six months ago, our payment gateway nodes started throwing sporadic 504 gateway timeouts at roughly 3,500 requests per second. With stripped-down, headless Ubuntu instances tucked inside private AWS VPCs, running a GUI tool like Wireshark was out of the question. We had to inspect raw wire traffic directly on the box. That outage forced us to turn tcpdump into our default triage utility, and it remains our fastest way to track down socket-level packet drops.

Getting started takes only a few minutes. Most enterprise distros ship with tcpdump pre-installed. If you are starting from a minimal Debian, Alpine, or RHEL container image, grab it first:

# On Debian/Ubuntu
sudo apt-get update && sudo apt-get install -y tcpdump

# On RHEL/AlmaLinux/Rocky
sudo dnf install -y tcpdump

Verify which network interfaces are available for capture:

sudo tcpdump -D

Running this prints active physical and virtual interfaces like eth0, ens3, and the synthetic any interface. To start watching live traffic on a specific interface without waiting on DNS lookups, run:

sudo tcpdump -i eth0 -n

The -n flag disables reverse DNS resolution. Skipping PTR queries prevents tcpdump from spamming your upstream DNS resolvers while you inspect active flows. Press Ctrl+C when you have enough samples. The command exits with a clean summary tallying captured frames, filter matches, and kernel buffer drops.

Deep dive

Unfiltered packet dumps will instantly overwhelm your terminal buffer. On a 10 Gbps link, raw captures can scroll past 50,000 packets before you can even hit Ctrl+C. Making sense of the firehose means learning how to read the wire format and filtering precisely with Berkeley Packet Filters (BPF).

Dissecting tcpdump Output Format

A typical TCP packet header line captured with tcpdump -nn -S looks like this:

14:22:01.408219 IP 192.168.1.50.44322 > 10.0.0.15.80: Flags [P.], seq 1827401:1827945, ack 392019, win 502, length 544

Key components include:

Timestamp: High-resolution system time down to microseconds (14:22:01.408219).
Source > Destination: Client 192.168.1.50:44322 sent data to web server 10.0.0.15:80.
TCP Flags: [P.] indicates Push (P) and Acknowledge (.). Common flags include [S] (SYN), [S.] (SYN-ACK), [F] (FIN), [R] (RST), and [.] (pure ACK).
Sequence and ACK numbers: Byte offsets tracking payload transmission. Here, seq 1827401:1827945 confirms 544 bytes of data were sent (1,827,945 – 1,827,401 = 544).
Window size & Length: The advertised receive buffer size (win 502) and segment payload size (length 544).

Targeted Filtering with Berkeley Packet Filters (BPF)

BPF expressions run inside kernel space before packets ever hit user-space memory buffers. Discarding irrelevant frames early prevents CPU spikes and keeps ring buffers from overflowing. You can narrow down captures by IP address, CIDR block, port, or transport protocol:

# Capture HTTP and HTTPS traffic involving a specific client IP
sudo tcpdump -i eth0 -nn "host 192.168.1.100 and (port 80 or port 443)"

# Monitor outgoing traffic directed to an external database subnet
sudo tcpdump -i eth0 -nn "dst net 10.10.0.0/16 and port 5432"

Combining conditional operators such as and, or, and not allows you to filter out SSH noise from your active terminal session:

sudo tcpdump -i any -nn "not port 22"

Advanced usage

When tracking edge cases—such as asymmetric routing or transient upstream resets—basic port filters fall short. Two techniques make hard problems solvable: streaming captures to binary .pcap files and filtering against raw byte offsets in the TCP header.

Writing and Reading PCAP Files for Offline Inspection

Reading live terminal output during a traffic surge is a recipe for missed clues. Write the raw binary stream directly to disk instead, then parse it locally or load it into Wireshark:

# Save full payloads into capture.pcap with a 50,000 packet limit
sudo tcpdump -i eth0 -nn -s 0 -c 50000 -w /tmp/api_traffic.pcap "port 8080"

To analyze the saved capture without replaying live packets:

tcpdump -nn -r /tmp/api_traffic.pcap "tcp[tcpflags] & tcp-rst != 0"

Isolating TCP Handshakes and Connection Resets

Diagnosing intermediate proxy timeouts or dropped sessions requires hunting down unexpected RST packets and unacknowledged SYNs. BPF lets you run bitwise checks against the 13th byte of the TCP header where control flags live:

# Capture only initial connection requests (SYN without ACK)
sudo tcpdump -i eth0 -nn "tcp[tcpflags] & (tcp-syn) != 0 and tcp[tcpflags] & (tcp-ack) == 0"

# Detect unexpected connection drops (TCP RST packets)
sudo tcpdump -i eth0 -nn "tcp[tcpflags] & (tcp-rst) != 0"

If you need to inspect unencrypted HTTP headers or raw Redis queries, use the ASCII (-A) or dual hex/ASCII (-X) flags while filtering for packets that actually carry payload data:

# Filter out pure ACKs and display payload bytes in ASCII
sudo tcpdump -i eth0 -nn -A -s 0 "port 80 and (((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12:1] & 0xf0)>>2)) != 0)"

Practical tips

Capturing packets on high-throughput systems can strain CPU and disk I/O if you leave parameters uncapped. These operational rules keep monitoring safe on busy production boxes:

Restrict snaplen (-s): Modern tcpdump releases default to capturing the full packet (262,144 bytes). If you only need routing metadata and flags, restrict the capture slice with -s 96. On our 4-core, 16 GB API nodes pushing 12,000 packets per second, dropping snaplen to 96 bytes cut disk write throughput from 18 MB/s down to under 1.2 MB/s.
Always pair -n with -nn: While -n halts IP reverse DNS lookups, -nn stops port-to-service translation. Without it, tcpdump consults /etc/services to convert port 443 to https or port 3306 to mysql. Running both flags shaves off subtle CPU latency during heavy bursts.
Rotate files to prevent disk exhaustion: Long debugging sessions can quietly eat up your root partition. Combine -C (file size limit in millions of bytes) with -W (maximum file count) to establish a bounded circular buffer:

# Rolling ring buffer: keeps at most 5 files of 100MB each
sudo tcpdump -i eth0 -nn -s 96 -C 100 -W 5 -w /var/log/traffic_roll.pcap "port 443"

Check the kernel drop count: When terminating a capture session with Ctrl+C, check the summary line for packets dropped by kernel. A non-zero count means the socket buffer filled up before tcpdump could read from it. If you see drops, restrict your snaplen with -s 96, tighten your BPF expression, or write temporary captures directly to a ramdisk like /dev/shm.

Once you understand BPF expressions and flag offsets, tcpdump turns opaque network glitches into clear, actionable data. It takes the guesswork out of packet delivery and lets you pinpoint failed handshakes or dropped sessions in minutes.