The 2 AM Latency Nightmare
It was 2 AM, and I was staring at a monitoring dashboard that looked like a heart attack. A remote team was trying to push a massive database backup over a 100Mbps fiber link, and simultaneously, our VoIP system started dropping calls. The bandwidth wasn’t even fully saturated—we were hitting about 85%—but the ping times jumped from 20ms to 600ms. This is the classic signature of Bufferbloat.
In my real-world experience, this is one of the essential skills to master. You can have the fastest fiber connection in the world, but if your router or Linux server manages its buffers poorly, your network will feel sluggish the moment someone starts a download or a YouTube stream. I spent that night digging through traffic control (tc) manuals, and that’s when I discovered that the old ways of handling Quality of Service (QoS) were no longer enough. We needed something smarter.
Understanding the Enemy: What is Bufferbloat?
Bufferbloat happens when network equipment is designed with excessively large buffers. When your link gets busy, these buffers fill up with packets. Instead of dropping packets early to signal the sender (TCP) to slow down, the router holds onto them. These packets sit in a long queue, waiting their turn, which adds massive delay (latency) to every other bit of traffic.
Think of it like a supermarket checkout. If the store allows a line of 100 people to form at one register, even if you only have one candy bar, you’re stuck behind everyone with full carts. Smart Queue Management (SQM) is like a manager opening a dedicated lane for people with just a few items, ensuring nobody waits too long.
Why CAKE is the Solution
For years, fq_codel was the gold standard for fighting Bufferbloat. But today, we have CAKE (Common Applications Kept Enhanced). CAKE is a comprehensive queue discipline (qdisc) for the Linux kernel that combines several advanced techniques:
- Bandwidth Shaper: It limits traffic slightly below your actual line speed to ensure the bottleneck happens on your Linux machine (where you control the queue) rather than in a dumb ISP modem.
- Flow Isolation: It ensures that one heavy download doesn’t drown out a gaming session or a DNS query.
- DiffServ Awareness: It respects traffic priority markings (like those used by VoIP).
- Zero Configuration: Unlike older methods that required complex “tc” scripts with many classes, CAKE is mostly “set and forget.”
Prerequisites and Installation
To follow along, you need a Linux machine acting as a gateway or a server handling significant traffic. Most modern kernels (4.19+) have CAKE built-in. You will need the iproute2 package, which provides the tc command.
First, check if your system supports CAKE:
modinfo sch_cakeIf you see a description of the module, you’re good to go. If not, you might need to update your kernel or install the iproute2-next package depending on your distribution.
On Debian/Ubuntu:
sudo apt update
sudo apt install iproute2Implementing SQM with CAKE
The strategy is simple: we tell Linux to limit its outgoing and incoming speeds to about 90-95% of our actual ISP bandwidth. This prevents the ISP’s unmanaged buffers from ever filling up.
Step 1: Identify your interface and speed
Find your network interface name (e.g., eth0, wan0, enp1s0):
ip link showRun a speed test to find your baseline. Let’s assume you have a 100Mbps Down / 20Mbps Up connection.
Step 2: Apply CAKE to the Outbound Traffic
We apply CAKE to the interface facing the internet. For the upload (egress), we set the bandwidth to 18Mbit (90% of 20Mbps).
# Clear any existing qdisc
sudo tc qdisc del dev eth0 root 2> /dev/null
# Add CAKE for upload
sudo tc qdisc add dev eth0 root cake bandwidth 18mbit besteffort triple-isolate washHere is what those flags do:
bandwidth 18mbit: Forces the bottleneck to happen here.besteffort: The default mode for most internet traffic.triple-isolate: Isolates flows based on source/destination IP and port. This is the secret sauce that keeps your game ping low while downloading.wash: Cleans up extra headers that might confuse the shaper.
Step 3: Handling Inbound Traffic (Ingress)
Handling download is trickier because you can’t technically control what the internet sends you. However, you can use an Intermediate Functional Block (IFB) to redirect incoming traffic into a queue that you can shape.
First, enable the IFB module:
sudo modprobe ifb
sudo ip link set dev ifb0 upNow, redirect incoming traffic from eth0 to ifb0 and apply CAKE there (setting it to 90Mbps for our 100Mbps link):
# Redirect ingress traffic
sudo tc qdisc add dev eth0 handle ffff: ingress
sudo tc filter add dev eth0 parent ffff: protocol ip u32 match u32 0 0 action mirred egress redirect dev ifb0
# Apply CAKE to the IFB device
sudo tc qdisc add dev ifb0 root cake bandwidth 90mbit besteffort triple-isolate washVerification & Monitoring
Once applied, you should immediately feel the difference. But as engineers, we don’t rely on feelings; we rely on data. Use the tc statistics command to see how CAKE is performing:
tc -s qdisc show dev eth0
tc -s qdisc show dev ifb0Look for the backlog and dropped stats. If you see “dropped” packets, don’t panic! That is CAKE doing its job—dropping packets to tell the sender to slow down so the buffer doesn’t overflow.
To truly verify the fix, I recommend using the Waveform Bufferbloat Test or DSLReports Speedtest. Run these while the SQM is active. You are looking for an “A” or “A+” grade, meaning your latency remains stable even when the connection is fully loaded.
Fine-tuning
If you still see latency spikes, lower the bandwidth limit by another 5%. ISPs often over-provision or have fluctuating speeds. If your 100Mbps link sometimes drops to 80Mbps during peak hours, you should set your CAKE bandwidth to 75Mbps to be safe.
If you are on a connection with specific overhead (like DSL or PPPoE), CAKE can handle that too. Just add the keyword to the command:
sudo tc qdisc replace dev eth0 root cake bandwidth 18mbit pppoe-vcmuxMaking it Persistent
The tc commands will disappear after a reboot. To make them permanent, the easiest way on modern Linux systems is to create a small shell script and trigger it via a systemd service or use the network-dispatcher (on Ubuntu) or if-up.d scripts.
For most users, creating a simple systemd service is the most reliable method:
[Unit]
Description=Apply CAKE SQM
After=network.target
[Service]
Type=oneshot
ExecStart=/usr/local/bin/apply-sqm.sh
RemainAfterExit=yes
[Install]
WantedBy=multi-user.targetThe network should feel snappy now. No more yelling across the house for someone to stop their download so you can finish a meeting or a gaming match. CAKE handles the heavy lifting, ensuring every flow gets its fair share without ruining the latency for everyone else.
