Dealing with Bandwidth Congestion
We’ve all been there: a 50GB database backup starts, and suddenly your web server’s response time spikes from 20ms to 2 seconds. In a default Linux environment, the kernel handles packets on a first-come, first-served basis. This means a massive ISO download can easily shove your critical SSH session or database query to the back of the line.
Traffic Control (tc), part of the iproute2 suite, is the industry-standard tool for fixing this. It allows you to implement Quality of Service (QoS) by shaping and prioritizing traffic. By moving away from “best-effort” delivery, you ensure that high-priority services stay responsive even when your network pipe is at 99% utilization.
Quick Start: The 5-Minute Bandwidth Cap
You don’t always need a complex hierarchy. Sometimes you just need to stop a specific interface from saturating a shared 1Gbps uplink. The Token Bucket Filter (TBF) is the fastest way to achieve this.
Let’s say you want to cap eth0 at 10Mbps to keep a noisy backup node in check. Run these commands as root:
# Clear any existing rules on eth0
sudo tc qdisc del dev eth0 root 2>/dev/null
# Add a TBF qdisc to limit rate to 10mbit
sudo tc qdisc add dev eth0 root tbf rate 10mbit burst 32kbit latency 400msBreakdown of the parameters:
- rate: Your sustained speed limit.
- burst: The maximum data allowed to pass at hardware speed before the limit kicks in. Think of this as a small buffer for short-lived spikes.
- latency: How long a packet can sit in the queue before the kernel drops it.
To see your rules in action, use:
tc -s qdisc show dev eth0Deep Dive: Hierarchical Token Bucket (HTB)
TBF works for simple limits, but real-world QoS requires HTB (Hierarchical Token Bucket). HTB organizes traffic into a tree. Different services share a total bandwidth pool, but you can guarantee minimum speeds for critical apps while allowing them to “borrow” extra capacity when the network is quiet.
The Three Pillars of tc
- Qdisc (Queueing Discipline): The high-level algorithm managing the queue, such as HTB or FQ_CoDel.
- Class: The subdivisions where you define specific bandwidth limits.
- Filter: The logic that assigns incoming packets to a specific class.
Building a Production-Ready QoS Tree
Imagine a 100Mbps link. We want to guarantee 5Mbps for SSH, 50Mbps for Web traffic, and let everything else fight for the remaining 45Mbps. If the web server is idle, the other classes should be able to burst up to the full 100Mbps.
First, set up the root and the main parent class:
# 1. Create the HTB root
sudo tc qdisc add dev eth0 root handle 1: htb default 30
# 2. Set the total capacity (100mbit)
sudo tc class add dev eth0 parent 1: classid 1:1 htb rate 100mbitNow, define your priority tiers:
# SSH: Tier 1 (Class 10) - Guaranteed 5mbit, can burst to 100mbit
sudo tc class add dev eth0 parent 1:1 classid 1:10 htb rate 5mbit ceil 100mbit prio 1
# HTTP/HTTPS: Tier 2 (Class 20) - Guaranteed 50mbit
sudo tc class add dev eth0 parent 1:1 classid 1:20 htb rate 50mbit ceil 100mbit prio 2
# Bulk: Tier 3 (Class 30) - Guaranteed 10mbit
sudo tc class add dev eth0 parent 1:1 classid 1:30 htb rate 10mbit ceil 100mbit prio 3In this setup, rate is your “safety net” bandwidth. The ceil parameter defines the absolute maximum a class can borrow from its siblings.
Advanced Usage: Marking Packets with iptables
Writing tc filters based on raw IP addresses is a maintenance nightmare. It is much cleaner to use the iptables mangle table to “tag” packets. This lets you use friendly iptables syntax—like matching by port, state, or even process ID—to steer traffic into your tc classes.
Step 1: Link tc to iptables Marks
Tell tc to look for numeric handles (marks) on outgoing packets:
sudo tc filter add dev eth0 protocol ip parent 1:0 prio 1 handle 10 fw flowid 1:10
sudo tc filter add dev eth0 protocol ip parent 1:0 prio 2 handle 20 fw flowid 1:20Step 2: Tag Your Traffic
Now, mark your SSH and Web traffic so tc knows where they belong:
# Tag SSH (Port 22) as handle 10
sudo iptables -t mangle -A POSTROUTING -p tcp --dport 22 -j MARK --set-mark 10
# Tag Web (80/443) as handle 20
sudo iptables -t mangle -A POSTROUTING -p tcp -m multiport --dports 80,443 -j MARK --set-mark 20This separation of concerns is powerful. If you move your web server to a non-standard port, you only update one firewall rule rather than rebuilding your entire tc hierarchy.
Practical Tips from the Field
1. Egress is King
Linux tc is designed for egress (outgoing) traffic. You cannot easily control what other people send to you once it has already arrived. If you need to limit downloads, you usually shape the replies (ACKs). Controlling what leaves your server is 90% of the battle in production.
2. The Safety of Default Classes
Always define a default class in your HTB root (like default 30). Any packet that doesn’t match your filters will fall here. Without a default, unclassified traffic might bypass your limits entirely, potentially starving your high-priority classes.
3. Live Monitoring
Configuring QoS blindly is a recipe for disaster. Use this command to watch your traffic hit different classes in real-time:
watch -n 1 tc -s class show dev eth0Keep an eye on the “Sent” bytes. If a class hits its rate, you will see the “Tokens” decrease as HTB begins delaying packets to maintain the limit.
4. The Hardware Offloading Trap
Modern NICs use Generic Segmentation Offload (GSO) to bundle small packets into massive “super-packets” before they hit the driver. This can confuse tc‘s math. If your rate limiting feels erratic, try disabling offloading to see if the behavior stabilizes:
sudo ethtool -K eth0 gso off tso off5. Making it Stick
Settings applied via tc and iptables vanish after a reboot. For a reliable production setup, wrap these commands into a shell script. You can trigger this script via a systemd service or use iptables-persistent. Keeping your logic in one script makes it easier for the rest of your team to audit and modify.
Gaining control over tc gives you the power to dictate exactly how your network behaves under pressure. It is the difference between a server that crashes during a spike and one that stays perfectly responsive for your users.
