Context & Why
At 2:15 AM, our on-call pager lit up with alerts across 40 nodes. A mid-tier enterprise tenant on our hypervisor cluster had just spun up three database nodes, and half their cluster dropped offline immediately. The root cause was classic VLAN collision.
Both Tenant A and Tenant B had built internal environments on VLAN 100. Our distribution layer treated all frames as standard single-tagged 802.1q traffic. Because of that, their broadcast domains merged instantly. Within 90 seconds, our top-of-rack Arista switches logged over 14,000 MAC flap notifications, dropping customer packets by the thousands.
IEEE 802.1q uses a 12-bit VLAN identifier. That gives you 4,094 usable broadcast domains. In a multi-tenant colocation or managed hosting facility running 50+ clients, that pool dries up fast. More critically, you cannot control a customer’s internal addressing. Tenant A might demand VLAN 100 for web tiers, while Tenant B uses VLAN 100 for Ceph storage. Rewriting their tags breaks their automation. Stripping them breaks their topology.
IEEE 802.1ad—universally known as QinQ—solves this problem. The standard encapsulates an inner 802.1q Customer VLAN tag (C-TAG, EtherType 0x8100) within an outer Service VLAN tag (S-TAG, EtherType 0x88a8). Your core network handles switching based exclusively on the outer S-TAG. The customer’s inner tag remains untouched until traffic arrives at their specific VM or gateway router.
While software overlays like VXLAN and Geneve dominate hyperscale clouds, bare-metal deployments and carrier-grade Linux edge nodes still lean heavily on QinQ. Pure Layer 2 dual-tagging eliminates the 50-byte UDP encapsulation overhead of VXLAN. It delivers raw line-rate throughput without eating into CPU cycles.
Installation & Prerequisites
Every modern Linux kernel since 3.10 includes 802.1ad support. However, minimal server distributions rarely load the required kernel module during boot.
Load the 8021q module immediately:
sudo modprobe 8021qPersist the module across reboots so a scheduled host restart does not break connectivity:
echo "8021q" | sudo tee /etc/modules-load.d/8021q.confInstall the required networking and inspection toolsets:
# Debian, Ubuntu, or Proxmox VE
sudo apt-get update && sudo apt-get install -y iproute2 ethtool tcpdump
# RHEL, AlmaLinux, or Rocky Linux
sudo dnf install -y iproute ethtool tcpdumpNext, verify your physical interface capabilities. A standard untagged Ethernet frame measures 1518 bytes including headers and FCS. Stacking two 4-byte tags pushes that minimum frame size to 1526 bytes. If your physical NIC or switch port drops anything over 1500 bytes, QinQ will drop payloads silently. Inspect the link state on your uplink interface (we will use eth0):
ethtool eth0 | grep -i speed
ip link show eth0Configuration
Always configure your stack from the underlying hardware up to the customer namespace. Never define sub-interfaces before confirming parent link readiness.
1. Adjust Physical Interface MTU
First, raise the parent interface MTU. You need at least 1508 bytes to account for both 4-byte tags while maintaining a standard 1500-byte IP payload. If your physical switch supports jumbo frames, set it directly to 9000 bytes:
sudo ip link set dev eth0 mtu 9000
sudo ip link set dev eth0 up2. Create the Outer Service VLAN (802.1ad)
Assign Tenant A an outer Service VLAN ID of 1000. Pass proto 802.1ad to force Linux to write EtherType 0x88a8 into the outer header rather than the standard 0x8100:
sudo ip link add link eth0 name eth0.1000 type vlan proto 802.1ad id 1000
sudo ip link set dev eth0.1000 mtu 8996 up3. Stack the Customer VLAN (802.1q)
Build the inner customer VLAN directly on top of the outer S-TAG interface. This interface processes the standard 802.1q tag (EtherType 0x8100). In this scenario, Tenant A uses internal VLAN 100:
sudo ip link add link eth0.1000 name eth0.1000.100 type vlan proto 802.1q id 100
sudo ip link set dev eth0.1000.100 mtu 8992 up4. Isolating Conflicting Tenants with Network Namespaces
Now consider Tenant B. They also configure VLAN 100 with identical private IPs (192.168.100.1/24). Assign Tenant B an outer S-TAG of 2000 on the same physical server:
# Outer S-TAG for Tenant B
sudo ip link add link eth0 name eth0.2000 type vlan proto 802.1ad id 2000
sudo ip link set dev eth0.2000 mtu 8996 up
# Inner C-TAG for Tenant B (reusing VLAN ID 100 without collisions)
sudo ip link add link eth0.2000 name eth0.2000.100 type vlan proto 802.1q id 100
sudo ip link set dev eth0.2000.100 mtu 8992 upBinding duplicate IP subnets to the default routing table triggers kernel ARP flux and route table corruption. Isolate them cleanly using network namespaces:
# Create isolated routing domains
sudo ip netns add tenant-a
sudo ip netns add tenant-b
# Move customer interfaces into their respective namespaces
sudo ip link set dev eth0.1000.100 netns tenant-a
sudo ip link set dev eth0.2000.100 netns tenant-b
# Configure overlapping IP schemes safely inside each namespace
sudo ip -n tenant-a addr add 192.168.100.1/24 dev eth0.1000.100
sudo ip -n tenant-a link set dev eth0.1000.100 up
sudo ip -n tenant-b addr add 192.168.100.1/24 dev eth0.2000.100
sudo ip -n tenant-b link set dev eth0.2000.100 upBoth tenants now run 192.168.100.0/24 on internal VLAN 100 simultaneously. Their traffic paths never intersect.
5. Persistent Configuration with systemd-networkd
Commands run through ip link vanish upon reboot. Use systemd-networkd to keep your dual-tagged hierarchy permanent.
Define the outer S-TAG netdev in /etc/systemd/network/10-eth0.1000.netdev:
[NetDev]
Name=eth0.1000
Kind=vlan
[VLAN]
Id=1000
Protocol=802.1adDefine the stacked C-TAG netdev in /etc/systemd/network/20-eth0.1000.100.netdev:
[NetDev]
Name=eth0.1000.100
Kind=vlan
[VLAN]
Id=100
Protocol=802.1qBind the outer VLAN to your physical interface in /etc/systemd/network/05-eth0.network:
[Match]
Name=eth0
[Network]
VLAN=eth0.1000
LinkLocalAddressing=noVerification & Troubleshooting
Once interfaces are up, inspect your stack layer by layer before routing customer workloads.
1. Inspect Interface Protocol Flags
Query interface metadata with ip -d link show:
ip -d link show eth0.1000Confirm the protocol field matches 802.1ad:
vlan protocol 802.1ad id 1000 <REORDER_HDR>Check the nested customer interface:
ip -n tenant-a -d link show eth0.1000.100You should see standard 802.1q encapsulation:
vlan protocol 802.1q id 100 <REORDER_HDR>2. Wire-Level Validation with tcpdump
Packet captures reveal the truth. Run tcpdump on the underlying interface with -e to inspect 802.1 header fields directly:
sudo tcpdump -e -n -i eth0 vlanGenerate test traffic from Tenant A’s namespace:
sudo ip netns exec tenant-a ping -c 2 192.168.100.2The capture will show both tags present in the frame header:
02:42:10.198302 52:54:00:12:34:56 > 52:54:00:ab:cd:ef, ethertype 802.1Q-QinQ (0x88a8), vlan 1000, p 0, ethertype 802.1Q (0x8100), vlan 100, p 0, IPv4 (0x0800), length 98: 192.168.100.1 > 192.168.100.2: ICMP echo request, id 412, seq 1The output confirms outer encapsulation via 0x88a8 wrapping the inner 0x8100 tag.
3. Production Traps: Hardware Offload & MTU Blackholes
If packets appear in your outbound capture but remote peers receive nothing, check these two issues first:
- NIC Hardware Offloading: Older 10G/25G controllers (such as Intel 82599ES using the
ixgbedriver or early virtualizedvirtio_netdevices) often fail to parse stacked VLANs. They strip or corrupt the outer0x88a8tag in hardware. Disable hardware VLAN offloading so the Linux kernel handles tag operations directly:
sudo ethtool -K eth0 rxvlan off txvlan off- MTU Blackholes: Small 56-byte ICMP pings might work while larger transfers hang indefinitely. Test this with the Don’t Fragment (DF) bit set:
sudo ip netns exec tenant-a ping -s 1472 -M do 192.168.100.2If that ping times out, an upstream switch port is dropping oversized 1508-byte frames. You have two options. Either increase the switch port MTU to at least 1508 bytes (or 9000 for jumbo frames), or decrease the customer sub-interface MTU to 1492 bytes.
