Context & Why: Moving Beyond Static resolv.conf
For decades, Linux relied on a plain text file at /etc/resolv.conf, filled in by DHCP leases or hardcoded nameservers. That simple pattern falls apart in modern infrastructure. Static setups cannot cache query answers locally. They don’t recognize split-horizon routing across VPN tunnels, and they ship every lookup across untrusted networks in cleartext UDP.
systemd-resolved fixes these bottlenecks. It runs as a local stub resolver bound to 127.0.0.53:53. By caching records in RAM, it cuts repeat lookup latencies down to practically zero. It handles DNSSEC validation, steers queries toward interface-specific tunnels (like WireGuard or OpenVPN), and wraps upstream traffic in TLS on port 853.
DNS changes can easily isolate a host. In one staging rollout across 12 production nodes, an untested DoT change choked outbound API lookups and broke our metrics agents within minutes. Always test resolver changes on a canary box before rolling them fleet-wide.
Installation: Ensuring systemd-resolved Runs Cleanly
Distributions like Ubuntu (18.04+) and Debian 12 install systemd-resolved out of the box. Minimal cloud images, LXC containers, or stripped base boxes might leave it out. Install the package if your system lacks the binary:
# Debian/Ubuntu systems
sudo apt-get update
sudo apt-get install systemd-resolved -y
# RHEL/Fedora/Rocky Linux
sudo dnf install systemd-resolved -yStart the daemon and configure it to boot automatically:
sudo systemctl enable --now systemd-resolved.serviceStandard C library lookup routines (glibc getaddrinfo) still inspect /etc/resolv.conf. Point this path directly to systemd’s dynamic stub file with a symlink:
sudo ln -sf /run/systemd/resolve/stub-resolv.conf /etc/resolv.confVerify the link targets the stub resolver:
cat /etc/resolv.confThe file should show nameserver 127.0.0.53 and options edns0 trust-ad.
Configuration: DNS Caching, DoT, and Fallbacks
Default daemon parameters live in /etc/systemd/resolved.conf. Never modify that file directly. Package updates will overwrite your changes. Instead, place modular override files inside /etc/systemd/resolved.conf.d/.
1. Setting Up In-Memory DNS Caching
Local caching saves round-trips over public transit. Shaving 40ms off every microservice API call noticeably reduces cumulative request times. Create a dedicated drop-in file:
sudo mkdir -p /etc/systemd/resolved.conf.d
sudo tee /etc/systemd/resolved.conf.d/01-cache.conf << 'EOF'
[Resolve]
DNSCache=yes
CacheFromLocalhost=no
EOFDNSCache=yes instructs the resolver to store positive and negative lookups in RAM based on authoritative TTL limits. Setting CacheFromLocalhost=no prevents double-caching if you run intermediate proxies like dnsmasq or unbound on the same box.
2. Enforcing DNS-over-TLS (DoT)
Standard DNS sends hostnames across public transit in plain text. Any intermediate router or ISP can read them. DNS-over-TLS secures lookups using TLS over TCP port 853. Specify your upstream resolvers along with their Server Name Indication (SNI) hostnames:
sudo tee /etc/systemd/resolved.conf.d/02-dot.conf << 'EOF'
[Resolve]
DNS=1.1.1.1#cloudflare-dns.com 1.0.0.1#cloudflare-dns.com 2606:4700:4700::1111#cloudflare-dns.com
FallbackDNS=9.9.9.9#dns.quad9.net 149.112.112.112#dns.quad9.net
DNSOverTLS=yes
DNSSEC=allow-downgrade
EOFThree directives keep this setup reliable:
#cloudflare-dns.com: Supplies the SNI hostname. The resolver verifies this against the upstream server’s TLS certificate.DNSOverTLS=yes: Requires TLS. If firewalls filter port 853, queries fail hard instead of falling back to cleartext. Set this toopportunisticif you prefer unencrypted fallback during network blips.DNSSEC=allow-downgrade: Validates DNSSEC signatures when domains provide them, but won’t drop lookup requests on unsigned domains.
Apply the new rules:
sudo systemctl restart systemd-resolvedVerification & Monitoring: Inspecting Cache & Troubleshooting
Never assume a new DNS configuration works simply because the service restarted cleanly. Confirm the running state and live packet flows.
Inspecting Active Upstreams and Status
Run resolvectl status to view link configurations and active nameservers:
resolvectl statusLook at the Global block. Verify that DNSOverTLS: yes is active and your configured upstreams appear in the list.
Testing Latency and Cache Hits
Benchmark cache efficiency with dig:
# First query: cache miss (travels across WAN)
dig @127.0.0.53 kernel.org | grep "Query time"
# Second query: cache hit (pulled straight from local RAM)
dig @127.0.0.53 kernel.org | grep "Query time"The first request typically takes 35ms to 75ms to query the upstream provider over TLS. The second request resolves in 0 msec or 1 msec.
Reviewing Cache Statistics
Inspect cache health with resolvectl statistics:
resolvectl statisticsThis command reports active cache entries, transactions, hit counts, and cache misses. A healthy node often maintains a hit ratio above 70% during normal web and background job operations.
Flushing Stale Records
Purge local records after record updates or infrastructure migrations without bouncing the service:
resolvectl flush-cachesDeep Troubleshooting Scenarios
When lookups fail, trace issues with these targeted checks:
1. Testing Outbound Port 853
Strict DoT requires outbound TCP access on port 853. Cloud security groups, corporate firewalls, and ISP blocks frequently drop this port. Test raw connectivity directly:
nc -zv -w3 1.1.1.1 853If the handshake times out after 3 seconds, loosen firewall egress rules. If you cannot open port 853 upstream, revert DNSOverTLS=yes to DNSOverTLS=opportunistic in your drop-in file.
2. Fixing Overwritten resolv.conf Files
DHCP clients, netplan, and NetworkManager often fight for control over /etc/resolv.conf. Check the inode status:
ls -la /etc/resolv.confIf /etc/resolv.conf is a regular file rather than a symlink to /run/systemd/resolve/stub-resolv.conf, external tools overwrote it. Restore the symlink. Then instruct NetworkManager to delegate DNS control to systemd by adding dns=systemd-resolved under the [main] block in /etc/NetworkManager/NetworkManager.conf.
3. Sniffing Live Query Packets
Monitor resolution logs live with journalctl:
journalctl -u systemd-resolved -f --no-tailTo inspect real network traffic, run tcpdump. Verify queries leave your external interface on TCP port 853 rather than cleartext UDP port 53:
sudo tcpdump -ni any port 853 or port 53Internal apps will query 127.0.0.53:53 locally. External outbound traffic should only target upstream IPs on port 853 wrapped inside TLS sessions.
