4G/5G USB Modem Failover on Ubuntu Server with ModemManager and NetworkManager

Table of Contents

The Problem: When Your Fiber Goes Dark at 2 AM

Three times in six months, servers hosting critical services went dark because the upstream fiber connection dropped. Not hardware failure — just ISP issues that took anywhere from 20 minutes to four hours to resolve. Each time, I watched monitoring alerts pile up while users couldn’t reach anything.

After the third outage, I plugged in a Huawei E3372 USB modem with a 4G SIM and spent a weekend getting automatic failover working on Ubuntu 22.04. Six months later, it’s handled seven more fiber disruptions without a single page or manual action needed. That’s the goal: you shouldn’t know a failover happened until you check the logs.

This guide covers exactly what I did: configuring ModemManager and NetworkManager to automatically switch to 4G/5G when the primary connection fails, then switch back cleanly when it recovers.

Core Concepts: How the Failover Stack Works

ModemManager — The Modem Abstraction Layer

ModemManager is a D-Bus system service that handles USB and WWAN modems. It abstracts the messy reality of mobile broadband hardware — different AT command sets, USB modes, carrier initialization sequences — into a clean interface that NetworkManager can talk to.

Without ModemManager, you’d be sending AT commands directly to /dev/ttyUSB0 and hoping your specific modem’s firmware cooperates. With it, you describe what you want and the daemon handles the hardware specifics. HiLink-mode modems are different — the modem presents itself as a USB Ethernet adapter, so NetworkManager handles it like any other network interface and ModemManager never enters the picture.

NetworkManager — Route Metrics as Failover Logic

NetworkManager uses route metrics to decide which interface handles outbound traffic. Lower metric wins. The whole failover mechanism rests on this:

Fiber connection gets metric 100 (primary, wins when both are up)
4G connection gets metric 600 (backup, only used when primary is gone)

When fiber is up, traffic flows through the lower-metric route. When fiber drops, that route disappears from the routing table and traffic falls to the 4G route. Fiber recovers, its route returns with the lower metric, and it takes over again. No scripts. No cron jobs. No manual steps.

Choosing Your Hardware

Not all USB modems work equally well. Three models I’ve tested:

Huawei E3372 (HiLink mode) — Simplest setup, widely available, ~$25–35 used. Presents as USB Ethernet, no ModemManager needed.
ZTE MF833 — Works after switching USB mode with usb_modeswitch, slightly more setup.
Sierra Wireless EM7455 — Excellent Linux support, enterprise-grade, significantly pricier.

The E3372 in HiLink mode is what I’m running in production. You lose some control over modem parameters, but for failover purposes it’s perfect — plug in, get an interface, set the metric, done.

Hands-On: Configuring Automatic 4G Failover

Step 1: Install Required Packages

sudo apt update
sudo apt install -y modemmanager network-manager usb-modeswitch

# Enable and start ModemManager
sudo systemctl enable ModemManager
sudo systemctl start ModemManager

Minimal server installs often don’t run NetworkManager by default. Verify it’s active:

systemctl status NetworkManager

Step 2: Detect Your Modem

Plug in the USB modem, then check if ModemManager sees it:

mmcli -L

Expected output:

/org/freedesktop/ModemManager1/Modem/0 [Huawei Technologies Co., Ltd.] E3372

HiLink-mode modems won’t show up here — that’s expected. Check nmcli device status instead and look for a new ethernet interface. Confirm hardware detection with:

lsusb | grep -i huawei
ip link show
nmcli device status

If the modem appears as a CD-ROM drive (storage mode), usb_modeswitch flips it automatically on replug. It usually just works.

Step 3: Configure the 4G Connection

For modems in proper modem mode (not HiLink), create a GSM connection:

nmcli connection add \
  type gsm \
  ifname "*" \
  con-name "4g-failover" \
  apn "your.carrier.apn" \
  -- \
  connection.autoconnect yes \
  connection.autoconnect-priority -100

Replace your.carrier.apn with your actual APN. Most carriers use internet or web — when in doubt, check your provider’s documentation. Japanese carriers: SoftBank is smile.world, IIJmio is iijmio.jp.

HiLink-mode modems that appear as ethernet interfaces get a DHCP connection auto-created by NetworkManager. Just note the connection name:

nmcli connection show

Step 4: Set Route Metrics for Failover Priority

This is where the actual failover logic lives. High metric on 4G means it only kicks in when the primary fails:

# Set high metric on 4G backup connection
nmcli connection modify "4g-failover" ipv4.route-metric 600
nmcli connection modify "4g-failover" ipv6.route-metric 600

# Set low metric on primary fiber connection
# Replace "Wired connection 1" with your actual connection name
nmcli connection modify "Wired connection 1" ipv4.route-metric 100
nmcli connection modify "Wired connection 1" ipv6.route-metric 100

Apply the changes:

nmcli connection up "4g-failover"
nmcli connection up "Wired connection 1"

Step 5: Verify the Routing Table

ip route show table main | grep default

With both connections active, you should see two default routes:

default via 192.168.1.1 dev eth0 proto dhcp metric 100
default via 192.168.8.1 dev eth1 proto dhcp metric 600

The fiber route wins (metric 100). When it disappears, traffic automatically shifts to the 4G route (metric 600).

Step 6: Enable Connectivity Checking

Link state alone isn’t enough. Your fiber interface can be up while your ISP’s routing is completely broken. Connectivity checking catches this:

sudo nano /etc/NetworkManager/conf.d/connectivity.conf

[connectivity]
uri=http://connectivity-check.ubuntu.com/
interval=60
response=NetworkManager is online

sudo systemctl restart NetworkManager

With this in place, NetworkManager detects lost internet reachability even when the physical link is still up, then triggers failover. On my setup, this cut failover detection time from ~45 seconds down to under 20 seconds.

Step 7: Test Before You Trust It

Start a continuous ping from another machine to a service on your server, then simulate fiber loss:

# On the server — simulate fiber loss
sudo ip link set eth0 down

# Immediately check routing
ip route show table main | grep default
# Should now only show the 4G route

# Restore fiber
sudo ip link set eth0 up

# Verify primary route returns
ip route show table main | grep default

Watch the ping from the remote machine. Expect a gap of 15–45 seconds during failover — that’s the detection and rerouting window. Acceptable for most workloads.

Step 8: Monitor Failover Events

Check NetworkManager logs after any event:

journalctl -u NetworkManager --since "1 hour ago" | grep -E "(up|down|default|route|connectivity)"

A small cron script logs the active interface every minute, making it easy to review failover history:

#!/bin/bash
# /usr/local/bin/check-failover.sh
DEFAULT_IFACE=$(ip route show default | awk 'NR==1{print $5}')
DEFAULT_GW=$(ip route show default | awk 'NR==1{print $3}')
echo "$(date '+%Y-%m-%d %H:%M:%S') - Default: $DEFAULT_IFACE via $DEFAULT_GW" >> /var/log/failover.log

chmod +x /usr/local/bin/check-failover.sh
echo "* * * * * root /usr/local/bin/check-failover.sh" | sudo tee /etc/cron.d/failover-monitor

At month end, one grep eth1 /var/log/failover.log | wc -l tells you exactly how many minutes the system spent on 4G. Handy for tracking data usage and ISP reliability.

What Six Months of Production Use Taught Me

A few things I’d want to know before setting this up:

Failover time is 15–45 seconds. With connectivity checking enabled, you’re usually on the faster end. Stateful connections — SSH sessions, database connections — will drop during this window and need to reconnect.
Failback is clean and automatic. When fiber recovers, NetworkManager re-adds the low-metric route within seconds. No manual steps, no sticky sessions stuck on the wrong interface.
Watch your data usage. My SIM accumulated around 800MB over six months of failover events. On a capped plan, a prolonged outage adds up fast.
Alert on failover, even though it’s automatic. You don’t need to act — but knowing it happened lets you investigate the root cause before it repeats. Wire up a Telegram or email notification from that monitoring script.
HiLink mode is simpler for failover. If your modem supports it, you skip ModemManager entirely and manage two ethernet interfaces. Less to configure, less to debug when something goes wrong.

Need sub-20-second failover? You’re looking at BGP multihoming or active-active load balancing — a different class of complexity. For most small servers and home labs, this metric-based setup hits the right balance: automatic recovery, minimal hardware cost, and nothing to maintain once it’s running.