Three years ago, a junior engineer on my team accidentally pushed the wrong VLAN configuration to a core switch during business hours. Within minutes, half the office lost connectivity. The fix took two hours — not because the problem was complex, but because we had no idea what the switch config looked like before the change. We were working from memory.
That incident taught me something I tell every new engineer: config backup is not optional. Relying on manual exports is the same as having no backup at all.
The Configuration Loss Problem Nobody Talks About
Most network teams have some version of the same story. Someone makes a change, something breaks, and everyone scrambles to remember what the original settings were. The “backup” turns out to be a six-month-old exported config that nobody touched after the last maintenance window.
The failure modes are broader than most people expect:
- Hardware failures where the replacement device needs to be configured from scratch
- Accidental misconfigurations with no reference point for rollback
- Compliance audits requiring proof of configuration changes over time
- Staff turnover where institutional knowledge walks out the door
Why Manual Backup Scripts Always Break
The instinctive solution is a cron job that SSH’s into each device and saves the running config to a file. This works for the first week. Then someone adds a new switch and forgets to update the script. A password rotates. An SSH key expires. The script fails silently for three months while everyone assumes backups are running fine.
The root cause isn’t laziness. Scripts have no awareness of your device inventory. They can’t handle vendor differences in command syntax. There’s no mechanism for detecting what actually changed between runs. You end up with a pile of timestamped text files and no easy way to answer “what changed on this router between Monday and Tuesday?”
Five switches? Manageable. Fifty devices across Cisco IOS, Juniper JunOS, MikroTik RouterOS, and HP ProCurve? Now you’re maintaining a maintenance tool instead of a backup system.
Comparing Your Options: RANCID vs Scripts vs Oxidized
RANCID (Really Awesome New Cisco confIg Differ) is the classic solution. It has been around since the early 2000s and handles multi-vendor devices with CVS or Subversion for version control. Setup is the real obstacle — it requires heavy Perl configuration, a specific directory structure, and the workflow feels dated next to modern Git tooling.
Ansible playbooks can pull device configs, but they’re designed for pushing changes, not continuous config collection. Running a playbook every hour to grab configs is technically possible but wasteful and operationally awkward.
Oxidized was built specifically for this problem. Written in Ruby, it supports over 100 device types — Cisco, Juniper, MikroTik, Arista, HP ProCurve, and more — connecting via SSH or Telnet with native Git integration. It runs as a daemon, polls your devices on a schedule, and commits any changes it detects directly to a Git repository. That repo becomes your complete, searchable history of every configuration change across your network.
Setting Up Oxidized with Git Backend
Installation
Oxidized requires Ruby 2.5+. On Debian/Ubuntu:
sudo apt update
sudo apt install -y ruby ruby-dev libsqlite3-dev libssl-dev pkg-config cmake libgit2-dev
sudo gem install oxidized oxidized-script oxidized-webOn RHEL/Rocky Linux:
sudo dnf install -y ruby ruby-devel sqlite-devel openssl-devel cmake libgit2-devel
sudo gem install oxidized oxidized-script oxidized-webBase Configuration
Oxidized stores its config in ~/.config/oxidized/config. Create a minimal working config:
---
username: netbackup
password: yourpassword
model: ios
resolve_dns: true
interval: 3600
use_syslog: false
debug: false
pid: "/var/run/oxidized/oxidized.pid"
output:
default: git
git:
single_repo: true
path: /opt/oxidized/configs
source:
default: csv
csv:
file: /etc/oxidized/router.db
delimiter: !ruby/regexp /:/
map:
name: 0
ip: 1
model: 2
username: 3
password: 4
model_map:
cisco: ios
juniper: junos
mikrotik: routerosKey settings to understand:
interval: 3600— poll each device every 3600 seconds (1 hour)output.git.single_repo: true— all device configs stored in one Git repositorysource.csv— device inventory loaded from a plain CSV file
Device Inventory File
Create /etc/oxidized/router.db with your device list. Each line follows the format hostname:ip:model:username:password:
core-sw-01:192.168.1.1:ios:admin:secretpass
core-sw-02:192.168.1.2:ios:admin:secretpass
edge-router:192.168.1.254:junos:netbackup:junpass
mikrotik-ap:192.168.2.1:routeros:admin:mtpassInitializing the Git Repository
Oxidized won’t create the Git repo automatically — initialize it first:
mkdir -p /opt/oxidized/configs
cd /opt/oxidized/configs
git init
git config user.email "[email protected]"
git config user.name "Oxidized Backup"If you want to push to a remote (GitHub, GitLab, or self-hosted Gitea), add it now:
git remote add origin [email protected]:yourorg/network-configs.gitOxidized handles commits locally. Set up a post-commit hook or a separate cron job to push to the remote after each commit.
Running Oxidized as a Service
Start Oxidized manually first to catch connection errors before committing to a background service:
oxidizedTwo issues show up most often at first run. The first is SSH host key acceptance failures — add this to /etc/ssh/ssh_config for your management network range:
Host 192.168.1.*
StrictHostKeyChecking no
UserKnownHostsFile /dev/nullThe second is model detection mismatches, where the device prompt doesn’t match Oxidized’s expected pattern. If a device shows as unreachable but SSH works fine manually, run oxidized -d to see the raw session output and identify the mismatch.
Once the first poll succeeds, convert Oxidized to a systemd service:
sudo useradd -r -s /bin/false oxidized
sudo chown -R oxidized:oxidized /opt/oxidized /etc/oxidized
sudo tee /etc/systemd/system/oxidized.service <<EOF
[Unit]
Description=Oxidized Network Config Backup
After=network.target
[Service]
User=oxidized
ExecStart=/usr/local/bin/oxidized
Restart=on-failure
RestartSec=30
[Install]
WantedBy=multi-user.target
EOF
sudo systemctl enable --now oxidized
sudo systemctl status oxidizedViewing Configuration History in Git
After the first successful poll, the Git repo has one commit per reachable device. Every subsequent poll that detects a change generates a new commit, building a complete audit trail:
cd /opt/oxidized/configs
# All changes to a specific device
git log --oneline -- core-sw-01
# Compare today's config to last week
git diff HEAD~7 HEAD -- core-sw-01
# Full change history with line-level diffs
git log --follow -p -- edge-routerThe diff output shows exactly which lines changed between commits:
-interface GigabitEthernet0/1
- description Old_Uplink
- switchport mode trunk
+interface GigabitEthernet0/1
+ description Uplink_to_Core
+ switchport mode trunk
+ switchport trunk allowed vlan 10,20,30That diff would have saved us two hours on the incident I described at the start.
Securing Credentials in Production
Plaintext passwords in router.db are the most obvious weakness in a basic Oxidized setup. For production environments:
- Create a dedicated read-only backup account on each device with only config-read privileges
- Use environment variables for shared credentials via Oxidized’s
!ruby/object:Oxidized::Envsyntax - Keep the Git repository private if configs contain sensitive IP addressing or SNMP community strings
- Restrict SSH access to the Oxidized server IP using an ACL on each managed device
Creating a minimal read-only account on Cisco IOS:
username netbackup privilege 1 secret StrongBackupPass!
ip access-list standard MGMT-ACL
permit 192.168.100.10
line vty 0 4
login local
access-class MGMT-ACL inAfter One Week of Running Oxidized
Run git log after a week of operation. You’ll almost certainly find changes you didn’t know about — NMS software pushing automatic updates, other engineers making quiet tweaks, configuration drift that accumulated without anyone noticing. That visibility alone justifies the setup time.
The repo also doubles as your audit trail, compliance evidence, and disaster recovery starting point. Instead of asking “what was on that switch before it died?”, run git show HEAD:core-sw-01. Complete answer, under a second.
