When “Good Enough” Monitoring Stops Being Good Enough
Six months ago, our infrastructure team was running a hybrid setup — about 180 network devices across three sites, a mix of Cisco switches, Linux servers, and a handful of legacy Windows boxes. We had Zabbix doing basic uptime checks and a patchy collection of shell scripts that emailed alerts. It worked, until it didn’t.
The breaking point: a core switch spent three days slowly degrading before it died completely. Nobody caught it. The CPU was climbing steadily the whole time, but our monitoring had no historical baseline to compare against — and no automated threshold to fire on “CPU at 75% for 6 consecutive hours.”
That incident forced a real evaluation. Six months of OpenNMS Horizon in production later, I have concrete numbers and hard-won config files worth sharing.
The Root Cause: Monitoring Architecture That Doesn’t Scale
The fundamental problem with most lightweight monitoring stacks isn’t feature gaps — it’s architectural. When you’re polling 200 devices every minute via SNMP, you need:
- Efficient SNMP collection with adaptive scheduling
- A time-series database that doesn’t buckle under the write load
- Auto-discovery that can map topology without manual device entry
- Threshold alerting with hysteresis to avoid alert flapping
Our Zabbix setup was solid for server monitoring. SNMP bulk walks across 6 VLANs were a different story — polling times drifted steadily, and the logs filled with timeout noise. The core issue: Zabbix’s SNMP poller treats network devices as just another host. Fine for 10 devices. At 180, the lack of topology awareness becomes a real operational problem.
Comparing the Alternatives
Before committing to OpenNMS, I ran a three-week evaluation comparing three candidates.
LibreNMS
LibreNMS is excellent for straightforward network device monitoring. SNMP auto-discovery works immediately after install, the UI is clean, and bandwidth graphing is solid. At our scale, though, the PHP-based backend showed latency during discovery runs. Extending alerting logic required more workarounds than I wanted to own long-term.
Zabbix (kept as baseline)
Already in place. Strong for server and application monitoring, noticeably weaker for network topology and SNMP auto-discovery. Device templates require heavy manual configuration, and there’s no native topology discovery model.
OpenNMS Horizon
Java-based, open source, built specifically for network operations centers. Steepest learning curve of the three — the web interface feels dated and configuration is XML-heavy in places. The SNMP collector, though, is genuinely purpose-built. It uses a provisioning daemon (Provisiond) for auto-discovery, a separate polling engine with configurable per-service schedules, and RRD/JRobin for time-series storage. After six months, polling reliability is the best I’ve seen at this scale.
Installing OpenNMS Horizon on Ubuntu 22.04
OpenNMS requires PostgreSQL and Java 17+. Here’s the production-ready install sequence:
Step 1: Install Prerequisites
# Install Java 17
sudo apt update
sudo apt install -y openjdk-17-jdk
# Verify Java version
java -version
# Install PostgreSQL
sudo apt install -y postgresql postgresql-contrib
sudo systemctl enable postgresql
sudo systemctl start postgresqlStep 2: Configure PostgreSQL for OpenNMS
sudo -u postgres psql -c "CREATE USER opennms WITH PASSWORD 'yourpassword';"
sudo -u postgres psql -c "CREATE DATABASE opennms OWNER opennms;"
sudo -u postgres psql -c "GRANT ALL PRIVILEGES ON DATABASE opennms TO opennms;"Step 3: Add the OpenNMS Repository and Install
# Import the GPG key
curl -fsSL https://debian.opennms.org/OPENNMS-GPG-KEY | \
sudo gpg --dearmor -o /usr/share/keyrings/opennms.gpg
# Add the repository
echo "deb [signed-by=/usr/share/keyrings/opennms.gpg] https://debian.opennms.org stable main" | \
sudo tee /etc/apt/sources.list.d/opennms.list
sudo apt update
sudo apt install -y opennms-horizonStep 4: Initialize the Database Schema
# Detect Java runtime
sudo /usr/share/opennms/bin/runjava -s
# Install database schema and IPLIKE function
sudo /usr/share/opennms/bin/install -disThe install -dis flags run database schema installation, install the IPLIKE PostgreSQL extension (used for IP range queries), and seed initial configuration. First run takes 2–3 minutes.
Step 5: Start OpenNMS
sudo systemctl enable opennms
sudo systemctl start opennms
sudo systemctl status opennmsThe web UI listens on port 8980. Default credentials are admin / admin — change these immediately after login via Admin → Change Password.
Configuring SNMP Auto-Discovery
Most monitoring tools at this scale require you to add each device by hand. OpenNMS works differently. Define IP ranges, point the Provisiond daemon at them, and discovery, SNMP classification, and service detection happen automatically — no per-device configuration needed.
Define Discovery Ranges
Edit /etc/opennms/discovery-configuration.xml:
<discovery-configuration xmlns="http://xmlns.opennms.org/xsd/config/discovery"
packets-per-second="1"
initial-sleep-time="30000"
restart-sleep-time="86400000"
retries="1"
timeout="2000">
<include-range retry="1" timeout="2000">
<begin>192.168.10.1</begin>
<end>192.168.10.254</end>
</include-range>
<include-range retry="1" timeout="2000">
<begin>192.168.20.1</begin>
<end>192.168.20.254</end>
</include-range>
</discovery-configuration>Set SNMP Community Strings
Credentials are managed per subnet via the REST API or the Admin → SNMP Configuration screen:
# Set SNMPv2c community string for an entire /24 subnet
curl -u admin:yourpassword -X PUT \
-H "Content-Type: application/xml" \
-d '<snmp-info><community>public</community><version>v2c</version><port>161</port><retries>2</retries><timeout>1800</timeout></snmp-info>' \
http://localhost:8980/opennms/rest/snmpConfig/192.168.10.0/24Restart the discovery daemon and OpenNMS starts pinging the defined ranges, attempting SNMP walks on anything that responds. Each device gets categorized automatically by sysObjectID — Cisco IOS, Juniper JunOS, Linux, Windows — then matched to the appropriate data collection profile. My 180-device network was fully mapped in under five minutes on first run.
Setting Up Performance Collection and Threshold Alerting
This is the piece that would have saved that switch. Thresholds live in /etc/opennms/thresholds.xml and support hysteresis — the rearm value — to prevent alert storms.
CPU Utilization Threshold for Cisco Devices
<group name="cisco" rrdRepository="/var/lib/opennms/rrd/snmp/" ds-type="node">
<!-- Fire if CPU exceeds 80% for 5 consecutive polls (25 minutes at 5-min interval) -->
<threshold type="high"
ds-name="CiscoLocalCPU5SecUtil"
ds-label=""
value="80.0"
rearm="70.0"
trigger="5"
description="CPU utilization high on Cisco device"
triggeredUEI="uei.opennms.org/threshold/highThresholdExceeded"
rearmedUEI="uei.opennms.org/threshold/highThresholdRearmed" />
</group>rearm="70.0" is the hysteresis point — the alert clears only when CPU drops back below 70%, not the moment it briefly dips under 80%. Pair that with trigger="5" (must persist for 5 consecutive polling cycles at 5-minute intervals, so 25 minutes total), and alert flapping drops off sharply. Within the first two weeks, our false-positive count fell by roughly 90%.
Interface Bandwidth Threshold
<group name="mib2-interfaces" rrdRepository="/var/lib/opennms/rrd/snmp/" ds-type="if">
<threshold type="high"
ds-name="ifHCInOctets"
ds-label="ifDescr"
value="800000000"
rearm="700000000"
trigger="3"
description="Interface ingress traffic exceeds 800 Mbps" />
</group>Reload Thresholds Without Restart
# Signal OpenNMS to reload threshold configuration live
/usr/share/opennms/bin/send-event.pl \
uei.opennms.org/internal/eventsConfig/reloadDaemonConfig \
--host localhost \
--parm "daemonName Threshd"Wiring Up Notifications
Notification routing lives in Admin → Configure Notifications. In production, I send critical threshold events to a PagerDuty webhook and lower-priority events to email. The built-in email notifier uses standard SMTP — configure it in /etc/opennms/javamail-configuration.xml.
Before going live, verify the full notification pipeline end-to-end:
# Send a test event to verify notification routing
/usr/share/opennms/bin/send-event.pl \
uei.opennms.org/nodes/nodeLostService \
--host 192.168.10.1 \
--interface 192.168.10.1 \
--service ICMPWhat Six Months of Production Numbers Look Like
Running this configuration across 200+ nodes for six months, here are the real numbers from my setup:
- Initial discovery: 180 devices fully classified in under 5 minutes
- Polling overhead: ~3% CPU on a dedicated 4-core VM with 8 GB RAM
- SNMP data sources: 1,200+ metrics collected every 5 minutes with zero polling gaps
- Alert noise: dropped from ~40 false positives per week to under 5 after threshold tuning
- Uptime: zero OpenNMS service restarts required across 180 days
Honest Caveats Before You Commit
OpenNMS is not the right tool for every team. The XML configuration model has a genuine learning curve — budget a week of documentation reading before the architecture actually clicks. The web UI is functional but won’t win any design awards. Java runtime means 4 GB RAM minimum for a modest deployment; 8 GB for anything beyond 100 nodes.
Small environments or teams without a dedicated ops person will probably be happier with LibreNMS. For infrastructure teams that need SNMP-native monitoring at real scale — reliable collection, topology-aware discovery, threshold alerting without a vendor contract — OpenNMS delivers. Zero unplanned restarts across 180 days is a number that doesn’t need any qualification.
