Posted inNetworking

Real-Time Network Monitoring with Streaming Telemetry (gNMI), Telegraf, and InfluxDB

Networking tutorial - IT technology blog
Networking tutorial - IT technology blog

Real-World Problem: The 5-Minute Blind Spot of SNMP Polling

Last quarter, our primary SIP trunk started dropping calls randomly across three regional offices. Mean Opinion Score (MOS) values plummeted from a crisp 4.3 down to 1.8. Branch workers heard garbled audio every 20 minutes, yet our monitoring dashboards showed calm, steady green gauges. Average interface utilization hovered at 42%, core switch CPU sat at 18%, and interface error counters read zero.

Everything looked normal on paper. The reality on the wire was ugly.

After tapping the uplink with an optical splitter and running a hardware capture, the culprit jumped out immediately. Back-to-back database replication jobs were generating 450-millisecond bursts at full 100 Gbps line rate. These microbursts overwhelmed the switch’s 32 MB shared packet buffer, silently dumping RTP voice packets. Our Network Management Station (NMS) missed every single spike because it was polling interface statistics via SNMP every 300 seconds.

SNMP simply averages reality away. A 100G port transmits over 140 million packets every single second. When an NMS takes one sample every 5 minutes, it flattens transient traffic surges into a smooth, harmless curve. The microbursts vanish from your graphs, but your users still feel the fallout.

Root Cause Analysis: Why Pull-Based SNMP Fails Modern Networks

Traditional monitoring tools cannot keep up with high-speed fabrics. The underlying architecture of SNMP imposes hard physical limits that cannot be solved by simply lowering polling intervals.

1. The Inefficiency of Pull-Based Architecture

SNMP depends on a central collector sending repetitive UDP GetRequest queries across thousands of Object Identifiers (OIDs). Consider the overhead on a standard 48-port top-of-rack leaf switch:

  • The collector issues hundreds of OID requests every poll cycle just to gather basic byte and packet counts.
  • The switch route processor interrupts its control loop, context-switches, walks the in-memory Management Information Base (MIB), formats the response, and sends UDP packets back.
  • On modular switches with hundreds of physical and logical interfaces, SNMP walks push control-plane CPU above 40%. In severe cases, this starves routing protocols like BGP and OSPF of CPU time.

2. The Nyquist Problem and Incast Bursts

Polled metrics provide delta values between two arbitrary points in time, $T_1$ and $T_2$. Suppose an interface discards 14,000 packets during a 200 ms burst at second 12, then remains quiet for the remaining 299 seconds.

An SNMP query taken at second 300 distributes those 14,000 drops across 300 seconds, showing a trivial average drop rate of 46 packets per second. You cannot detect tail drops or TCP incast congestion without dropping polling timers to sub-second levels—a rate that immediately crashes legacy SNMP engines.

3. Brittle, Inconsistent MIB Trees

Vendor MIB structures are notoriously messy. Extracting transceiver optical power (DOM), queue-depth telemetry, or BGP peer states across an estate running Cisco, Arista, Juniper, and Linux SONiC requires maintaining bespoke OID lookup dictionaries. If an operating system upgrade shifts an enterprise OID branch, dashboards break without warning.

Solutions Compared: SNMP vs. Webhooks vs. Network Streaming Telemetry

Before overhauling our telemetry pipeline, we evaluated three production data collection models:

Attribute SNMPv2c / SNMPv3 HTTP Webhooks / Event Scripts gNMI / gRPC Streaming Telemetry
Mechanism Pull (Periodic Polling via UDP) Push (Reactive HTTP POST per event) Push (Persistent streaming via HTTP/2 over gRPC)
Cadence Coarse (60s – 300s) Event-driven only (state changes) Sub-second (continuous 1s or on-change)
Data Schema SMIv2 MIBs (OIDs) Unstructured JSON / Syslog Structured YANG (OpenConfig, IETF)
Control-Plane Impact High (software MIB walks) Moderate (forks local scripts) Negligible (offloaded to internal cache)
Transport Security Plaintext (v2c) or clunky USM (v3) TLS (HTTPS) Native mutual TLS (mTLS) with token auth

Webhooks work well for low-frequency alerts like a link failure. They collapse completely under millions of continuous interface counter updates. For real-time metrics, gNMI (gRPC Network Management Interface) is the uncontested industry standard.

The Architecture: gNMI, Telegraf, and InfluxDB

Streaming telemetry flips the monitoring paradigm on its head. Instead of an external server repeatedly asking for counters, the switch pushes data automatically over a persistent HTTP/2 connection. Counters stream directly out of internal state databases at microsecond resolution.

Our production pipeline uses three core layers:

  1. gNMI Network Nodes: Cisco IOS-XE and Arista EOS switches publishing OpenConfig YANG data paths.
  2. Telegraf: A lightweight Go daemon connecting to switches via the inputs.gnmi plugin to collect and parse protobuf streams.
  3. InfluxDB v2: A dedicated time-series engine storing metrics with downsampling and nanosecond precision.

Step 1: Enabling the gNMI Server on Switches

Most modern switch operating systems run a native gNMI daemon. Here is how to expose the gNMI service on standard enterprise gear.

On Cisco IOS-XE (17.3+), activate the secure gRPC server for incoming dial-in subscriptions:

! Enable cryptographic keys and gRPC endpoint
netconf-yang
!
gnmi
  port 57400
  secure-port 57400
  secure-trustpoint Telemetry-mTLS
!

On Arista EOS (4.25+), enable the Octa/gNMI provider under the management API context:

management api gnmi
   transport grpc default
      port 57400
      ssl profile Telemetry-TLS-Profile
   provider eos-native
   no shutdown
!

Step 2: Deploying InfluxDB and Telegraf with Docker

Containerizing the collector and database keeps deployment simple and reproducible. Here is the operational docker-compose.yml:

version: '3.8'

services:
  influxdb:
    image: influxdb:2.7
    container_name: influxdb
    restart: unless-stopped
    ports:
      - "8086:8086"
    environment:
      - DOCKER_INFLUXDB_INIT_MODE=setup
      - DOCKER_INFLUXDB_INIT_USERNAME=netadmin
      - DOCKER_INFLUXDB_INIT_PASSWORD=ChangeThisPasswordNow123!
      - DOCKER_INFLUXDB_INIT_ORG=infrastructure
      - DOCKER_INFLUXDB_INIT_BUCKET=telemetry_raw
      - DOCKER_INFLUXDB_INIT_RETENTION=14d
      - DOCKER_INFLUXDB_INIT_ADMIN_TOKEN=SuperSecretTelegrafToken123!
    volumes:
      - influxdb_data:/var/lib/influxdb2

  telegraf:
    image: telegraf:1.30
    container_name: telegraf
    restart: unless-stopped
    depends_on:
      - influxdb
    network_mode: "host"
    volumes:
      - ./telegraf.conf:/etc/telegraf/telegraf.conf:ro
      - ./certs:/etc/telegraf/certs:ro

volumes:
  influxdb_data:

Step 3: Configuring Dial-In gNMI Subscriptions in Telegraf

In dial-in mode, Telegraf connects outbound to switch port 57400 and requests specific YANG data paths. This avoids running custom scripts on each network box.

Add the following target blocks to your telegraf.conf:

[agent]
  interval = "1s"
  round_interval = true
  metric_batch_size = 5000
  metric_buffer_limit = 50000
  collection_jitter = "0s"
  flush_interval = "1s"
  flush_jitter = "0s"
  logfile = ""

[[outputs.influxdb_v2]]
  urls = ["http://127.0.0.1:8086"]
  token = "SuperSecretTelegrafToken123!"
  organization = "infrastructure"
  bucket = "telemetry_raw"

# Input: Dial-in gNMI client against leaf switches
[[inputs.gnmi]]
  addresses = ["10.10.1.1:57400", "10.10.1.2:57400"]
  username = "telemetry_svc"
  password = "ServiceAccountPass123!"
  redial = "10s"
  encoding = "proto"

  # Adjust TLS verification according to your PKI setup
  insecure_skip_verify = false
  tls_ca = "/etc/telegraf/certs/ca.pem"
  tls_cert = "/etc/telegraf/certs/telegraf.crt"
  tls_key = "/etc/telegraf/certs/telegraf.key"

  # Subscription 1: 1-second sampled interface counters
  [[inputs.gnmi.subscription]]
    name = "interfaces"
    origin = "openconfig-interfaces"
    path = "/interfaces/interface/state/counters"
    subscription_mode = "sample"
    sample_interval = "1s"

  # Subscription 2: Event-driven port operational changes
  [[inputs.gnmi.subscription]]
    name = "interface_oper_status"
    origin = "openconfig-interfaces"
    path = "/interfaces/interface/state/oper-status"
    subscription_mode = "on_change"

Step 4: Launching and Verifying the Stream

Start the telemetry collector stack in the background:

docker compose up -d

Tail the collector log output to verify that your TLS handshakes and YANG path subscriptions succeeded:

docker logs -f telegraf

You should see clean connection messages without schema parsing errors:

2026-10-14T12:00:01Z I! [inputs.gnmi] Connected [10.10.1.1:57400]
2026-10-14T12:00:01Z I! [inputs.gnmi] Connected [10.10.1.2:57400]
2026-10-14T12:00:02Z I! [inputs.gnmi] Subscription initiated for openconfig-interfaces:/interfaces/interface/state/counters

Step 5: Querying Granular Metrics in InfluxDB

Open the InfluxDB UI at http://<collector-ip>:8086. To calculate packet throughput and watch for microbursts on an individual 100G uplink, run this Flux query:

from(bucket: "telemetry_raw")
  |> range(start: -15m)
  |> filter(fn: (r) => r["_measurement"] == "interfaces")
  |> filter(fn: (r) => r["_field"] == "in_octets" or r["_field"] == "in_discards")
  |> filter(fn: (r) => r["interface_name"] == "HundredGigE1/0/1")
  |> derivative(unit: 1s, nonNegative: true)
  |> yield(name: "real_time_rate")

With 1-second sample intervals, the difference is immediate. The 450 ms replication burst that previously hid behind a flat 42% average line now renders as an explicit 98 Gbps spike, followed immediately by 2,300 input discards. Root cause identification drops from four hours of finger-pointing to 10 seconds of visual analysis.

Field Lessons and Practical Recommendations

Switching from pull polling to push telemetry cut our switch CPU utilization from 28% down to 2.4%. It also caught three silent packet-drop bugs in our spine switches within two weeks. To keep your pipeline reliable as it scales, follow these practical rules:

  • Use on_change subscriptions for state flags: Never poll routing states or link operational status on a 1-second timer. Subscribing with on_change mode eliminates empty samples. You get notified within milliseconds when a BGP session drops, saving database storage and network bandwidth.
  • Tier your sampling rates sensibly: High-priority interface packet counts and buffer discards justify 1-second updates. Environmental sensors, chassis temperatures, and optical transceivers (DOM) only drift over minutes; stream those every 30 or 60 seconds.
  • Harden switch management ports: Lock down TCP port 57400 via hardware control-plane policing (CoPP) and Access Control Lists. Restrict telemetry connections strictly to collector IP addresses, and isolate telemetry user accounts to read-only roles.

SNMP served its purpose for thirty years. Modern cloud networks need modern tooling. Combining gNMI, Telegraf, and InfluxDB gives you instant, sub-second visibility across every port, without taxing your switch control plane.

Share:
Tags: