Stop Guessing Why Your Network Is Slow: Real-Time Traffic Analysis with Akvorado and ClickHouse

Table of Contents

Why SNMP Fails When the Network Breaks at 2 AM

It’s 2 AM, and my phone is vibrating off the nightstand. The core uplink is screaming. I pull up the SNMP graphs in Zabbix, and they confirm the crisis: a flat line at 10Gbps. But SNMP is a blunt instrument. It tells me how much pipe is being used, but it’s silent on who is using it. Is it a backup job that missed its window? A massive DDoS attack? Or just an engineer moving a 5TB database between segments?

In the past, solving this meant logging into individual routers to run manual packet captures or scrape NAT tables. It was slow, tedious, and reactive. To fix this, I needed a system that ingests flow data—the metadata of every single connection—and makes it searchable in seconds. I’ve deployed this specific stack in environments handling over 50,000 flows per second. It doesn’t just work; it stays stable when the network is at its worst.

The Contenders: NetFlow vs. sFlow vs. Packet Capture

Before building the stack, you must choose your data source. Routers and switches generally won’t mirror full packets to a collector because that would melt the CPU. Instead, they send summaries called “flows.”

NetFlow/IPFIX (Stateful): The router caches every connection. When a session ends, it exports a summary containing the Source IP, Destination IP, Port, and total Bytes. It is incredibly accurate for billing but can increase CPU load on older hardware.
sFlow (Sampling): The hardware grabs a snapshot of 1 out of every N packets (for example, 1 out of 2048). Because it’s handled by the switching ASIC, it’s virtually free in terms of CPU overhead. This is the gold standard for high-speed 100Gbps backbones.
Packet Capture (Full): Think tcpdump or Wireshark. While perfect for debugging an application handshake, trying to store full captures for a 10Gbps link would require petabytes of storage.

The Stack: Why Akvorado and ClickHouse?

For a decade, the standard for flow analysis was the ELK stack (Elasticsearch, Logstash, Kibana). However, Elasticsearch is notorious for consuming massive amounts of RAM and struggling with high-cardinality log data. Akvorado flips the script by using ClickHouse as its engine.

ClickHouse is a columnar database built for speed. It can compress flow logs by 10x compared to traditional databases and query billions of rows in under a second. In my experience, a single ClickHouse node can do the work of a five-node Elasticsearch cluster.

Pros and Cons of the Akvorado Ecosystem

The Advantages

Efficiency: It handles high-velocity ingestion on modest hardware. You don’t need 128GB of RAM just to see your traffic spikes.
Batteries Included: Akvorado bundles the ingester, a sleek web UI, and a schema manager into one cohesive package.
Automatic Enrichment: It maps flows to ASNs and GeoIP data out of the box. You can see instantly if your traffic is heading to a local ISP or a data center in Frankfurt.

The Trade-offs

Disk Usage: Even with great compression, keeping 90 days of flow data for a busy network will eventually demand several terabytes of fast NVMe or SSD storage.
Initial Setup: You’ll need to get comfortable with ClickHouse partitions if you plan to scale beyond a handful of devices.

Recommended Setup Architecture

I prefer using Docker Compose for smaller deployments, though the components can be split across multiple servers as traffic grows:

Exporters: Your Cisco, MikroTik, or Juniper hardware configured to send NetFlow (UDP 2055) or sFlow (UDP 6343).
Akvorado Ingester: This service catches the UDP packets, decodes the binary data, adds the GeoIP/BGP context, and writes it to ClickHouse.
ClickHouse: The analytical powerhouse storing your records.
Akvorado Console: The dashboard where you actually run the queries.

Implementation Guide

1. Deploying the Stack

Create a directory for your stack and use this optimized configuration. Note the nofile limits; ClickHouse needs these to handle thousands of simultaneous data parts.

# docker-compose.yml
version: '3.8'
services:
  clickhouse:
    image: clickhouse/clickhouse-server:latest
    container_name: akvorado-clickhouse
    volumes:
      - ./ch-data:/var/lib/clickhouse
    ulimits:
        nofile:
            soft: 262144
            hard: 262144

  akvorado:
    image: akvorado/akvorado:latest
    container_name: akvorado-orchestrator
    ports:
      - "2055:2055/udp" # NetFlow
      - "6343:6343/udp" # sFlow
      - "8080:8080"     # Web UI
    environment:
      - AKVORADO_CLICKHOUSE_SERVERS=clickhouse:9000
      - AKVORADO_INGESTER_LISTEN_NETFLOW=0.0.0.0:2055
      - AKVORADO_INGESTER_LISTEN_SFLOW=0.0.0.0:6343
    depends_on:
      - clickhouse

2. Configuring the Hardware

Point your devices at your new collector. Here is how you do it on two popular platforms.

MikroTik (Traffic Flow)

/ip traffic-flow
set enabled=yes interfaces=ether1 cache-entries=128k
/ip traffic-flow target
add dst-address=192.168.1.50 port=2055 version=9

Cisco (NetFlow v9)

interface GigabitEthernet0/1
 ip flow ingress
 ip flow egress
!
ip flow-export destination 192.168.1.50 2055
ip flow-export version 9
ip flow-cache timeout active 1

3. Analyzing the Results

Fire up the UI at http://<server-ip>:8080. If the packets are reaching the server, the graphs will populate within 60 seconds. My first move is always to check the “Top Talkers.” By grouping by SrcIP and sorting by Bytes, the bandwidth hogs become immediately obvious. If an internal workstation is talking to a strange IP in another country over port 443, you can drill down into the specific timestamps to see if it’s a data breach or just a Windows update.

Lessons from Production at Scale

When I moved this into a production environment with multiple 40Gbps links, I hit a few bottlenecks. Here is how to avoid them:

Tune the Kernel: The Linux kernel will drop UDP packets if the ingester is busy. Run sysctl -w net.core.rmem_max=26214400 to give the system a 25MB buffer.
Be Aggressive with Sampling: Don’t try to sample 1:1 on a 10Gbps link. Use 1:2000 or 1:4000. Akvorado automatically scales the numbers back up so your totals remain accurate.
Set a Retention Policy: ClickHouse will happily eat your entire hard drive. Set Akvorado’s retention to 14 or 30 days to automatically drop old data.

Visibility changes everything. Instead of guessing why the network is slow, you have the evidence needed to fix the problem. It turns a 2 AM nightmare into a 5-minute diagnostic task.