Posted inProgramming

Building High-Performance eBPF Network Metrics in Go with cilium/ebpf

Programming tutorial - IT technology blog
Programming tutorial - IT technology blog

Six months ago, our Prometheus node-exporter setup hit a wall. Across our 2,500-node Kubernetes fleet, user-space observability agents were burning nearly 15% of total CPU just parsing /proc/net/tcp and sampling socket buffers under a 1.2M req/sec peak. We scrapped those pollers and deployed custom eBPF probes managed by Go using cilium/ebpf. Observability CPU usage dropped to under 0.8% immediately.

Kernel-level tracing delivers wire-speed socket metrics without kernel modifications or fragile out-of-tree kernel modules. Below is a production blueprint for compiling, loading, and streaming eBPF network telemetry in Go.

Quick Start (5 Min)

Forget embedding heavyweight toolchains like LLVM or Clang inside production containers. Cilium’s bpf2go utility compiles C source to eBPF bytecode ahead of time during go generate, outputting clean, type-safe Go bindings.

1. Install Prerequisites

You need a 5.15+ kernel for standard ring buffer support, plus the core build tools:

sudo apt-get update && sudo apt-get install -y clang llvm libbpf-dev linux-headers-$(uname -r)
go install github.com/cilium/ebpf/cmd/bpf2go@latest

2. Write the eBPF Kernel C Code

Save this as kprobe_tcp.c. We attach to tcp_v4_connect, which receives the destination IP and port inside the struct sockaddr argument before the socket connects:

//go:build ignore

#include <linux/bpf.h>
#include <linux/in.h>
#include <linux/ptrace.h>
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>
#include <bpf/bpf_core_read.h>
#include <bpf/bpf_endian.h>

struct event {
    __u32 saddr;
    __u32 daddr;
    __u16 dport;
};

struct {
    __uint(type, BPF_MAP_TYPE_RINGBUF);
    __uint(max_entries, 16 * 1024 * 1024); // 16 MB ring buffer
} events SEC(".maps");

SEC("kprobe/tcp_v4_connect")
int BPF_KPROBE(tcp_v4_connect, struct sock *sk, struct sockaddr *uaddr) {
    struct sockaddr_in *usin = (struct sockaddr_in *)uaddr;
    struct event *e;

    e = bpf_ringbuf_reserve(&events, sizeof(*e), 0);
    if (!e) {
        return 0; // Buffer saturated; drop to protect kernel throughput
    }

    BPF_CORE_READ_INTO(&e->daddr, usin, sin_addr.s_addr);
    BPF_CORE_READ_INTO(&e->dport, usin, sin_port);
    BPF_CORE_READ_INTO(&e->saddr, sk, __sk_common.skc_rcv_saddr);

    bpf_ringbuf_submit(e, 0);
    return 0;
}

char LICENSE[] SEC("license") = "Dual BSD/GPL";

3. Generate Bindings and Stream Events

Create main.go. The //go:generate line compiles our C code for little-endian architectures into an embedded object file:

package main

//go:generate bpf2go -target bpfel kprobeTCP kprobe_tcp.c

import (
	"bytes"
	"encoding/binary"
	"errors"
	"fmt"
	"log"
	"net"
	"os"
	"os/signal"
	"syscall"

	"github.com/cilium/ebpf/link"
	"github.com/cilium/ebpf/ringbuf"
	"github.com/cilium/ebpf/rlimit"
)

type tcpEvent struct {
	Saddr uint32
	Daddr uint32
	Dport uint16
}

func main() {
	// Kernels <5.11 require raising locked memory limits
	if err := rlimit.RemoveMemlock(); err != nil {
		log.Fatalf("Removing memlock failed: %v", err)
	}

	objs := kprobeTCPObjects{}
	if err := loadKprobeTCPObjects(&objs, nil); err != nil {
		log.Fatalf("Loading eBPF objects failed: %v", err)
	}
	defer objs.Close()

	kp, err := link.Kprobe("tcp_v4_connect", objs.TcpV4Connect, nil)
	if err != nil {
		log.Fatalf("Attaching kprobe failed: %v", err)
	}
	defer kp.Close()

	rd, err := ringbuf.NewReader(objs.Events)
	if err != nil {
		log.Fatalf("Opening ringbuf failed: %v", err)
	}
	defer rd.Close()

	log.Println("Probe attached. Streaming outbound TCP connections...")

	stop := make(chan os.Signal, 1)
	signal.Notify(stop, os.Interrupt, syscall.SIGTERM)

	go func() {
		var event tcpEvent
		for {
			record, err := rd.Read()
			if err != nil {
				if errors.Is(err, ringbuf.ErrClosed) {
					return
				}
				log.Printf("Ringbuf read error: %v", err)
				continue
			}

			if err := binary.Read(bytes.NewBuffer(record.RawSample), binary.LittleEndian, &event); err != nil {
				continue
			}

			dstIP := make(net.IP, 4)
			binary.LittleEndian.PutUint32(dstIP, event.Daddr)
			// Convert port from network byte order
			port := (event.Dport>>8) | (event.Dport<<8)
			fmt.Printf("Connect -> %s:%d\n", dstIP, port)
		}
	}()

	<-stop
	log.Println("Shutting down collector...")
}

Run go generate && sudo go run .. Open another terminal, issue any curl https://example.com request, and watch the event appear instantly.

Architecture & Internals

Why replace BCC? The difference comes down to runtime footprint and pipeline determinism.

Pure Go vs. BCC

BCC compiles probe source code on the fly using an embedded LLVM compiler on every individual target machine. In high-density environments, that creates serious operational headaches:

  • Worker nodes must install heavy kernel development headers and LLVM packages. This bloats base container images by 380MB to 500MB.
  • Just-in-time compilation spikes CPU usage to 100% on a core for several seconds when pods cycle.
  • Minor glibc or Clang library variations cause compilation errors across different base images.

Static compilation solves this entirely. bpf2go builds the ELF bytecode during your standard CI step. At startup, the Go runtime reads the ELF binary, parses BTF (BPF Type Format) tables, and loads programs straight through the bpf() syscall. No CGO. No shared system libraries. Zero compilation delay on boot.

Kernel-to-User Space Ring Buffers

Probes execute inside an isolated in-kernel virtual machine. They cannot write directly to process heap memory or issue arbitrary system calls. Instead, data crosses boundaries via BPF Maps.

Here is the communication pipeline for every connection attempt:

  1. Probe Trigger: The kernel runs tcp_v4_connect and diverts to our loaded bytecode in roughly 15 to 30 nanoseconds.
  2. Zero-Copy Reservation: bpf_ringbuf_reserve secures memory directly in a shared ring buffer. If the buffer fills up, it drops the record rather than stalling network packet processing.
  3. Unified Memory Architecture: Older PERF_EVENT_ARRAY structures allocated individual per-CPU buffers. If core 0 sat idle while core 15 handled network interrupts, core 0’s buffer sat empty while core 15 dropped packets. Modern BPF_MAP_TYPE_RINGBUF shares a single multi-producer buffer across all cores, cutting RAM consumption from 64MB down to 16MB per daemon.
  4. Epoll Wakeups: Go’s ringbuf reader sleeps on an epoll event descriptor until the kernel signals new data, keeping user-space CPU at 0% when idle.

Advanced Usage: In-Kernel Latency Aggregation

Streaming every packet event to user space can easily overwhelm Go channels during traffic surges. If your node handles 80,000 requests every second, aggregate latency directly inside kernel maps first.

Tracking Handshake Latency with Hash Maps

To measure connect duration, pair a kprobe with a kretprobe. Store the start timestamp keyed by the unique thread ID (bpf_get_current_pid_tgid()), then calculate total elapsed time on function return:

struct {
    __uint(type, BPF_MAP_TYPE_HASH);
    __uint(max_entries, 32768);
    __type(key, __u64);
    __type(value, __u64);
} start_times SEC(".maps");

SEC("kprobe/tcp_v4_connect")
int BPF_KPROBE(trace_connect_entry) {
    __u64 pid_tgid = bpf_get_current_pid_tgid();
    __u64 ts = bpf_ktime_get_ns();
    bpf_map_update_elem(&start_times, &pid_tgid, &ts, BPF_ANY);
    return 0;
}

SEC("kretprobe/tcp_v4_connect")
int BPF_KRETPROBE(trace_connect_exit, int ret) {
    __u64 pid_tgid = bpf_get_current_pid_tgid();
    __u64 *start_ts = bpf_map_lookup_elem(&start_times, &pid_tgid);

    if (start_ts) {
        if (ret == 0) {
            __u64 latency_ns = bpf_ktime_get_ns() - *start_ts;
            // Feed latency into an in-kernel BPF_MAP_TYPE_HISTOGRAM or counter
            (void)latency_ns;
        }
        bpf_map_delete_elem(&start_times, &pid_tgid);
    }
    return 0;
}

Always key entry-to-return probes by bpf_get_current_pid_tgid() rather than relying on input parameters in a kretprobe. Return probes cannot reliably inspect entry arguments because CPU registers get repurposed during function execution.

Ensuring Portability with CO-RE

Kernel struct layouts change across Linux releases. Field offsets for struct sock in kernel 5.4 do not match those in 6.2. Without relocations, your probe extracts corrupted memory or gets rejected outright by the verifier.

CO-RE (Compile Once – Run Everywhere) eliminates this failure mode. Generate a comprehensive vmlinux.h header from any host:

bpftool btf dump file /sys/kernel/btf/vmlinux format c > vmlinux.h

Macros like BPF_CORE_READ() record field relocations into BTF sections. When cilium/ebpf loads your program, it checks the live host’s /sys/kernel/btf/vmlinux and rewrites byte offsets on the fly. One static Go binary runs on Ubuntu 20.04, Debian 12, and Amazon Linux 2023 without rebuilds.

Production Pitfalls to Avoid

Rolling eBPF across thousands of machines quickly exposes low-level constraints that standard tutorials gloss over.

1. Mind the 512-Byte Stack

The kernel verifier allocates an unforgiving 512-byte stack per probe. Declaring a char path[256] alongside an event struct will instantly crash your build with look like stack overflow. When assembling payloads larger than 256 bytes, allocate them inside a per-CPU scratch array map (BPF_MAP_TYPE_PERCPU_ARRAY) or reserve space directly in the ring buffer.

2. Struct Alignment and Go Deserialization

C compilers insert hidden padding bytes to satisfy 4-byte and 8-byte word alignments. If your C struct does not match your Go binary decoding layout, metrics will shift into the wrong fields. Order struct members strictly by descending size, and inspect memory layouts with pahole:

// Optimized layout: 24 bytes, 0 internal padding gaps
struct event {
    __u64 timestamp_ns; // 8 bytes (offset 0)
    __u32 saddr;        // 4 bytes (offset 8)
    __u32 daddr;        // 4 bytes (offset 12)
    __u16 dport;        // 2 bytes (offset 16)
    __u8  protocol;     // 1 byte  (offset 18)
    __u8  pad;          // 1 byte  (offset 19) to complete 4-byte alignment
    __u32 reserved;     // 4 bytes (offset 20) pads struct to 24 bytes total
};

3. Bypassing Root with Linux Capabilities

Do not run your Go monitoring daemons as full root. Since Linux 5.8, grant granular capabilities to your binary using setcap:

sudo setcap cap_bpf,cap_perfmon,cap_net_admin+ep ./network-collector

Keep in mind that Linux 5.11 switched map accounting to memory cgroups. On older 5.4 or 5.8 kernels, you must still invoke rlimit.RemoveMemlock() before loading maps. The default 64KB locked-memory ceiling will fail immediately when initializing any practical map.

Combining cilium/ebpf with Go’s lightweight concurrency gives you kernel-level precision at microscopic CPU cost. You get deep network visibility while leaving host compute where it belongs: serving user workloads.

Share:
Tags: