Deploy Vector on Kubernetes: High-Throughput Log Routing with a Minimal Footprint

Table of Contents

Real-World Problem: The Kubernetes Logging Tax

Last year, our engineering team expanded our production AWS EKS cluster from 15 to 85 worker nodes, supporting roughly 420 microservices. Almost overnight, our monthly AWS compute bill spiked by $3,400. Even worse, node alerts fired continuously during peak traffic hours.

Node profiling pinpointed the culprit: Fluentd. Each agent consumed between 350 MB and 700 MB of RAM, and pinned up to 1.4 CPU cores per node just to tail stdout logs, parse JSON, and push records to Elasticsearch. On c6i.large instances (2 vCPU, 4 GB RAM), our logging stack consumed over 60% of the available CPU. That is an unacceptable infrastructure penalty for background telemetry.

During sudden traffic surges—such as an upstream gateway throwing 18,000 error lines per second—the collector hit Linux cgroup memory limits and died with OOMKilled (exit code 137). Buffered streams backed up on the root volume, disks filled to 90%, and on-call engineers lost visibility precisely when critical incidents occurred.

Root Cause Analysis: Why Traditional Collectors Choke Nodes

Kubernetes logging seems straightforward. The container runtime (containerd or CRI-O) writes container stdout and stderr streams to files on the host at /var/log/pods/*/*/*.log. An agent tails these files, fetches pod metadata from the API server, parses payloads, and forwards the data.

Under heavy write volume, traditional logging engines run into three critical design flaws:

Garbage Collection Pauses: Agents written in Ruby or running on the JVM allocate millions of short-lived string buffers during regex line parsing. When ingestion rates surge past 10,000 lines per second, garbage collection cycles choke CPU threads and stall ingestion pipelines.
API Server Saturation: Unoptimized agents query the Kubernetes API server for every active container to resolve pod labels. Without a local shared cache, these metadata lookups overwhelm both the agent process and the kube-apiserver control plane.
Unbounded Memory Queues: When remote sinks like OpenSearch or S3 slow down, agents hold logs in memory buffers. The buffer expands, trips the container memory limit, and triggers a crash loop that drops data.

Evaluating Collectors: Fluentd vs. Fluent Bit vs. Promtail vs. Vector

Before standardizing our cluster logging, we ran a synthetic benchmark across an isolated three-node Kubernetes cluster. We generated 50,000 JSON log events per second (averaging 512 bytes per event) across all candidates:

Collector	Runtime	Avg Memory / Node	Transform Engine	End-to-End Throughput
Fluentd	Ruby / C	420 MB – 680 MB	Ruby plugins & regex	14,000 events/sec (CPU-bound)
Promtail	Go	110 MB – 210 MB	Pipeline stages	26,000 events/sec (Loki-only)
Fluent Bit	C	28 MB – 55 MB	Lua scripts & C parsers	48,000 events/sec
Vector	Rust	35 MB – 65 MB	Vector Remap Language (VRL)	50,000+ events/sec

Fluent Bit performed well, but its routing logic and complex payload manipulations depend heavily on custom Lua scripts. These scripts quickly became fragile and difficult to test across our distributed teams. Promtail integrated neatly with Grafana Loki, but it offered no clean mechanism for multi-sink fan-out, such as archiving compliance audit logs to Amazon S3 while sending application errors to OpenSearch.

Vector delivered the balance we needed. Written in Rust, it provides raw native performance and memory safety without GC latency. Its built-in Vector Remap Language (VRL) offers an expressive, type-safe DSL for parsing, enriching, and scrubbing records on the fly.

The Architecture: Two-Tier Topologies

Do not run complex JSON parsing, regex redactions, and multi-cloud uploads directly inside edge DaemonSets. Instead, split your logging infrastructure into two distinct tiers:

Tier 1 — Vector DaemonSet (Agent): A lightweight collector deployed on every node. It mounts /var/log/pods, enriches streams with local pod metadata, applies zero complex transforms, and ships raw logs over gRPC to the internal aggregator.
Tier 2 — Vector Deployment (Aggregator): A stateless pool of Vector pods running behind a Kubernetes Service with a PodDisruptionBudget. This aggregator handles heavy processing: parsing JSON, scrubbing credit card numbers and bearer tokens, dropping routine health probes, and fanning out data to S3 and Loki.

This design keeps the per-node footprint locked under 50 MB, while horizontal pod autoscaling allows the aggregator pool to scale effortlessly with cluster-wide traffic spikes.

Step 1: Deploying the Vector DaemonSet

Add Vector’s official Helm repository, then create a minimal configuration file named values-daemonset.yaml:

role: "Agent"

resources:
  requests:
    cpu: 50m
    memory: 64Mi
  limits:
    cpu: 200m
    memory: 128Mi

customConfig:
  sources:
    kubernetes_logs:
      type: kubernetes_logs
      auto_partial_merge: true

  transforms:
    add_cluster_meta:
      type: remap
      inputs:
        - kubernetes_logs
      source: |
        .cluster = "production-us-east-1"

  sinks:
    to_aggregator:
      type: vector
      inputs:
        - add_cluster_meta
      address: "vector-aggregator.logging.svc.cluster.local:6000"
      version: "2"

Deploy the DaemonSet into your cluster:

helm repo add vector https://helm.vectordotdev.com
helm repo update
helm upgrade --install vector-agent vector/vector \
  --namespace logging \
  --create-namespace \
  -f values-daemonset.yaml

Step 2: Deploying the Central Aggregator

The aggregator manages CPU-intensive tasks. It uses VRL to parse JSON messages, discards synthetic health check requests (such as /healthz and /ready), strips Authorization headers, and routes audit events to S3 while sending application telemetry to Loki.

Create your configuration as values-aggregator.yaml:

role: "Stateless-Aggregator"

replicas: 3

podDisruptionBudget:
  minAvailable: 2

resources:
  requests:
    cpu: 250m
    memory: 256Mi
  limits:
    cpu: 1000m
    memory: 512Mi

service:
  ports:
    - name: vector-grpc
      port: 6000
      targetPort: 6000
      protocol: TCP

customConfig:
  sources:
    from_agents:
      type: vector
      address: "0.0.0.0:6000"
      version: "2"

  transforms:
    parse_and_clean:
      type: remap
      inputs:
        - from_agents
      source: |
        # Parse JSON payloads into a structured map
        parsed, err = parse_json(.message)
        if err == null {
          .payload = parsed
          del(.message)
        }

        # Drop synthetic readiness and liveness probes
        req_path = to_string(.payload.path) ?? ""
        if contains(req_path, "/healthz") || contains(req_path, "/ready") {
          abort
        }

        # Redact authentication credentials
        if !is_null(.payload.authorization) {
          .payload.authorization = "[REDACTED]"
        }

    route_logs:
      type: route
      inputs:
        - parse_and_clean
      route:
        security_audit: '.kubernetes.pod_namespace == "kube-system"'
        app_traffic: '.kubernetes.pod_namespace != "kube-system"'

  sinks:
    s3_cold_storage:
      type: aws_s3
      inputs:
        - route_logs.security_audit
      bucket: "company-prod-audit-logs"
      region: "us-east-1"
      encoding:
        codec: json
      batch:
        max_bytes: 10485760
        timeout_secs: 300

    loki_apps:
      type: loki
      inputs:
        - route_logs.app_traffic
      endpoint: "http://loki-gateway.monitoring.svc.cluster.local"
      encoding:
        codec: json
      labels:
        namespace: "{{ kubernetes.pod_namespace }}"
        pod: "{{ kubernetes.pod_name }}"
        container: "{{ kubernetes.container_name }}"

Apply the aggregator deployment to the logging namespace:

helm upgrade --install vector-aggregator vector/vector \
  --namespace logging \
  -f values-aggregator.yaml

Step 3: Verification and Live Inspection

Confirm that your agent pods are active across all nodes and communicating with the aggregator service:

# Verify DaemonSet readiness across all nodes
kubectl get daemonset -n logging

# Inspect aggregator logs for active gRPC streams
kubectl logs -n logging -l app.kubernetes.io/name=vector -c vector --tail=50

# Launch Vector's built-in top console
kubectl exec -it -n logging deploy/vector-aggregator -- vector top

The vector top utility launches an interactive console directly inside your terminal. It displays live throughput metrics, component-level processing times, and downstream buffer saturations, helping you identify backpressure issues before they impact production.

Key Takeaways

Decoupling log collection from data transformation resolves node resource exhaustion. By restricting edge DaemonSets to simple log ingestion and shifting parsing to a dedicated aggregator tier, we cut worker node logging memory from 700 MB down to 45 MB per node. This shift freed up dozens of CPU cores across our cluster and stopped log-related OOM kills permanently.