Posted inDatabase

Running InfluxDB on Docker: Resilient Time-Series Monitoring for High-Rate IoT Feeds

Database tutorial - IT technology blog
Database tutorial - IT technology blog

Real-World Problem: When Relational Databases Choke on Sensor Streams

Last year, I helped a manufacturing client architect an ingestion pipeline for 500 industrial IoT vibration and thermal probes. Each unit reported temperature, tri-axial vibration, and power draw every two seconds. That added up to 250 payloads per second—roughly 21.6 million database rows hitting disk every 24 hours.

We initially piped the stream straight into a tuned PostgreSQL 14 instance on an 8-core, 32 GB RAM virtual machine. For ten days, performance held steady. By week three, queries aggregating 30-day rolling averages hit a wall and began timing out after 60 seconds.

Disk I/O slammed against 98% saturation. The table’s composite B-Tree indexes ballooned past 45 GB—far larger than the available RAM. Buffer cache misses surged, incoming writes backed up in Redis, and our ingestion workers finally died of memory exhaustion.

Postgres is an exceptional engine. So are MySQL and MongoDB. But dumping continuous, append-only sensor telemetry into a general-purpose relational store triggers structural disk bottlenecks that index tuning simply cannot solve.

Root Cause Analysis: Why Traditional Engines Choke on Stream Writes

The core issue lies in how general-purpose engines arrange data on disk:

  • Index Maintenance Thrashing: Every incoming insert updates table indexes. Once table and index sizes exceed RAM, the storage layer spends its time reading and updating B-Tree pages scattered across disk. Throughput falls off a cliff.
  • Row-Oriented Amplification: Relational systems write full row tuples contiguously. To compute a 6-hour average temperature from warehouse sensors, the engine must scan megabytes of irrelevant vibration, wattage, and status columns off disk. This layout creates massive memory and bus overhead.
  • Painful Purges: Evicting sensor records past a 90-day retention window usually means running batched DELETE statements. In Postgres or MySQL, deleting 50 million rows leaves dead tuples behind. That causes table fragmentation, balloons the Write-Ahead Log (WAL), and forces aggressive autovacuum cycles that choke the write path.

Architecture Showdown: General Purpose vs. Purpose-Built Time-Series

Teams handling metric streams usually weigh three options:

1. Relational Databases (PostgreSQL / TimescaleDB)

Vanilla Postgres provides rock-solid ACID guarantees and relational joins, but bare tables require aggressive manual partitioning (declarative range partitioning by time). TimescaleDB fixes many of these issues via hypertables. However, you still pay a notable memory tax for write buffers and indexing structures.

2. Document Stores (MongoDB)

MongoDB scales writes horizontally and handles sparse or polymorphic payloads without upfront migrations. Even so, running temporal downsampling requires custom aggregation pipelines or background workers. These jobs compete directly with ingestion threads for CPU and WiredTiger cache space.

3. Purpose-Built Time-Series Engines (InfluxDB)

InfluxDB uses a Time-Structured Merge Tree (TSM) engine paired with columnar disk storage. It orders data strictly by timestamp and compresses values using delta-of-delta and Gorilla float encoding. In our testing, raw disk footprints dropped by 72% compared to Postgres. Writes append sequentially to a WAL, skipping random B-Tree seeks entirely. When data ages out, InfluxDB drops whole immutable file shards off the filesystem in milliseconds—zero vacuuming required.

Step-by-Step: Running InfluxDB v2 on Docker

Containerizing InfluxDB with Docker Compose lets you stand up an isolated, reproducible metrics store in minutes. Let’s walk through the setup.

Step 1: Set Up Directory and Credentials

Prepare a working directory for your configuration files:

mkdir -p ~/influxdb-iot && cd ~/influxdb-iot
touch docker-compose.yml .env

Add initial configuration credentials to .env. Make sure to generate a strong administrative token:

DOCKER_INFLUXDB_INIT_USERNAME=iot_admin
DOCKER_INFLUXDB_INIT_PASSWORD=UseAStrongProdPassphrase2024!
DOCKER_INFLUXDB_INIT_ORG=plant_ops
DOCKER_INFLUXDB_INIT_BUCKET=telemetry_raw
DOCKER_INFLUXDB_INIT_ADMIN_TOKEN=48f7b5a1b3924c738e4a9e403d15ef23a886f4a7c0d5

Define the service within docker-compose.yml:

services:
  influxdb:
    image: influxdb:2.7-alpine
    container_name: influxdb_iot
    restart: unless-stopped
    ports:
      - "8086:8086"
    environment:
      - DOCKER_INFLUXDB_INIT_MODE=setup
      - DOCKER_INFLUXDB_INIT_USERNAME=${DOCKER_INFLUXDB_INIT_USERNAME}
      - DOCKER_INFLUXDB_INIT_PASSWORD=${DOCKER_INFLUXDB_INIT_PASSWORD}
      - DOCKER_INFLUXDB_INIT_ORG=${DOCKER_INFLUXDB_INIT_ORG}
      - DOCKER_INFLUXDB_INIT_BUCKET=${DOCKER_INFLUXDB_INIT_BUCKET}
      - DOCKER_INFLUXDB_INIT_ADMIN_TOKEN=${DOCKER_INFLUXDB_INIT_ADMIN_TOKEN}
      - DOCKER_INFLUXDB_INIT_RETENTION=30d
    volumes:
      - influxdb_data:/var/lib/influxdb2
      - influxdb_config:/etc/influxdb2

volumes:
  influxdb_data:
  influxdb_config:

Step 2: Launch and Verify

Spin up the daemonized container:

docker compose up -d

Stream container logs to ensure the automated bootstrap completed:

docker logs -f influxdb_iot

Look for the log line msg=Listening log_id=... service=tcp-listener transport=http addr=:8086. Your telemetry_raw bucket is now live with an enforced 30-day retention window.

Step 3: Push Telemetry via Line Protocol

Line Protocol sends compact, delimited strings directly over HTTP. You avoid JSON serialization overhead entirely. Send a test reading from sensor unit 12:

curl -i -XPOST "http://localhost:8086/api/v2/write?org=plant_ops&bucket=telemetry_raw&precision=s" \
  --header "Authorization: Token 48f7b5a1b3924c738e4a9e403d15ef23a886f4a7c0d5" \
  --data-raw "machine_metrics,device_id=sensor_12,line=assembly_4 temperature=42.3,vibration=0.018,kw_load=3.75"

Line Protocol structure breaks down into three distinct sections:

  • Measurement (machine_metrics): Acts as the target grouping (analogous to a SQL table).
  • Tag Set (device_id=sensor_12,line=assembly_4): String metadata keys indexed by the TSM engine for microsecond filtering.
  • Field Set (temperature=42.3,vibration=0.018,kw_load=3.75): Unindexed typed payloads containing actual metric values.

Step 4: Query and Downsample with Python

Production monitoring pipelines pull these metrics downstream to evaluate alerts and build dashboards. Install the official InfluxDB client:

pip install influxdb-client

The script below pulls temperature metrics from the past 2 hours and calculates 5-minute rolling averages using Flux:

from influxdb_client import InfluxDBClient

URL = "http://localhost:8086"
TOKEN = "48f7b5a1b3924c738e4a9e403d15ef23a886f4a7c0d5"
ORG = "plant_ops"
BUCKET = "telemetry_raw"

client = InfluxDBClient(url=URL, token=TOKEN, org=ORG)
query_api = client.query_api()

flux_query = f'''
from(bucket: "{BUCKET}")
  |> range(start: -2h)
  |> filter(fn: (r) => r._measurement == "machine_metrics")
  |> filter(fn: (r) => r._field == "temperature")
  |> aggregateWindow(every: 5m, fn: mean, createEmpty: false)
  |> yield(name: "rolling_mean")
'''

tables = query_api.query(org=ORG, query=flux_query)

for table in tables:
    for record in table.records:
        ts = record.get_time().strftime("%Y-%m-%d %H:%M:%S")
        device = record.values.get("device_id")
        val = record.get_value()
        print(f"[{ts}] {device} | Mean Temp: {val:.2f}°C")

client.close()

Operational Results

Isolating high-frequency sensor writes to InfluxDB immediately relieved our primary relational database. Disk I/O dropped from 98% down to 12%. Storage requirements shrank from 18 GB per week to under 5 GB. Most importantly, 30-day aggregation queries that previously timed out now finish consistently in under 65 milliseconds.

Share:
Tags: