Posted inHomeLab

HomeLab Monitoring with Prometheus, Grafana, and Node Exporter on Docker: Metrics and Telegram Alerts

HomeLab tutorial - IT technology blog
HomeLab tutorial - IT technology blog

Context & Why

Most home labs start innocently. You spin up Pi-hole for DNS filtering, add Home Assistant, and deploy a Jellyfin or Plex media server. Everything hums along fine until a runaway container dumps 40 GB of logs into /var/lib/docker overnight, filling your boot drive to 100%. Or perhaps an unexpected 4K transcode pins all cores on an Intel N100 mini PC, locking up your hypervisor completely. When that happens, your family usually reports the outage before you even know something broke.

Reactive troubleshooting gets old fast. You need an automated observability pipeline that warns you before services crash.

Lightweight single-binary tools like Netdata work well for quick checks, but Prometheus, Node Exporter, and Grafana remain the gold standard. Prometheus uses a pull model to scrape metrics over HTTP at set intervals (typically every 15 seconds) and stores them in an efficient time-series database (TSDB) averaging just 1.5 to 2 bytes per sample.

Node Exporter runs as an agent on the host OS, exposing raw hardware statistics: CPU wait times, memory pressure, NVMe disk I/O, and thermal sensors. Grafana connects to Prometheus to render those metrics into clear, customizable dashboards.

Still, pretty graphs only solve half the problem. Nobody stares at a dashboard 24/7. Pairing Prometheus with Alertmanager and a custom Telegram bot routes critical notifications—like a disk hitting 85% capacity or a node dropping offline—straight to your phone for free.

Installation

Installing monitoring agents directly on the host OS turns into a maintenance headache during distro upgrades. Running the stack through Docker Compose keeps configurations isolated, reproducible, and easy to back up.

Directory Structure Setup

Create a dedicated directory on your host to store configurations and persistent volume mounts:

mkdir -p ~/homelab-monitoring/{prometheus,alertmanager,grafana/provisioning}
cd ~/homelab-monitoring

Docker Compose Configuration

Save the following as docker-compose.yml. Pay close attention to Node Exporter. It uses network_mode: host, reads host system paths directly, and strips container-level virtualization so you collect true bare-metal figures:

version: '3.8'

networks:
  monitoring:
    driver: bridge

services:
  node-exporter:
    image: prom/node-exporter:v1.8.2
    container_name: node-exporter
    restart: unless-stopped
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/rootfs:ro
    command:
      - '--path.procfs=/host/proc'
      - '--path.rootfs=/rootfs'
      - '--path.sysfs=/host/sys'
      - '--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)'
    network_mode: host

  prometheus:
    image: prom/prometheus:v2.54.1
    container_name: prometheus
    restart: unless-stopped
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - ./prometheus/alert_rules.yml:/etc/prometheus/alert_rules.yml:ro
      - prometheus_data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--web.console.libraries=/usr/share/prometheus/console_libraries'
      - '--web.console.templates=/usr/share/prometheus/consoles'
      - '--storage.tsdb.retention.time=30d'
    networks:
      - monitoring
    ports:
      - '9090:9090'

  alertmanager:
    image: prom/alertmanager:v0.27.0
    container_name: alertmanager
    restart: unless-stopped
    volumes:
      - ./alertmanager/alertmanager.yml:/etc/alertmanager/alertmanager.yml:ro
    command:
      - '--config.file=/etc/alertmanager/alertmanager.yml'
    networks:
      - monitoring
    ports:
      - '9093:9093'

  grafana:
    image: grafana/grafana:11.2.0
    container_name: grafana
    restart: unless-stopped
    environment:
      - GF_SECURITY_ADMIN_USER=admin
      - GF_SECURITY_ADMIN_PASSWORD=ChangeMeDevOps123!
      - GF_USERS_ALLOW_SIGN_UP=false
    volumes:
      - grafana_data:/var/lib/grafana
    networks:
      - monitoring
    ports:
      - '3000:3000'

volumes:
  prometheus_data:
  grafana_data:

Configuration

Now wire the components together: scrape targets, firing thresholds, and alert delivery.

1. Prometheus Scrape Configuration

Create prometheus/prometheus.yml. Because Node Exporter uses the host network on port 9100, reach it via your host LAN IP (for example, 192.168.1.50:9100) or the Docker gateway bridge IP (typically 172.17.0.1:9100):

global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files:
  - 'alert_rules.yml'

alerting:
  alertmanagers:
    - static_configs:
        - targets:
            - 'alertmanager:9093'

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'homelab-node'
    static_configs:
      - targets: ['172.17.0.1:9100']

2. Define Alert Rules

Create prometheus/alert_rules.yml. A common trap with Node Exporter is the filesystem mountpoint label. Passing --path.rootfs=/rootfs strips the prefix inside the collector, meaning your host root filesystem appears as mountpoint="/", not /rootfs.

Here are tested rules for host drops, runaway CPU spikes, low RAM, and disk limits:

groups:
  - name: HostAlerts
    rules:
      - alert: HostDown
        expr: up{job="homelab-node"} == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Host is unreachable"
          description: "Node Exporter target {{ $labels.instance }} stopped responding for over 1 minute."

      - alert: HighCpuLoad
        expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 85
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High CPU usage"
          description: "CPU usage on {{ $labels.instance }} has exceeded 85% for 5 minutes."

      - alert: DiskRunningLow
        expr: (node_filesystem_avail_bytes{mountpoint="/",fstype!~"tmpfs|overlay"} * 100) / node_filesystem_size_bytes{mountpoint="/",fstype!~"tmpfs|overlay"} < 15
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Low disk space on root volume"
          description: "Available disk space on {{ $labels.instance }} root partition is below 15%."

      - alert: HighMemoryUsage
        expr: (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100 > 90
        for: 3m
        labels:
          severity: warning
        annotations:
          summary: "Memory near capacity"
          description: "Host memory usage has remained above 90% for 3 minutes."

3. Telegram Bot & Alertmanager Setup

Generating credentials takes less than two minutes:

  1. Open Telegram, search for @BotFather, send /newbot, and save the HTTP API token provided.
  2. Send a direct message to your new bot, or add it to a dedicated alerting channel.
  3. Retrieve your chat ID by opening https://api.telegram.org/bot<YOUR_BOT_TOKEN>/getUpdates in your browser and grabbing the numeric value under message.chat.id.

Add these credentials to alertmanager/alertmanager.yml:

global:
  resolve_timeout: 5m

route:
  group_by: ['alertname', 'instance']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 3h
  receiver: 'telegram-alerts'

receivers:
  - name: 'telegram-alerts'
    telegram_configs:
      - bot_token: '123456789:ABCdefGhIJKlmNoPQRsTUVwxyZ'
        chat_id: -1001234567890
        parse_mode: 'HTML'
        send_resolved: true
        message: >-
          {{ if eq .Status "firing" }}🔥 <b>FIRING ALERT</b>{{ else }}✅ <b>RESOLVED</b>{{ end }}
          
          <b>Alert:</b> {{ .CommonAnnotations.summary }}
          <b>Severity:</b> {{ .CommonLabels.severity }}
          <b>Details:</b> {{ .CommonAnnotations.description }}

Verification & Monitoring

Bring up the stack:

docker compose up -d

Confirm every container is running cleanly:

docker compose ps

Validating Metric Ingestion

Open http://<your-server-ip>:9090 and head to Status > Targets. Both the prometheus and homelab-node endpoints should display a green UP badge. Next, open the Alerts tab to verify your four host alerts appear in a clean inactive state.

Importing the Node Exporter Full Dashboard

  1. Navigate to http://<your-server-ip>:3000 and log in with user admin and the password defined in your compose file.
  2. Go to Connections > Data Sources, select Add data source, choose Prometheus, and set the URL to http://prometheus:9090. Click Save & Test.
  3. Go to Dashboards > New > Import. Enter dashboard ID 1860 (the community-maintained “Node Exporter Full” dashboard), select your Prometheus source, and click Import.

Grafana will instantly show CPU core load, per-disk IOPS, network interface throughput, and memory buffer usage.

Testing the Telegram Alerting Pipeline

Never assume an alert system works until you see it trigger under load. You can test your notification chain safely using a lightweight stress container without affecting persistent services:

# Load 4 CPU workers for 6 minutes to trigger the HighCpuLoad alert
docker run --rm -it progrium/stress --cpu 4 --timeout 360s

Watch http://<your-server-ip>:9093. The HighCpuLoad alert will shift from Pending to Firing after five minutes, and your bot will push the message to Telegram. Once the stress container exits, Alertmanager automatically dispatches a green RESOLVED notification, confirming everything has returned to baseline.

Share:
Tags: