Posted inHomeLab

Building HA Storage for K3s with Longhorn on Proxmox VE

HomeLab tutorial - IT technology blog
HomeLab tutorial - IT technology blog

Problem Statement: The Storage Pitfall in Virtualized Kubernetes

Spinning up a multi-node K3s cluster on Proxmox VE is a go-to architecture for home labs, branch offices, and edge nodes. Yet stateful workloads—PostgreSQL, Nextcloud, or Prometheus—frequently hit a wall during maintenance. K3s defaults to Rancher Local Path Provisioner. It is fast and uses minimal resources, but it binds persistent data directly to the host filesystem where the pod first starts via hostPath.

Reboot a Proxmox virtual machine for a kernel update, and the problem surfaces immediately. The hypervisor node reboots, and Kubernetes reschedules your pod to a surviving worker node. Because the raw data sits locked on the powered-down VM virtual disk, the new pod stalls in CrashLoopBackOff or Pending. True high availability requires distributed block storage that mirrors disk writes across distinct nodes over your network.

Core Concepts: How Longhorn Powers Distributed Block Storage

Longhorn is a lightweight distributed block storage platform built specifically for Kubernetes. It runs directly inside the cluster as native microservices. Unlike Ceph, which routinely demands 2 to 4 GB of RAM per OSD daemon plus dedicated storage nodes, Longhorn operates comfortably with roughly 1 to 1.5 GB of RAM per node across its control plane and data engines.

Longhorn divides storage management into three core layers:

  • Longhorn CSI Plugin: Integrates with the Kubernetes Container Storage Interface to provision PersistentVolumes (PVs), attach devices, and coordinate volume snapshots.
  • Longhorn Engine: A lightweight controller pod created dynamically for each attached volume. It coordinates data paths and serves an iSCSI endpoint directly to the local worker node.
  • Longhorn Replicas: Thin-provisioned block layers assigned across separate cluster nodes. By default, Longhorn provisions 3 replicas per volume. When an application commits a write, the engine broadcasts the data synchronously across all three replicas before returning an I/O completion signal.

Under Proxmox VE, each VM passes an attached virtual disk into the storage pool. For optimal throughput and garbage collection, configure these disks on a VirtIO SCSI Single controller with discard=on enabled. When a hypervisor host fails, the engine continues serving reads and writes through the remaining live replicas.

Hands-on Practice: Deploying and Testing HA Storage

This deployment assumes three Proxmox VMs running Ubuntu 22.04 or Debian 12 with K3s installed (k3s-node-01, k3s-node-02, k3s-node-03). Each VM has 4 vCPUs, 8 GB of RAM, and a secondary 50 GB virtual disk at /dev/sdb formatted with ext4 and mounted to /var/lib/longhorn. For cluster replication traffic, run at least a 1 GbE dedicated network; a 2.5 GbE or 10 GbE interface keeps write latencies under 5 ms.

Step 1: Install Host Dependencies on All Nodes

Longhorn manages raw block devices via iSCSI and relies on NFS tools for handling backup targets. Run the following installation on each K3s node:

sudo apt-get update && sudo apt-get install -y open-iscsi nfs-common util-linux bash curl
sudo systemctl enable iscsid && sudo systemctl start iscsid

Check that all kernel modules—including iscsi_tcp and dm_crypt—are operational by running the official Longhorn validation script:

curl -sSfL https://raw.githubusercontent.com/longhorn/longhorn/v1.6.1/scripts/environment_check.sh | bash

Step 2: Deploy Longhorn via Helm

Add the official Longhorn repository using Helm on your control-plane node:

helm repo add longhorn https://charts.longhorn.io
helm repo update

Create a tuning configuration named longhorn-values.yaml. Setting replicaSoftAntiAffinity: false guarantees that replicas never double up on the same VM, enforcing strict node separation:

persistence:
  defaultClass: true
  defaultClassReplicaCount: 3
defaultSettings:
  defaultDataPath: /var/lib/longhorn
  storageOverProvisioningPercentage: 150
  storageMinimalAvailablePercentage: 15
  replicaSoftAntiAffinity: false

Execute the installation into the dedicated longhorn-system namespace:

helm install longhorn longhorn/longhorn \
  --namespace longhorn-system \
  --create-namespace \
  -f longhorn-values.yaml

Track pod startup until the managers and CSI drivers reach ready states:

kubectl -n longhorn-system get pods -w

Step 3: Deploy a Stateful Workload with Data Replication

To verify synchronous replication, deploy a testing pod configured with custom eviction tolerations. Adding tolerations for node.kubernetes.io/unreachable shortens failover delays from Kubernetes’ standard 300-second eviction timeout down to 10 seconds.

Save this configuration as ha-storage-test.yaml:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: longhorn-test-pvc
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: longhorn
  resources:
    requests:
      storage: 5Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: storage-writer
spec:
  replicas: 1
  selector:
    matchLabels:
      app: storage-writer
  template:
    metadata:
      labels:
        app: storage-writer
    spec:
      tolerations:
      - key: "node.kubernetes.io/unreachable"
        operator: "Exists"
        effect: "NoExecute"
        tolerationSeconds: 10
      - key: "node.kubernetes.io/not-ready"
        operator: "Exists"
        effect: "NoExecute"
        tolerationSeconds: 10
      containers:
      - name: writer
        image: busybox
        command: ["/bin/sh", "-c", "while true; do date >> /data/heartbeat.log; sleep 2; done"]
        volumeMounts:
        - name: data-volume
          mountPath: /data
      volumes:
      - name: data-volume
        persistentVolumeClaim:
          claimName: longhorn-test-pvc

Apply the manifest:

kubectl apply -f ha-storage-test.yaml

Inspect the generated PersistentVolumeClaim and verify replica distribution:

kubectl get pvc longhorn-test-pvc
VOL_NAME=$(kubectl get pvc longhorn-test-pvc -o jsonpath='{.spec.volumeName}')
kubectl -n longhorn-system get replicas -l longhornvolume=$VOL_NAME

Three distinct replica entries appear in the output, with each instance mapped to a different K3s node.

Step 4: Testing Failover and Node Survival

Identify which worker VM currently executes the writer pod:

kubectl get pod -l app=storage-writer -o wide

Suppose the pod runs on k3s-node-02 (VM ID 102). Simulate an unexpected hardware crash by pulling the plug from the Proxmox VE shell:

# Execute directly on your Proxmox host shell
qm stop 102

Kubernetes transitions k3s-node-02 to NotReady within 40 seconds. Thanks to our configured toleration, the cluster terminates the orphaned pod 10 seconds later and schedules a replacement on k3s-node-01 or k3s-node-03. The Longhorn Engine detaches the old volume instance, attaches it to the replacement node, and remounts the filesystem.

Confirm that file writes continued without corruption:

NEW_POD=$(kubectl get pod -l app=storage-writer -o jsonpath='{.items[0].metadata.name}')
kubectl exec -it $NEW_POD -- tail -n 10 /data/heartbeat.log

The output shows uninterrupted two-second timestamps spanning the reboot window, confirming zero lost records.

Conclusion

Combining K3s with Longhorn on Proxmox VE transforms fragile, single-node persistent storage into an elastic, fault-tolerant platform. Host reboots, VM migrations, and hardware hiccups no longer hold application state hostage. To protect against hypervisor-wide data loss, open the Longhorn UI and establish an automated backup schedule pointing to an off-cluster NFS export or S3 bucket.

Share:
Tags: