The Real-World Problem: Post-Outage Cluster Drift and “ClickOps” Fatigue
Last winter, a tripped circuit breaker killed power to my three refurbished Lenovo ThinkCentre M720q nodes running K3s. When the nodes booted back up, four of my twelve core services stayed down.
Longhorn volumes refused to remount, Traefik ingress routes vanished, and my Pi-hole DNS instance stalled in a CrashLoopBackOff state. I spent four straight hours scouring bash histories across two laptops, trying to reconstruct the ad-hoc kubectl edit tweaks and undocumented Helm flags I had pushed at 1:00 AM two weeks earlier.
Most homelabbers hit this wall. You set up a fresh cluster with clean intentions. Then convenience takes over. You patch an environment variable directly on a live pod. You run an untracked helm install to test a media server. You leave manifests tucked away in random ~/k8s-test folders on a workstation. The moment hardware fails, your cluster’s actual state no longer matches anything on disk. Rebuilding it means starting from scratch.
Root Cause Analysis: Why Imperative Deployments Break Down
Imperative cluster management treats your nodes like delicate pets. Direct changes bypass source control entirely, leaving you with three major architectural liabilities:
- Silent Configuration Drift: A hotfix applied directly to a running deployment does not generate an audit trail. Three weeks later, nobody remembers why that memory limit was raised from 256Mi to 1Gi.
- Workstation Single Points of Failure: If your working YAML files live on your primary desktop’s NVMe drive, your entire cluster topology depends on one workstation surviving.
- Security Risks from Push-Based CI/CD: Running pipelines like GitHub Actions to push manifests inward forces you to open port 6443 on your home router, configure Dynamic DNS, or juggle brittle Cloudflare and WireGuard tunnels.
Over time, your cluster becomes a black box that nobody dares to restart.
Comparing Deployment Models for Bare-Metal K3s
Before settling on a dedicated GitOps controller, I benchmarked the three standard ways home operators deliver manifests:
1. Terminal-Driven Kubectl and Helm
Executing kubectl apply -f ./manifests from your terminal gives you immediate feedback during local testing. Beyond day one, it falls apart. The terminal gives you zero drift detection. If an update script or broken operator mutates your service definitions, your local directory has no way of knowing.
2. External Push Pipelines (GitHub Actions / GitLab CI)
In cloud environments like AWS EKS, managed runners deploy directly into your VPC. At home, that requires opening external ingress to your Kubernetes control plane. Even worse, if the pipeline succeeds but your pod crashes ten minutes later due to an out-of-memory kill, the external CI tool marks the pipeline as green and walks away.
3. In-Cluster GitOps (Argo CD)
Argo CD flips the network model. An agent runs inside your cluster and watches a remote Git repository. Every three minutes—or immediately via webhooks—it compares your committed YAML manifests against the live state inside etcd. When you commit changes, Argo CD pulls them down from the inside out. Your firewall keeps port 6443 shut tight.
Step-by-Step: Setting Up Argo CD on K3s
The core Argo CD stack consumes around 350Mi of RAM across its API server, repo server, and controller. That fits comfortably even on an 8GB memory footprint.
Step 1: Installing Argo CD
Create an isolated namespace and pull the upstream manifests:
# Create a dedicated namespace
kubectl create namespace argocd
# Apply the stable deployment manifests
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
# Wait for the five primary controller pods to hit Running status
kubectl wait --for=condition=ready pod -l app.kubernetes.io/name=argocd-server -n argocd --timeout=120s
Grab the automatically generated admin password:
kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo
Forward port 8080 to access the web dashboard securely on your local machine:
kubectl port-forward svc/argocd-server -n argocd 8080:443
Navigate to https://localhost:8080, accept the self-signed TLS warning, and sign in as admin.
Step 2: Structuring Your Git Repository
Set up a clean repository on GitHub, GitLab, or a local Gitea instance. Grouping applications in dedicated subdirectories keeps dependencies isolated:
homelab-gitops/
├── apps/
│ ├── whoami/
│ │ ├── deployment.yaml
│ │ └── service.yaml
│ └── monitoring/
│ ├── prometheus.yaml
│ └── grafana.yaml
└── bootstrap/
└── whoami-app.yaml
Populate apps/whoami/deployment.yaml with a lightweight web server for testing:
apiVersion: apps/v1
kind: Deployment
metadata:
name: whoami
namespace: default
labels:
app: whoami
spec:
replicas: 2
selector:
matchLabels:
app: whoami
template:
metadata:
labels:
app: whoami
spec:
containers:
- name: whoami
image: traefik/whoami:v1.10.1
ports:
- containerPort: 80
resources:
limits:
memory: 64Mi
cpu: 100m
---
apiVersion: v1
kind: Service
metadata:
name: whoami-service
namespace: default
spec:
type: ClusterIP
ports:
- port: 80
targetPort: 80
selector:
app: whoami
Push this commit to your main branch.
Step 3: Creating the Declarative Application Manifest
Avoid clicking around the UI to build applications. Managing Argo CD itself through YAML keeps your disaster recovery workflow repeatable. Save this manifest as bootstrap/whoami-app.yaml:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: whoami-service
namespace: argocd
finalizers:
- resources-finalizer.argocd.argoproj.io
spec:
project: default
source:
repoURL: 'https://github.com/your-username/homelab-gitops.git'
targetRevision: HEAD
path: apps/whoami
destination:
server: 'https://kubernetes.default.svc'
namespace: default
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
Two directives make this configuration self-sufficient:
prune: trueinstructs Argo CD to delete live cluster resources when you remove their YAML files from Git.selfHeal: trueforces the cluster back into alignment whenever someone edits resources manually withkubectl.
Apply the manifest:
kubectl apply -f bootstrap/whoami-app.yaml
Step 4: Verifying Automated Drift Correction
Within seconds, the Argo CD UI marks whoami-service as Synced and Healthy.
Let’s simulate an accidental change. Scale the deployment directly from your terminal:
kubectl scale deployment whoami --replicas=6
Check the pods immediately with kubectl get pods -l app=whoami. You might catch four extra pods starting for a brief moment. Then Argo CD steps in. It flags the difference between the live count (6) and your Git repository target (2), terminates the extra replicas, and locks the deployment back at 2. Your repository stays in command.
