Context & Why: When Your Database Drops at 2 AM
It was 2:15 AM on a Sunday when PagerDuty pulled me out of bed: 450 write queries failing every second across three critical payment microservices. The connection pool hit its 1,000-connection ceiling, and both read replicas fell hopelessly out of sync. If you manage stateful workloads on Kubernetes, you already know the dread that follows a flaky primary database node vanishing from the network.
Traditional PostgreSQL setups lean on external tooling like Patroni, keepalived, or custom Pacemaker scripts. Jamming those legacy patterns into vanilla Kubernetes StatefulSets usually invites disaster. You hit split-brain hazards, wait through 60-second election timeouts, and wrestle with stuck AWS EBS volume attachments. What should be a quick pod restart turns into a 20-minute production outage.
CloudNativePG (CNPG) takes a different path. Instead of forcing Kubernetes to act like a legacy hypervisor, CNPG treats the database as a native Kubernetes citizen. It manages streaming replication using PostgreSQL’s internal engine consensus. No external etcd clusters. No fragile bash probes.
Because the operator hooks straight into the Kubernetes API, it rolls updates smoothly and triggers failovers in under 10 seconds. On top of that, it backs up WAL archives directly to S3-compatible object storage for Continuous Archiving and Point-In-Time Recovery (PITR).
After migrating three production clusters handling roughly 12,000 queries per second to CNPG, our unplanned database downtime dropped to zero. Let’s walk through how to deploy, configure, and maintain PostgreSQL with CNPG using declarative GitOps so you never have to scramble during a midnight failover again.
Installation: Deploying the CloudNativePG Operator
Forget running imperative helm install commands from your laptop. Store your operator definition in Git alongside the rest of your cluster infrastructure. The cleanest way to run CNPG in production is by tracking its official controller release manifests.
First, create a dedicated namespace and apply the controller bundle:
# Create dedicated namespace for the operator
kubectl create namespace cnpg-system
# Deploy CloudNativePG operator release manifest
kubectl apply --server-side -f https://raw.githubusercontent.com/cloudnative-pg/cloudnative-pg/release-1.22/releases/cnpg-1.22.0.yamlConfirm the operator pod is healthy and the Custom Resource Definitions (CRDs) are registered:
kubectl get deployment -n cnpg-system cnpg-controller-manager
kubectl get crds | grep postgresql.cnpg.ioYou will see core definitions such as clusters.postgresql.cnpg.io, backups.postgresql.cnpg.io, and poolers.postgresql.cnpg.io. The controller manager watches these resources around the clock. Whenever state drifts, it immediately reconciles your live database topology against your declared spec.
Configuration: Production-Grade High Availability and Backup Manifests
A battle-tested database topology demands three core components: an odd-numbered replica count distributed across distinct availability zones, continuous Write-Ahead Log (WAL) archiving to S3, and strict failover timings.
1. Store Cloud Storage Credentials
CloudNativePG embeds Barman under the hood to stream 16MB WAL segments and push base physical snapshots straight to object storage (such as AWS S3, Cloudflare R2, or MinIO). Provision a Secret holding your credentials inside the database namespace:
kubectl create namespace database
kubectl create secret generic s3-backup-creds \
--namespace database \
--from-literal=ACCESS_KEY_ID=minioadmin \
--from-literal=SECRET_ACCESS_KEY=miniopassword2. The High Availability Cluster Manifest
Save this manifest as postgres-ha-cluster.yaml in your Git repository. It provisions a 3-instance PostgreSQL 16 cluster with zone-level anti-affinity, automatic WAL compression, and automated backups:
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
name: pg-prod-cluster
namespace: database
spec:
instances: 3
imageName: ghcr.io/cloudnative-pg/postgresql:16.2
# Enforce instances spread across separate nodes
affinity:
enablePodAntiAffinity: true
topologyKey: topology.kubernetes.io/zone
storage:
size: 50Gi
storageClass: gp3-ebs
postgresql:
parameters:
shared_buffers: "1GB"
work_mem: "16MB"
maintenance_work_mem: "128MB"
max_connections: "200"
wal_keep_size: "512MB"
# WAL archiving and continuous physical backups
backup:
barmanObjectStore:
destinationPath: s3://postgres-backups-bucket/pg-prod/
endpointURL: https://s3.us-east-1.amazonaws.com
s3Credentials:
accessKeyId:
name: s3-backup-creds
key: ACCESS_KEY_ID
secretAccessKey:
name: s3-backup-creds
key: SECRET_ACCESS_KEY
wal:
compression: gzip
maxParallel: 2
retentionPolicy: "30d"
monitoring:
enablePodMonitor: truePush this file to your repository. Your GitOps controller—whether ArgoCD or Flux—will pick it up and synchronize the cluster automatically.
3. Automated Point-In-Time Recovery (PITR)
Failover cannot save you when a buggy migration script drops an entire table at midnight. For that, you need point-in-time recovery. Restoring a cluster with CNPG requires zero manual shell access. Define a fresh cluster resource, point it at your bucket, and set your target timestamp right before the accident occurred:
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
name: pg-prod-restored
namespace: database
spec:
instances: 3
storage:
size: 50Gi
storageClass: gp3-ebs
bootstrap:
recovery:
source: pg-prod-cluster
recoveryTarget:
targetTime: "2026-11-07 02:10:00.000000+00"
externalClusters:
- name: pg-prod-cluster
barmanObjectStore:
destinationPath: s3://postgres-backups-bucket/pg-prod/
endpointURL: https://s3.us-east-1.amazonaws.com
s3Credentials:
accessKeyId:
name: s3-backup-creds
key: ACCESS_KEY_ID
secretAccessKey:
name: s3-backup-creds
key: SECRET_ACCESS_KEYVerification & Monitoring: Proving Failover Under Pressure
High availability is an illusion until you deliberately break your setup in a live test environment.
1. Check Cluster Health with the CNPG Plugin
The cnpg CLI plugin gives you an instant view of replication lag, role allocation, and write-ahead log status:
# Install CNPG kubectl plugin
curl -sSfL https://github.com/cloudnative-pg/cloudnative-pg/releases/download/v1.22.0/kubectl-cnpg_1.22.0_linux_x86_64.tar.gz | tar -xz
sudo mv kubectl-cnpg /usr/local/bin/kubectl-cnpg
# Check the cluster status
kubectl cnpg status pg-prod-cluster -n databaseThe summary highlights the active leader, replication modes, and sync status for each replica:
Cluster Summary
Name: pg-prod-cluster
Namespace: database
Status: Cluster in healthy state
Instances: 3
Ready instances: 3
Current Primary: pg-prod-cluster-1
Instances status
Name Database Status Current LSN Replication role Status
---- --------------- ----------- ---------------- ------
pg-prod-cluster-1 OK 0/A000040 Primary OK
pg-prod-cluster-2 OK 0/A000040 Standby (sync) OK
pg-prod-cluster-3 OK 0/A000040 Standby (sync) OK2. Simulate a Production Outage (Chaos Testing)
Simulate an abrupt host crash by forcefully killing the primary pod without an OS shutdown notice:
kubectl delete pod pg-prod-cluster-1 -n database --grace-period=0 --forceFollow the cluster events as the operator reacts:
kubectl get events -n database --field-selector reason=Switchover -wIn roughly 4 to 8 seconds, CNPG detects the missing heartbeat. It promotes pg-prod-cluster-2 to primary, instructs pg-prod-cluster-3 to follow the new leader, and rewires the native Kubernetes Services:
pg-prod-cluster-rw: Reassigns endpoint IPs so write operations immediately reach the newly elected primary.pg-prod-cluster-ro: Keeps routing read-only queries across healthy standbys without dropping connection pools.
When the dead pod restarts, CNPG handles reconciliation automatically. It streams missing WAL segments from the new primary, brings the database back up to date, and hooks it in as a standby. Nobody touches a terminal. By committing database topology to Git and validating failovers upfront, late-night fire drills become ordinary, quiet background events.
