Quick Start (5 min): Spinning Up a 3-Node Cluster
Six months ago, our payment gateway kept dropping transactions during primary database failovers. Our single-instance PostgreSQL setup took nearly 45 seconds to promote a standby node. That downtime cost us orders, so we migrated to CockroachDB. The objective was simple: survive hardware crashes without manual firefighting. Here is how to run a working three-node cluster on Docker Compose in under five minutes.
Start by setting up an isolated bridge network so your nodes can discover each other by container name:
docker network create -d bridge roach-netSave the following configuration as docker-compose.yml. It defines three database nodes and a one-shot container that initializes the Raft cluster:
version: '3.8'
networks:
roach-net:
external: true
services:
roach1:
image: cockroachdb/cockroach:v23.2.0
container_name: roach1
hostname: roach1
networks:
- roach-net
command: start --insecure --store=type=mem,size=1Gi --listen-addr=0.0.0.0:26257 --advertise-addr=roach1:26257 --http-addr=0.0.0.0:8080 --join=roach1,roach2,roach3
ports:
- "26257:26257"
- "8080:8080"
roach2:
image: cockroachdb/cockroach:v23.2.0
container_name: roach2
hostname: roach2
networks:
- roach-net
command: start --insecure --store=type=mem,size=1Gi --listen-addr=0.0.0.0:26257 --advertise-addr=roach2:26257 --http-addr=0.0.0.0:8081 --join=roach1,roach2,roach3
ports:
- "8081:8081"
roach3:
image: cockroachdb/cockroach:v23.2.0
container_name: roach3
hostname: roach3
networks:
- roach-net
command: start --insecure --store=type=mem,size=1Gi --listen-addr=0.0.0.0:26257 --advertise-addr=roach3:26257 --http-addr=0.0.0.0:8082 --join=roach1,roach2,roach3
ports:
- "8082:8082"
init-cluster:
image: cockroachdb/cockroach:v23.2.0
container_name: init-cluster
networks:
- roach-net
command: init --insecure --host=roach1:26257
depends_on:
- roach1
- roach2
- roach3Boot the cluster in the background:
docker compose up -dOnce all containers report healthy, open an interactive SQL session on roach1:
docker exec -it roach1 ./cockroach sql --insecureOpen http://localhost:8080 in your browser. The built-in CockroachDB console shows live metrics, including 3 active nodes, memory allocations, and query throughput in real time.
Architecture, Consensus, and PostgreSQL Compatibility
Do not treat CockroachDB like a conventional PostgreSQL primary with read replicas. It does not replicate entire databases sequentially. Under the hood, it operates as a sorted, transactional key-value store split into chunks and managed across nodes.
The Raft Consensus Engine
CockroachDB breaks every table into 64 MB chunks called Ranges. In a three-node setup, each range replicates across all three nodes using the Raft algorithm. One replica acts as the leaseholder and handles read and write coordination:
- Writes require a quorum of
(N/2) + 1nodes. In a 3-node cluster, 2 nodes must acknowledge each write. - If one node dies, transactions continue smoothly. The remaining two nodes still satisfy quorum.
- If a leaseholder drops, surviving replicas detect the missing heartbeat within 9 seconds and elect a successor in roughly 300 milliseconds.
PostgreSQL Compatibility Realities
CockroachDB speaks the standard PostgreSQL wire protocol on port 26257. Your existing drivers—such as Go’s pgx, Python’s psycopg2, or Node’s pg—connect without custom patches. But behavior differences will catch you off guard if you aren’t prepared.
The biggest shift is isolation level. CockroachDB enforces strict SERIALIZABLE isolation by default, whereas vanilla PostgreSQL defaults to READ COMMITTED. If two transactions touch overlapping rows concurrently, one transaction will abort with SQLSTATE 40001 (serialization_failure). Your application code must catch this code and rerun the transaction using exponential backoff.
Disaster Recovery and Data Ingestion
Resilience sounds great in documentation, but it only matters when nodes vanish under load. Here is how failover behaves when tested, along with efficient data loading workflows.
Simulating an Unplanned Node Outage
Let’s write test data directly to roach2:
docker exec -it roach2 ./cockroach sql --insecure -e "
CREATE DATABASE IF NOT EXISTS shop;
CREATE TABLE shop.orders (id UUID DEFAULT gen_random_uuid() PRIMARY KEY, total DECIMAL);
INSERT INTO shop.orders (total) VALUES (49.99), (120.50);
"Now simulate a sudden machine crash by terminating roach1:
docker stop roach1Immediately query the dataset from roach3:
docker exec -it roach3 ./cockroach sql --insecure -e "SELECT * FROM shop.orders;"The query returns in 2 milliseconds. No dropped connections, no failover scripts, and no human intervention. If roach1 stays down longer than 5 minutes (the default server.time_until_store_dead threshold), the cluster automatically redistributes under-replicated ranges across the remaining nodes.
Ingesting Large Datasets
Do not pipe large CSV files through repeated INSERT statements. Standard inserts route through Raft consensus row-by-row, choking coordination threads. Instead, use the native IMPORT INTO command. It writes directly into storage engine files, easily processing over 50,000 rows per second.
docker exec -i roach2 ./cockroach sql --insecure -e "
IMPORT INTO shop.orders (id, total)
CSV DATA ('nodelocal://1/initial_orders.csv');"If you have dirty CSV files that need quick schema mapping or JSON conversion before ingestion, utility tools like toolcraft.app/en/tools/data/csv-to-json parse files directly inside your browser so confidential records never leave your workstation.
Hard-Earned Production Lessons
Operating our cluster over the last six months revealed several traps that local Docker setups conceal. Here are three critical rules we learned the hard way:
1. Define Fault Domains with Localities
Running three containers on one EC2 instance or bare-metal host gives you zero redundancy. A single kernel panic takes down the entire quorum. When moving beyond local testing, place containers across distinct cloud availability zones and tag each node explicitly:
--locality=region=us-east-1,zone=us-east-1aCockroachDB uses these locality flags to guarantee that copies of the same 64 MB range never sit inside the same availability zone.
2. Ban Sequential Auto-Increment Keys
Traditional SERIAL or monotonic integer primary keys are dangerous in distributed systems. Sequential IDs route every subsequent write to the exact same range leaseholder. This creates a severe disk and CPU hotspot on a single node while the rest of your cluster sits idle. Use UUID via gen_random_uuid() or CockroachDB’s unique_rowid() instead. Both distribute writes uniformly across every disk in the cluster.
3. Enforce Hard Memory Limits
By default, CockroachDB sizes its caches based on total system RAM. Inside Docker, this can trigger Linux OOM kills if container limits disagree with host detection. Set explicit memory boundaries on startup:
--max-sql-memory=25%: Caps memory for in-flight aggregations, hash joins, and sorting buffers.--cache=25%: Allocates a dedicated Pebble cache for hot read pages.
Once you handle retry loops in code and spread primary keys randomly across ranges, CockroachDB delivers genuine zero-downtime persistence on Docker.
