The 2 AM Incident That Changed How I Handle Database Failures
It started with a traffic spike. Our PostgreSQL database slowed under load, queries started timing out, and our application — configured with aggressive retry logic — kept hammering the database with thousands of reconnect attempts per second. Instead of recovering, the database buckled completely. Forty minutes of downtime, a very unhappy team, and a postmortem that all pointed to the same root cause: no circuit breaker.
If you’ve been paged at 2 AM for something like this, you know exactly how it feels. This guide walks you through building a proper database circuit breaker in Python so you don’t have to learn the hard way like I did.
What’s Actually Happening When Your Database Fails
Here’s the failure cascade that catches junior developers off guard:
- Database gets slow due to high load or a long-running query
- Application requests start timing out
- Retry logic kicks in — each failed request retries 3 times
- Now you have 3× the database load from retries alone
- Database gets slower, more timeouts, more retries
- Total collapse
This is called a retry storm (sometimes “thundering herd”). Your retry logic, meant to improve reliability, becomes the thing that kills the database. The root cause isn’t the retries themselves — it’s that retries have no awareness of the overall system health. They keep firing even when it’s obvious the database needs a break.
Three Approaches — And Why Most Teams Pick the Wrong One
Option 1: Simple Retry
import time
import psycopg2
def query_with_retry(sql, max_retries=3):
for attempt in range(max_retries):
try:
conn = psycopg2.connect("postgresql://localhost/mydb")
cursor = conn.cursor()
cursor.execute(sql)
return cursor.fetchall()
except Exception:
if attempt == max_retries - 1:
raise
time.sleep(1)
This is what most people start with. The problem: during database overload, this amplifies the stress. 100 concurrent users × 3 retries each = 300 connection attempts hitting a struggling database. It actively makes things worse.
Option 2: Exponential Backoff with Tenacity
Tenacity is a Python retry library that handles backoff cleanly. This is a genuine improvement:
pip install tenacity psycopg2-binary
from tenacity import retry, stop_after_attempt, wait_exponential
import psycopg2
@retry(
stop=stop_after_attempt(5),
wait=wait_exponential(multiplier=1, min=1, max=30)
)
def query_db(sql):
conn = psycopg2.connect("postgresql://localhost/mydb")
cursor = conn.cursor()
cursor.execute(sql)
return cursor.fetchall()
Exponential backoff means retry delays grow: 1s, 2s, 4s, 8s… This gives the database time to breathe between attempts. Much better than Option 1. But there’s still no mechanism to stop retrying entirely when the database is clearly down for an extended period. You’re still throwing requests at a broken system, just slower.
Option 3: Circuit Breaker Pattern (The Right Tool)
The circuit breaker concept comes from electrical engineering. When a circuit gets overloaded, a breaker trips to protect the system. Once things cool down, you reset it. In software, a circuit breaker has three states:
- Closed — normal operation, requests pass through
- Open — circuit is tripped, requests fail immediately without touching the database
- Half-Open — after a cooldown period, allows one test request to check if the database has recovered
The key insight: when the circuit is Open, you stop hammering the database entirely. This gives it the space to recover on its own.
Building It: Circuit Breaker + Tenacity Together
Here’s the implementation I use in production. Combining Tenacity’s backoff with a custom circuit breaker gives you the best of both worlds — smart retries on transient errors, and a hard stop when the database is genuinely down.
Step 1: The Circuit Breaker Class
import time
import threading
from enum import Enum
class CircuitState(Enum):
CLOSED = "closed"
OPEN = "open"
HALF_OPEN = "half_open"
class DatabaseCircuitBreaker:
def __init__(self, failure_threshold=5, recovery_timeout=30):
self.failure_threshold = failure_threshold
self.recovery_timeout = recovery_timeout
self.failure_count = 0
self.last_failure_time = None
self.state = CircuitState.CLOSED
self._lock = threading.Lock()
def record_failure(self):
with self._lock:
self.failure_count += 1
self.last_failure_time = time.time()
if self.failure_count >= self.failure_threshold:
self.state = CircuitState.OPEN
print(f"[Circuit Breaker] OPEN — {self.failure_count} failures detected")
def record_success(self):
with self._lock:
self.failure_count = 0
self.state = CircuitState.CLOSED
def can_attempt(self):
with self._lock:
if self.state == CircuitState.CLOSED:
return True
if self.state == CircuitState.OPEN:
elapsed = time.time() - self.last_failure_time
if elapsed >= self.recovery_timeout:
self.state = CircuitState.HALF_OPEN
print("[Circuit Breaker] HALF-OPEN — testing recovery")
return True
return False
return True # HALF_OPEN: allow one probe
Step 2: Wrapping Database Calls
from tenacity import retry, stop_after_attempt, wait_exponential, RetryError
import psycopg2
db_circuit = DatabaseCircuitBreaker(failure_threshold=5, recovery_timeout=30)
@retry(
stop=stop_after_attempt(3),
wait=wait_exponential(multiplier=1, min=1, max=10)
)
def _execute_query(sql, params=None):
conn = psycopg2.connect("postgresql://localhost/mydb")
cursor = conn.cursor()
cursor.execute(sql, params or ())
result = cursor.fetchall()
conn.close()
return result
def safe_query(sql, params=None):
if not db_circuit.can_attempt():
raise Exception("Circuit OPEN — database unavailable, skipping request")
try:
result = _execute_query(sql, params)
db_circuit.record_success()
return result
except RetryError as e:
db_circuit.record_failure()
raise Exception(f"Query failed after retries: {e}")
except Exception as e:
db_circuit.record_failure()
raise
Step 3: Background Health Check for Automatic Recovery
The half-open state handles test requests from incoming traffic automatically. For background services with low traffic, add a periodic health probe so the circuit doesn’t stay open forever waiting for a real request:
def db_health_check(circuit_breaker, interval=15):
"""Background thread: probe DB when circuit is open."""
while True:
time.sleep(interval)
if circuit_breaker.state == CircuitState.OPEN:
try:
conn = psycopg2.connect(
"postgresql://localhost/mydb",
connect_timeout=3
)
conn.close()
circuit_breaker.record_success()
print("[Health Check] Database recovered — circuit CLOSED")
except Exception:
print("[Health Check] Database still unavailable")
health_thread = threading.Thread(
target=db_health_check,
args=(db_circuit,),
daemon=True
)
health_thread.start()
Step 4: Smoke Test
if __name__ == "__main__":
for i in range(20):
try:
result = safe_query("SELECT NOW()")
print(f"Query {i+1}: OK — {result}")
except Exception as e:
print(f"Query {i+1}: FAILED — {e}")
time.sleep(0.5)
Tuning the Parameters for Your Setup
Two numbers matter most:
- failure_threshold: Failures before the circuit opens. Set this too low (1-2) and you’ll trip on transient network hiccups. Too high and you’re still flooding a struggling database. Starting at 5 works for most setups.
- recovery_timeout: Seconds to wait before allowing a probe request. 30 seconds is a safe default for Postgres. Under heavy load where the database needs longer to drain its connection queue, I’ve used 60-120 seconds.
One practical note from my own workflow: when I’m setting up test scenarios that need sample data, I often need to convert CSV fixtures to JSON for import scripts. I use toolcraft.app/en/tools/data/csv-to-json — it runs entirely in the browser, so no data leaves your machine. Useful when you’re working with anything close to real production data.
State Transitions at a Glance
CLOSED → (failure_threshold reached) → OPEN
OPEN → (recovery_timeout elapsed) → HALF_OPEN
HALF_OPEN → (test request succeeds) → CLOSED
HALF_OPEN → (test request fails) → OPEN
What This Pattern Does Not Cover
A circuit breaker protects your app from overloading the database, but it sits alongside — not instead of — other reliability layers:
- Connection pooling: Use PgBouncer or SQLAlchemy’s built-in pool to cap total concurrent connections at the OS level
- Read replicas: Circuit-break against primary and replicas independently — they have different failure modes
- Graceful degradation: When the circuit is open, decide what your API returns — cached data, an explicit 503, or a safe default value. Failing silently is worse than failing loudly
Once this is in place, database incidents that used to cascade into full application outages tend to self-heal in under a minute. The database gets breathing room, and your app reconnects automatically once it’s healthy. That 40-minute outage I mentioned at the start? After adding circuit breakers across our database layer, the same failure mode became a 45-second blip that resolved itself before anyone woke up.
