Quick start (5 min)
Blocking the web request loop kills user experience. If an endpoint generates a 40-page PDF report or calls a sluggish payment gateway, response times can easily spike from 45ms to over 4 seconds. Offloading heavy work to background workers keeps your API responsive.
Celery and Redis handle this delegation cleanly. Spin up a local Redis broker with Docker to get started:
docker run -d --name redis-broker -p 6379:6379 redis:7-alpine
Install the required Python dependencies:
pip install celery redis
Define your background task inside tasks.py:
from celery import Celery
import time
app = Celery(
"tasks",
broker="redis://localhost:6379/0",
backend="redis://localhost:6379/1"
)
@app.task
def process_signup_email(user_email: str) -> str:
time.sleep(2) # Simulates remote SMTP latency
return f"Welcome email dispatched to {user_email}"
Launch the worker process in your terminal:
celery -A tasks worker --loglevel=info --concurrency=2
Enqueue your first job from a Python REPL:
>>> from tasks import process_signup_email
>>> async_result = process_signup_email.delay("[email protected]")
>>> async_result.id
'2e9564f2-9860-498c-85a7-96a24eb5ef74'
>>> async_result.ready()
False
>>> async_result.get(timeout=5)
'Welcome email dispatched to [email protected]'
The caller hands off the payload in milliseconds, receives a UUID, and immediately frees the HTTP thread.
Deep dive
Scaling Celery requires understanding how the client, message transport, and OS processes coordinate workloads.
Message broker vs. result backend
Celery separates queueing logic from output tracking:
- Broker (Redis DB 0): Acts as the transport queue. Calling
delay()serializes task arguments into a JSON string and pushes it into a Redis list. The web worker disconnects right away. - Result Backend (Redis DB 1): Tracks execution status (
PENDING,SUCCESS,FAILURE) and return values. If you only care about side effects—like firing transactional emails—setignore_result=True. Disabling the backend removes temporary Redis keys entirely and eliminates 50% of your write IOPS.
Concurrency models: prefork vs. gevent
The default prefork pool relies on Python’s multiprocessing module. Because each worker process gets its own memory space, prefork bypasses the Global Interpreter Lock (GIL) for CPU-bound jobs such as image compression or data analysis. But OS processes are heavy. Spawning 32 prefork workers can consume 1.5 GB to 3 GB of RAM.
If your workload is primarily network I/O—like polling external REST APIs or scraping pages—switch to an event loop. A single gevent worker process uses less than 80 MB of RAM while managing hundreds of concurrent connections:
celery -A tasks worker --pool=gevent --concurrency=100 --loglevel=info
Task acknowledgments and idempotency
Celery confirms messages early by default. The broker marks a task complete the exact second a worker dequeues it. If Linux kills the worker process due to an OOM spike mid-execution, that task disappears without a trace.
Set task_acks_late = True to hold acknowledgment until the function successfully exits. However, this creates a trade-off. If a node crashes midway, Redis will redeliver the job to another worker. Design every database mutation and payment call to be idempotent using unique idempotency keys before turning this on.
Advanced usage
Orchestrating pipelines with Canvas
Production systems frequently require multi-step operations. You might need to charge a card, record an invoice in PostgreSQL, and send a push notification in sequence. Celery Canvas chains and groups assemble these pipelines cleanly without nested callbacks:
from celery import Celery, chain, group
app = Celery("ecommerce", broker="redis://localhost:6379/0", backend="redis://localhost:6379/1")
@app.task
def charge_customer(order_id: int, amount: float):
return {"order_id": order_id, "transaction_id": f"tx_{order_id}", "amount": amount}
@app.task
def generate_invoice(charge_data: dict):
invoice_number = f"INV-{charge_data['order_id']}"
return {**charge_data, "invoice": invoice_number}
@app.task
def notify_customer(invoice_data: dict):
return f"Notification sent for invoice {invoice_data['invoice']}"
@app.task
def sync_inventory_item(item_id: int):
return f"Item {item_id} stock decremented"
# Sequential pipeline: pipe operator passes output to the next task
workflow = chain(
charge_customer.s(order_id=4021, amount=99.50) |
generate_invoice.s() |
notify_customer.s()
)
workflow.apply_async()
# Parallel execution: run independent jobs concurrently
inventory_sync = group(
sync_inventory_item.s(item_id=101),
sync_inventory_item.s(item_id=102),
sync_inventory_item.s(item_id=103)
)
inventory_sync.apply_async()
Rate limiting external integrations
Third-party APIs enforce strict rate quotas. Twilio, for instance, rejects SMS requests exceeding 10 per second on standard numbers. Celery provides a built-in token-bucket rate limiter at the task definition level:
@app.task(rate_limit="10/s", max_retries=3, default_retry_delay=5)
def send_sms_alert(phone_number: str, message: str):
try:
# Remote SMS provider dispatch
pass
except ExternalServiceError as exc:
# Exponential backoff retry: 5s, 10s, 20s
countdown = 5 * (2 ** send_sms_alert.request.retries)
raise send_sms_alert.retry(exc=exc, countdown=countdown)
Keep in mind that Celery enforces rate_limit per worker instance rather than globally across your entire cluster. If you run 10 separate worker containers with rate_limit="10/s", your cluster can hit the provider with up to 100 requests per second. Use an atomic Redis token bucket or a proxy for hard global guarantees.
Practical tips
1. Isolate priority queues
Placing instant authentication codes and massive 2-hour report exports on the same default queue is a recipe for downtime. A burst of 5,000 PDF exports will occupy every worker slot. Password reset emails will sit waiting in the backlog.
Split tasks into dedicated queues across your routing config:
app.conf.task_routes = {
"tasks.send_sms_alert": {"queue": "priority"},
"tasks.generate_invoice": {"queue": "billing"},
"tasks.export_large_csv": {"queue": "batch"},
}
Run isolated worker processes dedicated to each queue:
# High-throughput priority worker
celery -A tasks worker -Q priority -c 4
# Dedicated batch worker (prevents heavy jobs from starving API queues)
celery -A tasks worker -Q batch -c 1
2. Enforce worker memory bounds
Libraries like Pandas, ReportLab, or Pillow frequently cause memory fragmentation in long-running processes. Child processes gradually bloat until Linux triggers a SIGKILL. Tell Celery to replace child workers after processing a designated volume of work:
# Restart child processes after 500 tasks or 200MB memory usage
app.conf.worker_max_tasks_per_child = 500
app.conf.worker_max_memory_per_child = 200000 # 200,000 KB (~195 MB)
3. Lower prefetch multipliers for uneven jobs
Celery grabs tasks in batches by default. With a prefetch_multiplier of 4, a concurrency-4 worker will reserve 16 tasks up front. If four tasks each require 5 minutes of processing, 12 remaining tasks sit idle in that worker’s local buffer while adjacent workers sit empty.
For long or variable jobs, set prefetching to 1 so unpicked tasks remain visible in Redis:
app.conf.worker_prefetch_multiplier = 1
app.conf.task_acks_late = True
4. Set explicit execution deadlines
An unhandled socket hang in an upstream HTTP request can freeze a worker indefinitely. Protect every task with explicit soft and hard timeout barriers:
# Soft timeout triggers a catchable SoftTimeLimitExceeded exception for cleanup
# Hard timeout sends SIGKILL to the child process 10 seconds later
app.conf.task_soft_time_limit = 120
app.conf.task_time_limit = 130
Configuring these boundaries protects your server memory, prevents stuck worker slots, and ensures predictable queue latency across production deployments.
