Build a Centralized AI Gateway with LiteLLM and Docker: Key Management, Load Balancing, and Cost Controls

Table of Contents

The Messy Reality of Multi-LLM Deployments

Last quarter, our infrastructure was in chaos. We ran three core services—a FastAPI document parser, a Node.js notification engine, and an internal Slack bot—alongside six engineers testing prompt pipelines. Every single system held raw OpenAI and Anthropic API keys in separate .env files. Nobody knew who was spending what.

Then things broke. An engineer kicked off an evaluation script that flooded OpenAI with 3,200 requests per minute. That blew past our 1,000 RPM Tier-3 rate limit and starved production traffic for 20 minutes. Three weeks later, an unmonitored test script racked up $1,400 in Claude Opus calls over a single weekend. Our monthly LLM spend shot up 45% with zero attribution.

Scattered keys and zero traffic governance are a recipe for downtime. Hardcoding provider secrets across multiple repos also creates a security headache. To fix this, we put a reverse proxy between our applications and upstream AI providers: a centralized AI gateway.

Why LiteLLM?

Writing a custom LLM proxy from scratch takes weeks of maintenance. LiteLLM solves this out of the box. It runs as a lightweight, OpenAI-compatible proxy server. That means any Python script, LangChain agent, or cURL request written for the OpenAI SDK works instantly—you only swap the base URL.

Here is what this setup gives you:

Unified API Contract: Upstream requests to Gemini 1.5 Pro, Claude 3.5 Sonnet, or self-hosted vLLM instances all accept standard /v1/chat/completions payloads.
Virtual Tenant Keys: Services receive scoped virtual tokens (prefixed with sk-) instead of your master provider secrets. You can tie each token to specific models, requests-per-minute (RPM) quotas, and dollar budgets.
Smart Load Balancing & Fallbacks: Traffic splits cleanly across multiple organization keys or regions. If GPT-4o throws a 429 or 503 error, the proxy transparently reroutes the payload to Claude 3.5 Sonnet within milliseconds.
Automated Cost Capping: PostgreSQL tracks token consumption in real time. Once an intern’s virtual key hits its $50 monthly limit, LiteLLM rejects further calls automatically.

Step-by-Step: Deploying LiteLLM with Docker

We will configure a production-ready LiteLLM proxy paired with PostgreSQL for user-key tracking, spend logging, and health checks.

Step 1: Set Up Project Directory and Secrets

Create a dedicated folder on your host machine for the configuration files:

mkdir -p /opt/ai-gateway && cd /opt/ai-gateway
touch docker-compose.yml litellm_config.yaml .env
chmod 600 .env

Add your database password and master provider API keys to .env:

# .env
POSTGRES_USER=litellm
POSTGRES_PASSWORD=Secr3t_Postgres_P@ss!2024
POSTGRES_DB=litellm_proxy

LITELLM_MASTER_KEY=sk-admin-master-prod-9988
OPENAI_API_KEY_PRIMARY=sk-proj-primaryKey-12345
OPENAI_API_KEY_SECONDARY=sk-proj-backupKey-67890
ANTHROPIC_API_KEY=sk-ant-liveKey-abcdef

Step 2: Configure Model Routing and Automated Failovers

Define your routing rules inside litellm_config.yaml. The setup below splits GPT-4o requests evenly between two OpenAI accounts (1,000 RPM each) and fails over to Anthropic Claude 3.5 Sonnet if both return errors.

model_list:
  - model_name: gpt-4o
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_API_KEY_PRIMARY
      rpm: 1000
  - model_name: gpt-4o
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_API_KEY_SECONDARY
      rpm: 1000
  - model_name: gpt-4o-fallback
    litellm_params:
      model: anthropic/claude-3-5-sonnet-20241022
      api_key: os.environ/ANTHROPIC_API_KEY

router_settings:
  routing_strategy: usage-based-routing-v2
  fallbacks:
    - gpt-4o: ["gpt-4o-fallback"]
  num_retries: 2
  retry_after: 1
  timeout: 30

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  database_url: postgresql://${POSTGRES_USER}:${POSTGRES_PASSWORD}@db:5432/${POSTGRES_DB}

Step 3: Define Services in Docker Compose

Add PostgreSQL 16 and the LiteLLM proxy container to docker-compose.yml. A health check ensures LiteLLM does not boot until the database socket accepts connections.

version: '3.8'

services:
  db:
    image: postgres:16-alpine
    restart: unless-stopped
    environment:
      POSTGRES_USER: ${POSTGRES_USER}
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
      POSTGRES_DB: ${POSTGRES_DB}
    volumes:
      - pgdata:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U ${POSTGRES_USER} -d ${POSTGRES_DB}"]
      interval: 5s
      timeout: 3s
      retries: 5

  litellm:
    image: ghcr.io/berriai/litellm:main-latest
    restart: unless-stopped
    ports:
      - "4000:4000"
    environment:
      DATABASE_URL: "postgresql://${POSTGRES_USER}:${POSTGRES_PASSWORD}@db:5432/${POSTGRES_DB}"
      LITELLM_MASTER_KEY: ${LITELLM_MASTER_KEY}
      OPENAI_API_KEY_PRIMARY: ${OPENAI_API_KEY_PRIMARY}
      OPENAI_API_KEY_SECONDARY: ${OPENAI_API_KEY_SECONDARY}
      ANTHROPIC_API_KEY: ${ANTHROPIC_API_KEY}
    volumes:
      - ./litellm_config.yaml:/app/config.yaml
    command: ["--config", "/app/config.yaml", "--port", "4000", "--detailed_debug"]
    depends_on:
      db:
        condition: service_healthy

volumes:
  pgdata:

Start the containers in detached mode:

docker compose up -d
docker compose logs -f litellm

Look for LiteLLM: Proxy running on http://0.0.0.0:4000 in the logs to confirm the database migrations completed.

Step 4: Generate Scoped, Budget-Capped Virtual Keys

Never share your master provider keys again. Use LiteLLM’s management API to generate scoped virtual tokens for teams or environments. Here is how to create a key for a staging team with a strict $50 monthly cap:

curl -s -X POST http://localhost:4000/key/generate \
  -H "Authorization: Bearer sk-admin-master-prod-9988" \
  -H "Content-Type: application/json" \
  -d '{
    "models": ["gpt-4o"],
    "max_budget": 50.0,
    "budget_duration": "30d",
    "user_id": "team-analytics-staging",
    "metadata": {"environment": "staging", "cost_center": "cc-402"}
  }' | jq

LiteLLM returns a virtual key formatted like sk-litellm-1234.... The proxy logs token counts from every response. The second this key accumulates $50.00 in usage, LiteLLM cuts it off with an HTTP 400 BudgetExceededError. Production keys remain completely unaffected.

Step 5: Switch Client Code to the Gateway

Integrating existing apps takes under two minutes. Just change base_url to your gateway endpoint and supply the virtual token:

from openai import OpenAI

# Point client to the LiteLLM reverse proxy
client = OpenAI(
    base_url="http://localhost:4000/v1",
    api_key="sk-litellm-analytics-key-here"
)

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[
        {"role": "system", "content": "You are a site reliability engineer."},
        {"role": "user", "content": "Explain how TCP connection pools prevent thread starvation."}
    ]
)

print(response.choices[0].message.content)

Our microservices have run against this gateway setup for six months without incident. During a major OpenAI latency spike in November where p99 times exceeded 14 seconds, LiteLLM automatically shifted 800 live requests over to Claude 3.5 Sonnet. Our end users saw zero dropped connections.

Next Steps for Production Readiness

Putting an AI gateway in place ends secret sprawl and uncontrolled spending. Your primary vendor keys remain safely isolated inside the proxy host. Load balancing happens at the networking tier, where it belongs.

Before moving to production, put a reverse proxy like Traefik or Caddy in front of port 4000 for TLS termination and rate-limiting. Finally, enable LiteLLM’s built-in /metrics endpoint to pipe latency histograms, token burn, and error rates straight into Prometheus and Grafana dashboards.