The Problem That Woke Me Up at 2 AM
It was 2:17 AM when my monitoring dashboard lit up. The self-hosted LLM inference server we’d been running for a customer-facing chatbot had crawled to a halt. Response latency spiked from 800ms to 45 seconds. Memory usage was flatlined at 100%. The service wasn’t crashed — it was just sitting there, completely saturated, unable to process a queue of 200 pending requests.
We were running a naive HuggingFace Transformers setup with a standard Flask API wrapper. It worked fine in staging with 5 concurrent users. Production had 300. I’d made a classic mistake: assuming that loading a model and calling model.generate() in a loop was a production-ready architecture.
It’s not. That night convinced me vLLM exists for a reason — and that if you’re running any LLM in production, understanding it now is cheaper than debugging it at 2 AM.
Root Cause: Why Naive LLM Serving Breaks Under Load
Standard transformer inference has a fundamental memory problem. Each inference request requires a KV (key-value) cache that holds attention states across all generation steps. With a naive implementation, that cache gets allocated statically. Worst case: you pre-allocate memory for the maximum sequence length for every request — whether the actual output ends up being 5 tokens or 500.
Under concurrent load, this means:
- GPU memory fills up with mostly empty KV cache allocations
- New requests get queued or rejected because there’s no memory available
- Throughput collapses as the GPU sits idle waiting for memory to free up
The Flask wrapper I had was processing one request at a time (synchronous). Every request held GPU memory for its entire duration, blocking everything else. 300 users × sequential processing = disaster.
Solutions I Looked At
Option 1: TGI (Text Generation Inference by HuggingFace)
HuggingFace’s own serving solution. Solid, well-documented, Rust-based for the HTTP layer. The limitation: it uses continuous batching but the memory management isn’t as aggressive as vLLM. For large models on consumer-grade GPUs, I found it hit OOM errors sooner under burst load.
Option 2: Triton Inference Server (NVIDIA)
Extremely powerful, used in enterprise setups. The setup complexity is real though — you need model-specific backends, configuration files, and deep NVIDIA tooling knowledge. Overkill for a team that just needs to serve one or two models reliably.
Option 3: vLLM
Python-native, OpenAI API-compatible, and built specifically around a technique called PagedAttention. This is what I ended up using, and it’s what I’d recommend for 90% of teams running LLMs on their own hardware.
What Makes vLLM Different: PagedAttention Explained
PagedAttention is the core innovation. Borrowed from virtual memory concepts in operating systems, it breaks KV cache into fixed-size blocks (“pages”) that are allocated on demand, not pre-allocated per request.
Practically, this means:
- Short responses don’t waste memory reserved for long ones
- Memory from completed requests gets immediately reclaimed
- The GPU can serve far more concurrent requests with the same VRAM
On top of that, vLLM implements continuous batching. Instead of waiting to assemble a full batch, it slots new requests into the running batch as capacity opens up. GPU utilization stays high even during bursty, uneven traffic.
The result: in my benchmarks, vLLM handled 8–12× higher throughput than the naive HuggingFace setup on the same hardware.
Deploying vLLM with Docker on Linux
Prerequisites
- Linux host (Ubuntu 20.04+ recommended)
- NVIDIA GPU with at least 16GB VRAM (24GB+ for 7B models at full precision)
- NVIDIA drivers 525+ installed
- Docker Engine with
nvidia-container-toolkit - HuggingFace account with model access (if using gated models)
Step 1: Install NVIDIA Container Toolkit
If you haven’t already set up GPU access in Docker:
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart dockerVerify GPU access in containers:
docker run --rm --gpus all nvidia/cuda:12.1.0-base-ubuntu22.04 nvidia-smiStep 2: Pull and Run vLLM
vLLM maintains an official Docker image. For a quick test with meta-llama/Llama-3.2-3B-Instruct (requires HuggingFace token):
docker run --gpus all \
-v ~/.cache/huggingface:/root/.cache/huggingface \
-e HUGGING_FACE_HUB_TOKEN=your_token_here \
-p 8000:8000 \
--ipc=host \
vllm/vllm-openai:latest \
--model meta-llama/Llama-3.2-3B-Instruct \
--dtype bfloat16Key flags here:
--ipc=host: Required for shared memory between processes (important for performance)-v ~/.cache/huggingface: Mount the cache so models aren’t re-downloaded on each restart--dtype bfloat16: Half-precision inference, reduces VRAM usage by ~50% with minimal quality loss
Step 3: Docker Compose for Production
For production, use a proper Compose file:
version: '3.8'
services:
vllm:
image: vllm/vllm-openai:latest
restart: unless-stopped
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: 1
capabilities: [gpu]
volumes:
- ~/.cache/huggingface:/root/.cache/huggingface
environment:
- HUGGING_FACE_HUB_TOKEN=${HF_TOKEN}
ports:
- "8000:8000"
ipc: host
command: >
--model meta-llama/Llama-3.2-3B-Instruct
--dtype bfloat16
--max-model-len 4096
--gpu-memory-utilization 0.90
--tensor-parallel-size 1The --gpu-memory-utilization 0.90 flag tells vLLM to use 90% of VRAM for the KV cache pool. I leave 10% headroom to avoid OOM errors. If you’re seeing OOM crashes, drop this to 0.85.
Step 4: Testing the OpenAI-Compatible API
The API is a drop-in replacement for OpenAI. Once the server starts — watch for the Uvicorn running on http://0.0.0.0:8000 log line — hit it with curl:
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "meta-llama/Llama-3.2-3B-Instruct",
"messages": [{"role": "user", "content": "Explain KV cache in one sentence."}],
"max_tokens": 100
}'Or using the OpenAI Python SDK, pointing it at your local server:
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:8000/v1",
api_key="not-needed", # vLLM doesn't require auth by default
)
response = client.chat.completions.create(
model="meta-llama/Llama-3.2-3B-Instruct",
messages=[{"role": "user", "content": "What is PagedAttention?"}],
max_tokens=200,
temperature=0.7,
)
print(response.choices[0].message.content)Any code that works with the OpenAI SDK works with vLLM — zero application changes required.
Tuning for Real Load
Increasing Concurrency
Add these flags based on your hardware:
# For multi-GPU setups (e.g., 2× A100)
--tensor-parallel-size 2
# Limit max concurrent sequences to prevent memory spikes
--max-num-seqs 128
# Enable prefix caching for repeated system prompts (huge win for chatbots)
--enable-prefix-cachingThat last flag made the biggest difference for my chatbot use case. If every request shares a long system prompt, vLLM caches its KV state after the first request and reuses it for all subsequent ones. The prompt portion costs near-zero compute. On a 1,000-token system prompt hitting 10,000 requests a day, that’s a meaningful reduction in GPU time.
Quantization for Lower VRAM
Running a 13B model on a 16GB GPU? Use AWQ or GPTQ quantization:
docker run --gpus all -p 8000:8000 --ipc=host \
vllm/vllm-openai:latest \
--model TheBloke/Llama-2-13B-chat-AWQ \
--quantization awq \
--dtype float16AWQ 4-bit cuts VRAM usage roughly in half. Throughput takes a small hit, but for chatbot workloads the output quality difference is barely noticeable. It’s not ideal for everything — if you need precise or deterministic outputs, benchmark carefully before committing.
What I’d Do Differently That Night
If I could go back, I’d deploy vLLM from day one instead of the HuggingFace Transformers hack. The API compatibility means zero migration effort — just update the base URL in your application config. The Docker image handles all the CUDA setup. And PagedAttention makes the memory behavior predictable enough that you can actually capacity-plan based on VRAM instead of guessing.
One thing that caught me off guard: the first startup takes 10–20 minutes while vLLM loads and optimizes the model. Add a proper health check in your Docker Compose or load balancer — wait for the /health endpoint to return 200 before routing traffic.
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
interval: 30s
timeout: 10s
retries: 3
start_period: 300sThat start_period: 300s gives vLLM 5 minutes to initialize before Docker starts counting health check failures. Without it, Docker will restart the container thinking it’s unhealthy during the normal startup phase.
The 2 AM incident cost us 40 minutes of degraded service. After migrating to vLLM, the same server handles 300 concurrent users with latency back under a second. The root cause was never the hardware. It was the memory management — and that’s the part a framework change actually fixes.
