Quick Start: Double Your Token Speed in 5 Minutes
Serving large language models in interactive apps always hits the same wall: memory bandwidth. In autoregressive decoding, generating each token demands reading every single weight from high-bandwidth memory (HBM). On a 70B parameter model running across two A100s, this memory-bound step slows generation to a crawl—often peaking at just 18 to 22 tokens per second at batch size 1.
Speculative decoding solves this bottleneck. A tiny, lightweight draft model quickly generates several candidate tokens. Then, your large target model verifies all candidates simultaneously in one forward pass. The compute cost is nearly identical to generating one token, but you pocket 3 to 5 tokens at once.
Here is how to set up working pipelines in both vLLM and llama.cpp.
Option A: Fast Setup with vLLM
If you already run vLLM on an NVIDIA GPU cluster, enabling speculative decoding requires only three additional command-line flags pointing to an aligned smaller draft model.
pip install vllm
# Serve Llama-3.1-70B-Instruct with Llama-3.1-8B-Instruct as the draft model
vllm serve meta-llama/Meta-Llama-3.1-70B-Instruct \
--speculative-model meta-llama/Meta-Llama-3.1-8B-Instruct \
--num-speculative-tokens 5 \
--tensor-parallel-size 4 \
--port 8000After the weights finish loading, fire a test completion to verify the engine:
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "meta-llama/Meta-Llama-3.1-70B-Instruct",
"messages": [{"role": "user", "content": "Write a Python function to parse JSONL files safely."}],
"temperature": 0.1
}'Option B: Low-Resource Setup with llama.cpp
On Apple Silicon, local workstations, or edge nodes, llama.cpp provides native speculative sampling right out of the box via the -md (model draft) flag.
# Build llama.cpp with Metal / CUDA support
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp && cmake -B build && cmake --build build --config Release -j
# Run Qwen2.5-14B with Qwen2.5-1.5B as the draft model
./build/bin/llama-speculative \
-m models/qwen2.5-14b-instruct-q4_k_m.gguf \
-md models/qwen2.5-1.5b-instruct-q4_k_m.gguf \
-p "Explain TCP flow control in three bullet points:" \
-ngl 99 -ngld 99 \
--draft-max 6On an M2 Pro Mac (32GB unified memory), this configuration took our generation rate from 17.8 tokens/sec straight to 33.4 tokens/sec—an immediate 87% speedup without code changes.
How Speculative Decoding Actually Works
Why does this trick work without hurting output quality? It comes down to arithmetic intensity.
The Memory Bandwidth Bottleneck
Standard token generation is desperately memory bound. For every token emitted, the GPU reads tens of gigabytes of weights into on-chip SRAM, performs a tiny matrix-vector multiplication, and discards those weights. Raw compute cores sit idle up to 80% of the time.
Batching multiple tokens changes the equation. Processing a prompt containing 6 tokens takes roughly the same elapsed time on modern GPUs as processing a single token. Speculative decoding exploits this idle compute capacity.
The Two-Tier Execution Loop
The decoding cycle operates in three tight stages:
- Draft Step: A compact sibling model (e.g., 1.5B or 8B) runs autoregressively for K iterations (typically 4 to 6 tokens). Because the draft model is small, memory transfers finish in milliseconds.
- Parallel Verification: The large target model processes the prompt together with all K candidate tokens in a single parallel forward pass.
- Acceptance & Correction: The target model compares draft token probabilities against its own. It accepts valid tokens in sequence until finding a mismatch. At that first incorrect token, the target model samples the replacement directly from its own distribution.
Does this change the output? No. Speculative decoding uses modified rejection sampling. Mathematically, the output distribution matches the target model’s true probability distribution down to the bit. You preserve identical reasoning depth while cutting end-to-end latency in half.
Advanced Production Configuration
Getting speculative decoding to run on a single prompt is easy. Maintaining throughput across thousands of concurrent production requests requires deliberate sizing and tuning. Over the past six months, we have handled over 14 million daily search and coding requests using this exact architecture.
1. Zero-VRAM Drafting via Prompt Lookup
You do not always need a secondary neural network. When GPU memory is already saturated, heuristic drafting like Prompt Lookup Decoding or N-gram caching delivers massive gains at zero extra VRAM cost.
from vllm import LLM, SamplingParams
sampling_params = SamplingParams(temperature=0.0, max_tokens=256)
# Configure n-gram speculative decoding without loading a draft model
llm = LLM(
model="mistralai/Mistral-7B-Instruct-v0.3",
speculative_model="[ngram]",
num_speculative_tokens=5,
ngram_prompt_lookup_max=4,
trust_remote_code=True
)
prompts = ["Summarize customer service transcript: Order #9823 was delayed due to weather in Chicago..."]
outputs = llm.generate(prompts, sampling_params)
for output in outputs:
print(output.outputs[0].text)In document summarization, code refactoring, and structured extraction, output tokens frequently repeat terms from the prompt. N-gram speculative lookup yields acceptance rates between 58% and 64% on these workloads while keeping memory footprints completely flat.
2. Calibrating Speculative Lookahead (K)
The --num-speculative-tokens flag specifies how many candidate tokens the draft model generates before target verification. Picking this number is an optimization problem:
- K = 3 to 4: Best for open-ended prose, creative text, or complex multi-step reasoning where draft acceptance dips below 50%.
- K = 5 to 6: The sweet spot for structured generation (JSON payloads, SQL syntax, schema validation). High token predictability keeps acceptance above 75%.
- K > 8: Diminishing returns. If the draft engine misses token 2, verifying tokens 3 through 8 simply burns target GPU cycles for zero yield.
3. Allocating VRAM on Multi-GPU Nodes
The draft model shares physical VRAM with the primary model. Loading an FP16 8B draft model demands roughly 16GB of VRAM, which cuts straight into your KV cache allocation and drops your maximum concurrent concurrency ceiling.
To avoid KV cache thrashing on a 4x A100 (80GB) node serving Llama-3.1-70B, quantize the draft model to FP8 or pin it cleanly across the topology:
vllm serve meta-llama/Meta-Llama-3.1-70B-Instruct \
--tensor-parallel-size 4 \
--speculative-model neuralmagic/Meta-Llama-3.1-8B-Instruct-FP8 \
--speculative-draft-tensor-parallel-size 1 \
--gpu-memory-utilization 0.90 \
--max-num-seqs 64Field Notes from Production
Enforce Strict Tokenizer Alignment
Draft and target models must share an identical vocabulary and tokenization scheme. Pairing a Llama-3 target with a Qwen draft model crashes immediately because their token ID tables differ. Stick to model pairs from the same architectural lineage:
- Target:
Llama-3.1-70B→ Draft:Llama-3.1-8BorLlama-3.2-3B - Target:
Qwen2.5-72B→ Draft:Qwen2.5-1.5BorQwen2.5-7B - Target:
DeepSeek-Coder-33B→ Draft:DeepSeek-Coder-1.3B
Keep Temperatures Below 0.4
Speculative decoding produces its largest performance gains when token distributions are sharp. At temperature = 0.0 (greedy decoding), target and draft models align most frequently, sustaining acceptance rates above 72%.
As temperature climbs toward 0.8 or 1.0, acceptance rates degrade toward 25%. At that threshold, draft overhead completely wipes out verification speedups, leading to higher overall latency than baseline autoregression.
Track Acceptance Rates in Prometheus
Expose vLLM metrics to Prometheus and set an alert on vllm:spec_decode_draft_acceptance_rate. In healthy production clusters, this metric tracks between 0.55 and 0.75. If acceptance sinks below 0.40, disable speculative decoding dynamically or drop down to n-gram lookup to stop wasting server capacity.
