Why Your LLM Inference Is Slower Than It Should Be
You’ve got a powerful NVIDIA GPU, a well-trained language model, and a production deployment to serve — yet inference latency sits at 2–3 seconds per request, and throughput falls apart past 10 concurrent users. The bottleneck isn’t your hardware. It’s how the model is being executed.
Most teams start with Hugging Face Transformers and PyTorch. Fair enough for prototyping. But PyTorch’s eager execution mode wasn’t built for high-throughput serving — it leaves a surprising amount of GPU compute unused, particularly when you’re running repeated inference on a fixed model architecture.
The root cause is missing graph optimization. Each forward pass recomputes operator fusion, memory layout, and kernel selection from scratch. TensorRT-LLM fixes this by compiling the model into an optimized TensorRT engine once, then reusing it for every request. On the same hardware, that typically translates to 3–5x higher throughput and 50–70% lower first-token latency.
If you’re running LLMs as a production service — not just a demo — this optimization matters more than most hardware upgrades. Getting it right means your A100 serves far more users on the same GPU budget.
Installation: Setting Up TensorRT-LLM on Docker
NVIDIA’s official Docker images take the pain out of environment setup. CUDA, cuDNN, TensorRT, and Python bindings come pre-installed and version-matched — which alone saves a few hours of dependency hell.
Prerequisites
- NVIDIA GPU with Ampere architecture or newer (A100, A10, RTX 3090/4090, H100)
- NVIDIA driver ≥ 525.85
- Docker Engine with NVIDIA Container Toolkit installed
- At least 20GB disk space for the image and model weights
First, verify your NVIDIA Container Toolkit is working:
docker run --rm --gpus all nvidia/cuda:12.1.0-base-ubuntu22.04 nvidia-smi
Your GPU should appear in the output. If it doesn’t, install the toolkit first:
# Install NVIDIA Container Toolkit
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
Pull the TensorRT-LLM Docker Image
NVIDIA publishes TensorRT-LLM images on their container registry. Pick the version that matches your CUDA driver:
# Pull TensorRT-LLM image (adjust version as needed)
docker pull nvcr.io/nvidia/tritonserver:24.05-trtllm-python-py3
# Or the lighter TensorRT-LLM development image
docker pull nvcr.io/nvidia/tensorrt-llm:0.11.0
Start a container with GPU access and mount a local directory for model weights:
docker run --gpus all \
--ipc=host \
--ulimit memlock=-1 \
--ulimit stack=67108864 \
-v /path/to/models:/models \
-p 8000:8000 \
-it nvcr.io/nvidia/tensorrt-llm:0.11.0 bash
The --ipc=host flag matters here — it enables shared memory between container processes, which TensorRT-LLM needs for multi-GPU tensor parallelism.
Configuration: Building and Serving Your First TensorRT Engine
TensorRT-LLM works in two stages: compile the model into a TensorRT engine, then serve it. Compilation is where the real work happens — CUDA kernels get fused, memory layouts get optimized, and the engine gets locked to your specific GPU hardware.
Step 1: Convert Model Weights
Using Llama-3-8B as an example, start by downloading the model inside the container:
# Install Hugging Face hub to download the model
pip install huggingface_hub
# Download model weights (requires HF token for gated models)
huggingface-cli download meta-llama/Meta-Llama-3-8B \
--local-dir /models/llama3-8b \
--token YOUR_HF_TOKEN
Then convert the weights to TensorRT-LLM checkpoint format:
cd /app/tensorrt_llm/examples/llama
python convert_checkpoint.py \
--model_dir /models/llama3-8b \
--output_dir /models/llama3-8b-trtllm-ckpt \
--dtype float16 \
--tp_size 1
On A100 or H100, switch to --dtype bfloat16 for better numerical stability. For multi-GPU setups, set --tp_size 2 (or 4, 8) to spread the model across GPUs via tensor parallelism.
Step 2: Build the TensorRT Engine
This compiles the model graph into an optimized binary. Budget 5–15 minutes depending on your GPU and model size:
trtllm-build \
--checkpoint_dir /models/llama3-8b-trtllm-ckpt \
--output_dir /models/llama3-8b-engine \
--gemm_plugin float16 \
--max_batch_size 32 \
--max_input_len 2048 \
--max_seq_len 4096 \
--use_paged_context_fmha enable
Key parameters to tune for your workload:
--max_batch_size: Maximum concurrent requests. Higher values improve throughput but increase memory usage.--max_input_len+--max_seq_len: Set these based on your actual use case. Larger values eat more KV cache memory.--use_paged_context_fmha enable: Enables paged KV cache attention — essential for handling variable-length sequences without wasting memory on padding.
Step 3: Launch with Triton Inference Server
Pair TensorRT-LLM with NVIDIA Triton for production serving. Create a model repository structure first:
mkdir -p /triton_model_repo/tensorrt_llm/1
# Copy the engine files
cp /models/llama3-8b-engine/* /triton_model_repo/tensorrt_llm/1/
# Create Triton config
cat > /triton_model_repo/tensorrt_llm/config.pbtxt << 'EOF'
name: "tensorrt_llm"
backend: "tensorrtllm"
max_batch_size: 32
parameters {
key: "executor_worker_path"
value: { string_value: "/opt/tritonserver/backends/tensorrtllm/trtllmExecutorWorker" }
}
parameters {
key: "max_tokens_in_paged_kv_cache"
value: { string_value: "2560" }
}
parameters {
key: "batch_scheduler_policy"
value: { string_value: "inflight_fused_batching" }
}
EOF
The inflight_fused_batching scheduler is what separates production-grade serving from naive batch processing. Instead of waiting for the current batch to finish, it continuously merges incoming requests into active batches. GPU utilization stays high even when traffic patterns are uneven.
Start the Triton server:
tritonserver \
--model-repository /triton_model_repo \
--http-port 8000 \
--grpc-port 8001 \
--metrics-port 8002
Verification and Monitoring: Confirm It’s Actually Faster
Measure it. Don’t take it on faith that the deployment is performing well — TensorRT-LLM and Triton expose enough telemetry to confirm exactly how much latency and throughput you’ve gained.
Quick Functional Test
import tritonclient.http as httpclient
import numpy as np
import json
client = httpclient.InferenceServerClient(url="localhost:8000")
# Check server health
print("Server ready:", client.is_server_ready())
print("Model ready:", client.is_model_ready("tensorrt_llm"))
# Send a test inference request
inputs = [
httpclient.InferInput("text_input", [1, 1], "BYTES"),
httpclient.InferInput("max_tokens", [1, 1], "INT32"),
]
prompt = "Explain Docker networking in simple terms."
inputs[0].set_data_from_numpy(np.array([[prompt.encode()]], dtype=object))
inputs[1].set_data_from_numpy(np.array([[256]], dtype=np.int32))
outputs = [httpclient.InferRequestedOutput("text_output")]
result = client.infer("tensorrt_llm", inputs, outputs=outputs)
print(result.as_numpy("text_output")[0][0].decode())
Throughput Benchmarking
TensorRT-LLM ships with a built-in benchmarking tool. Run it against your engine to get baseline numbers:
python /app/tensorrt_llm/benchmarks/python/benchmark.py \
-m llama \
--engine_dir /models/llama3-8b-engine \
--batch_size 1 8 16 32 \
--input_output_len 512,128 \
--dtype float16
Run the same benchmark against a vanilla PyTorch baseline to put the numbers in context. On an A100 80GB, Llama-3-8B typically goes from around 15 tokens/sec with PyTorch to 80–120 tokens/sec with TensorRT-LLM at batch size 1. The gap widens further at higher batch sizes.
Monitor GPU Utilization and KV Cache
Triton exposes Prometheus-compatible metrics at port 8002:
# Check key metrics
curl -s http://localhost:8002/metrics | grep -E \
'nv_inference_request_success|nv_inference_queue_duration|kv_cache_utilization'
Pay close attention to kv_cache_utilization. If it stays above 90% under load, either raise max_tokens_in_paged_kv_cache in your Triton config or lower max_batch_size. KV cache pressure — not compute — is the most common source of latency spikes in production TensorRT-LLM deployments.
Wire these metrics into Prometheus and Grafana for ongoing visibility. The panels worth tracking: tokens per second, time-to-first-token (p50/p99), queue depth, and KV cache utilization over time.
When to Rebuild the Engine
The TensorRT engine is baked for your exact GPU model, CUDA version, and compile-time parameters. It needs a rebuild when:
- Upgrading to a different GPU generation
- Changing CUDA or TensorRT versions
- Adjusting max batch size or max sequence length
- Switching between FP16 and BF16 precision
Keep a reproducible build script in version control. Rebuilds should be a single command, not a debugging session. The compile time — typically 10–15 minutes for an 8B model — is a one-time cost. After that, every inference call runs against the optimized engine.

