Context & Why
Spinning up a FastAPI or Flask container with a /predict route is fine for prototypes handling 5 to 10 requests per minute. But push that setup into production with a multimodal pipeline—say, a ResNet-50 vision model alongside a RoBERTa classifier—and Python-based microservices fall apart under load.
Python’s Global Interpreter Lock (GIL), JSON serialization penalties, and lack of native GPU queue management cripple hardware efficiency. On an NVIDIA A10G or T4 instance, GPU utilization often stalls at a disappointing 15% to 20%. When concurrent requests hit typical web workers, they execute sequentially. Expensive Tensor Cores sit starved for work while waiting on PCIe host-to-device memory copies, sending p99 latency climbing from 15ms past 600ms.
NVIDIA Triton Inference Server eliminates this bottleneck with a high-throughput C++ core. Triton acts as a unified inference gateway across TensorRT, ONNX Runtime, PyTorch, and OpenVINO without requiring custom Python wrappers.
Its standout capability is dynamic batching. Instead of executing requests one-by-one as they land, Triton pools individual incoming payloads inside a microsecond window into a single batch on the GPU. Clients query the API independently, yet the GPU processes them collectively.
In benchmarks on an NVIDIA L4 GPU, migrating from a Uvicorn/FastAPI stack to Triton with dynamic batching routinely jumps throughput from 120 QPS to 980 QPS, cutting p95 latency by roughly 65%.
Installation
GPU-accelerated Triton containers require an NVIDIA GPU with compute capability 6.0+, compatible host drivers (CUDA 12+), Docker Engine, and the NVIDIA Container Toolkit.
1. Verify NVIDIA Drivers and Container Toolkit
Verify that the host operating system communicates directly with your GPU:
nvidia-smiNext, confirm that Docker can access the GPU through the NVIDIA Container Toolkit runtime:
docker run --rm --gpus all nvidia/cuda:12.2.0-base-ubuntu22.04 nvidia-smiA successful run prints your GPU status table directly from the container environment.
2. Pull the Official Triton Docker Image
NVIDIA hosts ready-to-run Triton images on the public NGC catalog. You do not need an account or API key to pull standard releases:
docker pull nvcr.io/nvidia/tritonserver:24.08-py3The server listens on three default networking ports:
- 8000: HTTP/REST inference and management API
- 8001: gRPC inference API (preferred for low-latency production pipelines)
- 8002: Prometheus metrics endpoint
Configuration
Triton discovers and manages models through a directory tree called the Model Repository. Unlike Python scripts with hardcoded filesystem paths, Triton dynamically monitors this repository layout for model binaries and metadata.
1. Set Up the Model Repository Structure
Create the directory structure for two separate models—ResNet-50 for vision and a text classifier for NLP:
mkdir -p model_repository/resnet50_onnx/1
mkdir -p model_repository/text_classifier_onnx/1Every model folder requires numbered version subdirectories alongside a config.pbtxt schema:
model_repository/
├── resnet50_onnx/
│ ├── 1/
│ │ └── model.onnx
│ └── config.pbtxt
└── text_classifier_onnx/
├── 1/
│ └── model.onnx
└── config.pbtxt2. Configure ResNet50 with Dynamic Batching
Write the configuration for model_repository/resnet50_onnx/config.pbtxt:
name: "resnet50_onnx"
platform: "onnxruntime_onnx"
max_batch_size: 64
input [
{
name: "input"
data_type: TYPE_FP32
dims: [ 3, 224, 224 ]
}
]
output [
{
name: "output"
data_type: TYPE_FP32
dims: [ 1000 ]
}
]
dynamic_batching {
preferred_batch_size: [ 8, 16, 32, 64 ]
max_queue_delay_microseconds: 5000
}
instance_group [
{
count: 2
kind: KIND_GPU
gpus: [ 0 ]
}
]Here is what each parameter controls:
max_batch_size: 64: Activates Triton’s dynamic scheduler. Triton automatically treats the first dimension of every input tensor as the batch axis.dynamic_batching: Specifies batch assembly targets. Triton collects incoming requests for up to 5,000 microseconds (5ms). If 8 or 16 requests arrive in just 1.5ms, it dispatches immediately without waiting for the full window.instance_group: Launches two parallel model execution instances on GPU 0. While instance 1 executes compute kernels on Tensor Cores, instance 2 transfers incoming image tensors over PCIe.
3. Configure the Second Model (Text Classifier)
Now add the configuration for model_repository/text_classifier_onnx/config.pbtxt:
name: "text_classifier_onnx"
platform: "onnxruntime_onnx"
max_batch_size: 32
input [
{
name: "input_ids"
data_type: TYPE_INT64
dims: [ 128 ]
},
{
name: "attention_mask"
data_type: TYPE_INT64
dims: [ 128 ]
}
]
output [
{
name: "probabilities"
data_type: TYPE_FP32
dims: [ 2 ]
}
]
dynamic_batching {
preferred_batch_size: [ 4, 8, 16, 32 ]
max_queue_delay_microseconds: 2000
}
instance_group [
{
count: 1
kind: KIND_GPU
gpus: [ 0 ]
}
]Triton’s scheduler handles request queues for both models concurrently on one GPU. It swaps execution context in microseconds, avoiding separate container overhead.
4. Launch Triton via Docker
Mount the repository and run Triton in detached mode:
docker run -d --name triton-server \
--gpus all \
--shm-size=1g \
-p 8000:8000 \
-p 8001:8001 \
-p 8002:8002 \
-v $(pwd)/model_repository:/models \
nvcr.io/nvidia/tritonserver:24.08-py3 \
tritonserver --model-repository=/models --strict-model-config=truePay close attention to --shm-size=1g. Docker defaults to a tiny 64MB shared memory allocation. Triton relies on POSIX shared memory for fast IPC; starving it causes silent SIGBUS crashes under high concurrency.
Verification & Monitoring
Confirm that both models loaded properly and verify inference behavior before routing live user traffic.
1. Check Server Health and Model Status
Inspect container initialization logs directly:
docker logs -f triton-serverThe console prints an ASCII table confirming both resnet50_onnx and text_classifier_onnx are in the READY state. You can also validate health programmatically over HTTP:
# Verify server liveliness
curl -v http://localhost:8000/v2/health/live
# Verify ResNet50 model readiness
curl -v http://localhost:8000/v2/models/resnet50_onnx/readyBoth calls must return HTTP status 200 OK.
2. Execute a Test Inference Request via Python
Install the official client library in your local Python environment:
pip install tritonclient[http] numpyRun this Python test script to send a sample payload:
import numpy as np
import tritonclient.http as httpclient
client = httpclient.InferenceServerClient(url="localhost:8000")
# Prepare single dummy image tensor (shape: 1 x 3 x 224 x 224)
input_data = np.random.randn(1, 3, 224, 224).astype(np.float32)
infer_input = httpclient.InferInput("input", input_data.shape, "FP32")
infer_input.set_data_from_numpy(input_data)
infer_output = httpclient.InferRequestedOutput("output")
response = client.infer(
model_name="resnet50_onnx",
inputs=[infer_input],
outputs=[infer_output]
)
output_tensor = response.as_numpy("output")
print(f"Inference completed. Response tensor shape: {output_tensor.shape}")The server receives the single [1, 3, 224, 224] tensor and returns class probability logits for all 1,000 ImageNet categories.
3. Monitor Performance with Prometheus Metrics
Triton exposes Prometheus metrics out-of-the-box on port 8002:
curl http://localhost:8002/metricsFeed these key telemetry points into Prometheus and Grafana dashboards:
nv_inference_request_duration_us: Tracks total request latency, subdivided into queue wait time and actual compute time. If queue time spikes while GPU load remains below 40%, lower yourmax_queue_delay_microseconds.nv_inference_request_success: Records successful inference request counts broken down by model name and version.nv_gpu_utilizationandnv_gpu_memory_used_bytes: Monitors live GPU core activity and VRAM consumption in real time.
Running Triton in Docker bridges the gap between prototype machine learning code and production infrastructure. With dynamic batching and instance groups configured, a single GPU can comfortably handle hundreds of concurrent requests without dropping frames or breaking latency budgets.
