The Incident That Made Me Build This
About a year ago, our checkout service started failing intermittently in production. Response times spiked from 200ms to 8 seconds — but only for roughly 3% of requests. The logs showed errors, but they were scattered across six different services: API gateway, auth service, product catalog, cart, inventory, and payment processor.
We spent four hours correlating log lines by timestamp, trying to piece together what happened. Eventually we found it — a database connection pool exhaustion in the inventory service that cascaded upstream. The fix took 5 minutes. The diagnosis took half a workday.
That’s when I started taking distributed tracing seriously. Not as a nice-to-have, but as a necessity for any system with more than two services talking to each other.
Why Logs Alone Don’t Work for Microservices
The fundamental problem with logs in a microservice architecture is that they’re isolated. Each service writes its own logs with its own context. When a user request touches seven services, you get seven separate log streams with no native way to stitch them together.
You end up doing mental gymnastics: matching timestamps, copying request IDs across terminal windows, hoping someone actually propagated that trace ID header. Most of the time, someone didn’t.
Distributed tracing solves this by giving every request a single trace ID that follows it through every service. Each service adds a span — a timed unit of work — and they all connect into one tree showing exactly what happened, in what order, and how long each step took. This is the visibility layer that logs simply can’t provide on their own.
Comparing Your Options: Jaeger, Zipkin, and Grafana Tempo
Before settling on Grafana Tempo, I evaluated the common backends to understand the trade-offs.
Jaeger and Zipkin
Both are mature, well-documented projects. Jaeger is a CNCF graduated project with strong ecosystem support. The problem is storage: by default they use Elasticsearch or Cassandra as backends, which means significant operational overhead and cost. For high-volume services, you’ll find yourself managing a heavy database cluster just to store trace data that you only look at when something breaks.
Zipkin is simpler but has less active development and a smaller ecosystem. For a team that doesn’t already run Elasticsearch, the operational cost of Jaeger can outweigh its benefits.
Grafana Tempo
Tempo takes a fundamentally different approach: it stores traces as flat files in object storage — S3, GCS, Azure Blob, or local disk — rather than maintaining a search index. This makes it dramatically cheaper. Object storage typically costs around 20x less than block storage for the same data volume.
The trade-off worth understanding: Tempo doesn’t support searching traces by service name or arbitrary attributes on its own. It relies on correlating trace IDs found in your logs (via Loki) or metrics (via Prometheus). If you’re already in the Grafana ecosystem, this fits naturally. If you’re not, factor in the integration work.
For most teams building on the Grafana stack, Tempo is the clear choice for keeping costs manageable at scale.
The Real Cost Problem: You Can’t Store Everything
Here’s something that doesn’t get discussed enough when teams first set up tracing: a busy service can generate millions of spans per minute. At that volume, storing every single trace becomes expensive fast, even with cheap object storage.
The solution is sampling — only recording a fraction of your traces. But there are two fundamentally different strategies, and choosing the wrong one undermines the whole point.
Head-Based Sampling
With head-based sampling, the decision to record a trace happens at the start of the request, before any data is collected. A common setup is “sample 10% of requests randomly.” It’s simple and has zero overhead.
The problem: you’re sampling blindly. That one slow request you actually needed to debug? There’s a 90% chance you didn’t capture it. Head-based sampling is easy to implement but terrible at preserving the traces that actually matter.
Tail-Based Sampling
Tail-based sampling makes the decision after the request completes. The OpenTelemetry Collector buffers all spans, waits for the full trace, then decides what to keep based on the outcome: keep all errors, keep all slow requests over 2 seconds, keep 5% of everything else.
I’ve applied this approach in production and the results have been consistently stable — storage costs dropped significantly while we retained 100% of the traces that actually mattered for debugging. The random 5% kept enough data for performance trending without ballooning costs.
Setting Up Grafana Tempo with OpenTelemetry
Here’s a working setup using Docker Compose. This mirrors how you’d deploy to a VPS or adapt for Kubernetes.
Step 1: Docker Compose for the Stack
Create a docker-compose.yml:
version: '3.8'
services:
tempo:
image: grafana/tempo:latest
command: [ "-config.file=/etc/tempo.yaml" ]
volumes:
- ./tempo.yaml:/etc/tempo.yaml
- tempo-data:/var/tempo
ports:
- "3200:3200"
otel-collector:
image: otel/opentelemetry-collector-contrib:latest
command: [ "--config=/etc/otel-collector.yaml" ]
volumes:
- ./otel-collector.yaml:/etc/otel-collector.yaml
ports:
- "4317:4317" # OTLP gRPC
- "4318:4318" # OTLP HTTP
depends_on:
- tempo
grafana:
image: grafana/grafana:latest
ports:
- "3000:3000"
environment:
- GF_AUTH_ANONYMOUS_ENABLED=true
- GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
volumes:
- ./grafana-datasources.yaml:/etc/grafana/provisioning/datasources/datasources.yaml
volumes:
tempo-data:
Step 2: Tempo Configuration
Create tempo.yaml:
server:
http_listen_port: 3200
distributor:
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
storage:
trace:
backend: local
local:
path: /var/tempo/blocks
wal:
path: /var/tempo/wal
For production, swap backend: local with backend: s3 and add your S3 bucket configuration. Local disk works fine for development and low-traffic environments.
Step 3: OpenTelemetry Collector with Tail Sampling
This is where the cost control happens. Create otel-collector.yaml:
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
processors:
tail_sampling:
decision_wait: 10s # Wait 10s for complete trace before deciding
num_traces: 100000 # Buffer up to 100k pending traces
policies:
- name: errors-policy
type: status_code
status_code: {status_codes: [ERROR]}
- name: slow-traces-policy
type: latency
latency: {threshold_ms: 2000}
- name: probabilistic-policy
type: probabilistic
probabilistic: {sampling_percentage: 5}
exporters:
otlp:
endpoint: tempo:4317
tls:
insecure: true
service:
pipelines:
traces:
receivers: [otlp]
processors: [tail_sampling]
exporters: [otlp]
This configuration keeps 100% of error traces, 100% of requests slower than 2 seconds, and 5% of everything else. Adjust the percentage based on your traffic volume and storage budget — for very high traffic (10k+ req/min), even 1% of normal traces gives you plenty of data.
Step 4: Instrumenting a Python Application
Install the OpenTelemetry packages:
pip install opentelemetry-distro opentelemetry-exporter-otlp
opentelemetry-bootstrap -a install
Initialize tracing in your application startup:
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
provider = TracerProvider()
exporter = OTLPSpanExporter(endpoint="http://localhost:4317", insecure=True)
provider.add_span_processor(BatchSpanProcessor(exporter))
trace.set_tracer_provider(provider)
tracer = trace.get_tracer(__name__)
# Add custom spans where you need visibility
def process_order(order_id: str):
with tracer.start_as_current_span("process-order") as span:
span.set_attribute("order.id", order_id)
result = run_inventory_check(order_id)
span.set_attribute("inventory.available", result)
return result
The trace ID propagates automatically between services via HTTP headers (W3C TraceContext format) when you use the OpenTelemetry HTTP instrumentation library. For FastAPI specifically, add FastAPIInstrumentor().instrument_app(app) after app initialization and you get automatic span creation for every endpoint with zero manual code.
Step 5: Grafana Data Source
Create grafana-datasources.yaml:
apiVersion: 1
datasources:
- name: Tempo
type: tempo
url: http://tempo:3200
isDefault: true
Verifying the Setup
Start the stack with docker-compose up -d, send requests through your instrumented service, then open Grafana at http://localhost:3000. Navigate to Explore → select the Tempo data source → browse recent traces or search by trace ID.
You’ll see a waterfall view showing each span across your services, with timing information side by side. If a request was slow, you’ll immediately see which service was the bottleneck and what data it was processing.
One thing to calibrate: the decision_wait value in the tail sampler needs to be longer than your slowest possible traces. If a trace takes 15 seconds end-to-end but your decision window is 10 seconds, the sampler makes an incomplete decision and may drop important spans. For most HTTP APIs, 10–30 seconds is a safe window. For batch processing jobs, you may need longer.
What to Expect at Production Scale
After deploying this setup across a four-service Python application handling roughly 500 requests per minute, storage stayed under 2GB per day. Without sampling, the same workload would have generated around 40GB of trace data daily. The 5% probabilistic policy on normal requests gave us more than enough for performance trending and capacity planning, while the error and latency policies ensured we never missed an incident worth investigating.
The workflow shift is significant. Instead of log archaeology during incidents, engineers open Grafana, pull up the trace ID from a customer report or an alert, and within 30 seconds they know exactly where the slowdown happened and what data was involved at each step.
Start with one service, get comfortable reading the waterfall view, then expand instrumentation service by service. Instrument your database calls and external HTTP clients first — those are where latency surprises almost always hide.
