Why LLM Apps Are Hard to Debug Without Observability
Deploy an LLM-powered app and everything looks fine at first. Requests come in, responses go out, logs show 200 OK. Then a user reports a wrong answer — and you’re staring at a wall of text output with no idea where things went sideways.
Traditional logging doesn’t cut it. You need to know: what prompt was actually sent? What did the retrieval step return? Did the LLM hallucinate, or did it receive bad context from the RAG pipeline? How long did each step take?
A RAG chatbot I shipped last year made this painfully clear. Users kept getting outdated answers, but the app logs showed nothing but successful completions. Hours of print-statement debugging later, the culprit was a vector search pulling stale embeddings from an index that hadn’t been refreshed in three months. Completely invisible from the outside. That incident made LLM observability a hard requirement on every AI project I build.
Arize Phoenix is an open-source observability platform built specifically for this problem. Think of it like Jaeger or Zipkin for distributed tracing — but designed for LLMs, RAG pipelines, and AI agents. One UI gives you full trace visibility: prompts, completions, retrieved documents, latency per step, token counts, and evaluation scores.
Installation
Phoenix ships as a single Docker image, so it drops cleanly into any existing stack. Create a docker-compose.yml:
version: "3.8"
services:
phoenix:
image: arizephoenix/phoenix:latest
container_name: arize-phoenix
ports:
- "6006:6006" # Phoenix Web UI
- "4317:4317" # OTLP gRPC endpoint
- "4318:4318" # OTLP HTTP endpoint
volumes:
- phoenix_data:/root/.phoenix
restart: unless-stopped
environment:
- PHOENIX_WORKING_DIR=/root/.phoenix
volumes:
phoenix_data:Start the container:
docker compose up -dVerify it came up cleanly:
docker compose ps
docker logs arize-phoenixOpen http://localhost:6006 in your browser. The Phoenix dashboard should appear with no projects yet. Running on a remote server? Swap localhost for your server IP and confirm port 6006 is open in your firewall or security group.
The phoenix_data named volume handles persistence automatically — traces survive container restarts without any extra configuration.
Configuration
Connecting your app to Phoenix takes three steps: install the SDK, register the tracer provider, and instrument your framework.
Install the Packages
# Core OTEL + Phoenix
pip install arize-phoenix-otel opentelemetry-sdk opentelemetry-exporter-otlp-proto-grpc
# Pick the instrumentation package for your framework
pip install openinference-instrumentation-openai # OpenAI
pip install openinference-instrumentation-langchain # LangChain
pip install openinference-instrumentation-llama-index # LlamaIndexRegister the Tracer and Instrument OpenAI
from phoenix.otel import register
from openinference.instrumentation.openai import OpenAIInstrumentor
# Register Phoenix as the trace collector
tracer_provider = register(
project_name="my-llm-app",
endpoint="http://localhost:4317", # OTLP gRPC
)
# Auto-instrument all OpenAI calls — no other code changes needed
OpenAIInstrumentor().instrument(tracer_provider=tracer_provider)LangChain works the same way — just swap the instrumentor:
from phoenix.otel import register
from openinference.instrumentation.langchain import LangChainInstrumentor
tracer_provider = register(
project_name="rag-pipeline",
endpoint="http://localhost:4317",
)
LangChainInstrumentor().instrument(tracer_provider=tracer_provider)That’s it. Run your app normally — every LLM call, retrieval step, and chain execution is traced automatically.
Connecting from Another Docker Container
When your app also runs in Docker, use the Phoenix container’s service name instead of localhost:
tracer_provider = register(
project_name="my-llm-app",
endpoint="http://arize-phoenix:4317", # Docker service name
)Prefer HTTP over gRPC? That works too — handy behind proxies that don’t pass gRPC through cleanly:
tracer_provider = register(
project_name="my-llm-app",
endpoint="http://arize-phoenix:4318/v1/traces",
protocol="http/protobuf",
)Verification & Monitoring
Fire a few test queries through your app, then open http://localhost:6006. A project entry should appear with your first traces.
Reading Traces in the UI
Click into the Traces tab and select any trace to expand it into a span tree. A typical RAG pipeline looks like this:
- Top-level span: the incoming user query
- Child span: embedding generation for the query
- Child span: vector search with the retrieved document chunks
- Child span: the actual LLM call — full prompt visible here, including the injected context
- Metadata on each span: latency, token count, model name, finish reason
This is exactly what I was missing when debugging that stale embeddings issue. With Phoenix running, the retrieved chunks showed timestamps from months in the past — right there in the trace. What had previously taken hours of guesswork took under five minutes to pinpoint.
Running RAG Quality Evaluations
Phoenix includes built-in evaluators for hallucination detection and retrieval relevance. Pull your traced spans as a dataframe and run evals against them:
from phoenix.client import Client
from phoenix.evals import (
HallucinationEvaluator,
RelevanceEvaluator,
run_evals,
)
from phoenix.evals import OpenAIModel
# Fetch traced spans
client = Client(endpoint="http://localhost:6006")
spans_df = client.get_spans_dataframe(project_name="rag-pipeline")
# Use a small model to keep eval costs low
eval_model = OpenAIModel(model="gpt-4o-mini")
results = run_evals(
dataframe=spans_df,
evaluators=[
HallucinationEvaluator(eval_model),
RelevanceEvaluator(eval_model),
],
provide_explanation=True,
)
print(results[0].head())Scores get written back to Phoenix and appear inline on each span. Filter the trace list by hallucination score, sort by worst relevance, and click straight into the exact prompt-context pair that caused the problem.
Monitoring AI Agents
Agentic workflows get the same treatment. Phoenix traces every tool call as a child span. An agent that runs four web searches, fetches one URL, and writes a final answer shows up as six spans in the tree — inputs, outputs, and timing on each. That visibility alone replaces most of the print-statement scaffolding you’d otherwise sprinkle around just to understand what the agent was doing.
Metrics to Watch Regularly
Once you have a week of data flowing in, these are the three signals worth checking consistently:
- P95 latency per project — sudden spikes usually point to a retrieval bottleneck or a model timeout upstream
- Token usage trends — catch prompt bloat before it shows up on your billing dashboard
- Hallucination rate over time — a climbing rate almost always means something changed in your retrieval pipeline or prompt template
Attaching Custom Metadata
Standard OTEL attributes work in Phoenix out of the box. Tagging spans with user IDs and session IDs cuts incident triage time significantly:
from opentelemetry import trace
tracer = trace.get_tracer(__name__)
with tracer.start_as_current_span("user-query") as span:
span.set_attribute("user.id", user_id)
span.set_attribute("session.id", session_id)
span.set_attribute("query.type", "rag")
# ... your LLM call hereThe full setup — container running to first traces — takes about 20 minutes. After that, debugging LLM issues drops from hours of guesswork to minutes of clicking through traces. Running AI in production without observability is flying blind. Add Phoenix before you ship the next feature.
