Shipping a Retrieval-Augmented Generation (RAG) system to production is just the start. The real headaches emerge a week later. Users begin flagging subtle hallucinations, or engineers notice that answers quietly contradict your runbooks. Catching these regressions manually is impossible once you handle 5,000 queries a day.
You need quantitative guardrails. Ragas (Retrieval Augmented Generation Assessment) solves this problem by using an LLM-as-a-judge framework. It grades retrieval quality and generation accuracy automatically, saving your team from spending dozens of hours manually reviewing log outputs.
Quick Start (5 Min)
You can benchmark your baseline in under five minutes. All you need is Python 3.10+, an OpenAI API key (or any OpenAI-compatible endpoint), and a handful of sample interactions: questions, retrieved context passages, and generated answers.
First, install the core dependencies:
pip install ragas langchain-openai datasetsNext, assemble a minimal evaluation script. Ragas expects a dataset containing queries, retrieved document passages, generated answers, and ground-truth references:
import os
from datasets import Dataset
from ragas import evaluate
from ragas.metrics import faithfulness, context_precision
os.environ["OPENAI_API_KEY"] = "your-openai-api-key"
# Sample logs captured from your RAG service
eval_data = {
"question": [
"Where is the cluster backup configuration stored?",
"What is the timeout threshold for internal service mesh calls?"
],
"contexts": [
["All backup schedules and storage targets are configured under /etc/backup/cluster-spec.yaml."],
["HTTP requests through Envoy sidecars use a default connection timeout of 5 seconds."]
],
"answer": [
"The cluster backup configuration is located in /etc/backup/cluster-spec.yaml.",
"The timeout threshold is 15 seconds across all mesh sidecars."
],
"ground_truth": [
"/etc/backup/cluster-spec.yaml",
"5 seconds"
]
}
dataset = Dataset.from_dict(eval_data)
# Run evaluation for Faithfulness and Context Precision
results = evaluate(
dataset=dataset,
metrics=[faithfulness, context_precision]
)
print(results.to_pandas()[["question", "faithfulness", "context_precision"]])Run this script, and Ragas parses every answer against its retrieved context before returning scores from 0.0 to 1.0. Look at row two in the output. The faithfulness score plunges straight to 0.0 because the model hallucinated 15 seconds when the context clearly stated 5 seconds.
Deep Dive: Understanding Faithfulness and Context Precision
Every RAG pipeline splits work between two components: the retriever and the generator. When bad answers slip through, either piece could be the culprit. Evaluating them in isolation prevents wild goose chases.
1. Faithfulness (Generator Quality)
Faithfulness measures factual adherence: did the LLM stick to your documents, or did it pull phantom facts from its pre-training memory? Low faithfulness directly exposes hallucinations.
Ragas scores this through a clean two-step process:
- Claim Extraction: The judge LLM breaks the generated answer into isolated, verifiable assertions.
- Entailment Verification: The judge checks each statement against the retrieved passages to see if the text directly supports it.
The calculation is straightforward:
Faithfulness = (Number of supported claims) / (Total number of generated claims)Suppose your assistant outputs three technical claims. If your passages confirm two of them while the third is unsubstantiated, your faithfulness score lands at exactly 0.67.
2. Context Precision and Relevance (Retriever Quality)
Context Precision gauges signal-to-noise ratio: did your retriever surface relevant snippets near the top, or did it bury them under irrelevant junk?
Bloated vector results trigger two common operational problems:
- Token costs climb fast. Stuffing 8,000 irrelevant tokens into every prompt burns budget quickly.
- LLMs lose focus. Models frequently misplace critical facts when relevant details sit sandwiched between long, irrelevant paragraphs.
Ragas evaluates chunks by checking whether sentences in each passage directly inform the question. If your retriever pulls 1,200 words of network documentation but only two lines contain the target CIDR block, context precision drops. Adjusting chunk sizes, tuning hybrid search weights, or adding a cross-encoder reranker will immediately push this number back up.
Advanced Usage: Custom Judges and CI/CD Integration
Default cloud endpoints work fine for quick local spikes. However, enterprise compliance rules or evaluation batches with over 10,000 queries usually call for internal models. You can route evaluation tasks to private infrastructure using standard LangChain client wrappers.
Switching to a Self-Hosted Judge Model
Here is how to configure a local vLLM or LiteLLM gateway hosting an open-weights model like Llama 3.1 70B:
from langchain_openai import ChatOpenAI
from ragas.llms import LangchainLLMWrapper
from ragas.metrics import Faithfulness, ContextPrecision
from ragas import evaluate
# Connect to an internal vLLM or LiteLLM instance
custom_llm = ChatOpenAI(
model="meta-llama/Meta-Llama-3.1-70B-Instruct",
base_url="http://internal-vllm.infra.local:8000/v1",
api_key="dummy-token"
)
judge_llm = LangchainLLMWrapper(custom_llm)
# Bind the custom judge directly to each metric
faithfulness_metric = Faithfulness(llm=judge_llm)
precision_metric = ContextPrecision(llm=judge_llm)
results = evaluate(
dataset=dataset,
metrics=[faithfulness_metric, precision_metric]
)Automating Score Gates in CI Pipelines
Hard metrics become invaluable when refactoring retrieval code. When we shifted our ingestion service from rigid 500-token chunks to semantic chunking, our automated test suite flagged an immediate 18% regression in context precision. We caught and fixed the regression long before deploying to staging.
You can run this evaluation script inside a GitHub Actions workflow to block regressions automatically:
# eval_gate.py
import sys
from datasets import load_from_disk
from ragas import evaluate
from ragas.metrics import faithfulness, context_precision
test_dataset = load_from_disk("./tests/fixtures/rag_golden_dataset")
results = evaluate(test_dataset, metrics=[faithfulness, context_precision])
avg_faithfulness = results["faithfulness"]
avg_precision = results["context_precision"]
print(f"Faithfulness: {avg_faithfulness:.2f} | Context Precision: {avg_precision:.2f}")
# Enforce quality thresholds for deployment
if avg_faithfulness < 0.85 or avg_precision < 0.75:
print("Quality check failed: Scores fell below production threshold.")
sys.exit(1)
print("Quality check passed. Proceeding with deployment.")
sys.exit(0)Field Notes for Running Ragas at Scale
Automating RAG benchmarks across large test sets comes with practical traps. Keep these operational practices in mind:
- Curate a 50-to-100 question golden set: Skip synthetic happy paths. Include multi-hop queries, colloquial phrasing, and out-of-scope prompts where the model must refuse to answer.
- Cache generated model responses: Ragas triggers multiple evaluation calls per row. If you are only benchmarking retriever changes, load cached generator responses to avoid unnecessary API billing.
- Avoid small judge models: Models under 13B parameters consistently struggle with logical entailment tasks. Stick to models with at least 70B parameters or proven 8x7B MoE architectures to keep scores reproducible.
- Diagnose root causes before tweaking prompts: High context precision paired with low faithfulness points to poor system prompts. If context precision is low, stop rewriting prompts; tune your chunking strategy, top-k retrieval depth, or embedding index instead.
