Real-World Problem: Silent Failures in LLM Features
Shipping an LLM feature usually feels triumphant—until real users start hitting edge cases. Traditional unit tests check predictable inputs against exact outputs. Send 2 + 2, expect 4. If a function throws an unhandled exception or dumps a 500 status, your test runner flags it within seconds.
Generative models strip away that safety net. An API call can return an HTTP 200 with clean, parseable JSON, yet deliver nonsense. The output might invent non-existent features or ignore what the user asked.
We hit this exact wall last quarter. Our team trimmed a system prompt to reduce token latency by 15%. All 240 of our standard Pytest unit tests passed. Two days later, an enterprise customer flagged that our documentation bot was hallucinating API endpoints that never existed in version 3.x. The code worked flawlessly; the language output failed silently.
Root Cause Analysis: Why Traditional Testing Misses LLM Regressions
Standard testing tools expect deterministic code. LLMs violate those assumptions in four fundamental ways:
- Non-deterministic output: Even set to
temperature=0.0, models can produce subtle wording variations across runs. Hardcoded string assertions likeassert response == expectedfail almost immediately. - Semantic nuances: Checking whether an answer is helpful or factually grounded requires semantic context. Simple substring searches or regex rules cannot evaluate factual consistency.
- Prompt and retrieval drift: Changing a prompt prefix, adjusting vector chunk sizes from 500 to 800 tokens, or swapping base models can inject hallucinations across dozens of queries. None of these changes trigger standard runtime tracebacks.
- The review bottleneck: Asking engineers to spot-check 50 completions in a spreadsheet adds hours to review cycles. It is slow, expensive, and inevitably misses rare edge cases.
Solutions Compared: Finding the Right Evaluation Strategy
Teams typically attempt three strategies to guard LLM quality:
1. Manual Review & Spreadsheets
Engineers export batches of prompt completions to Google Sheets, scoring each on a 1-to-5 scale. Human reviewers bring reliable intuition, but this setup cannot scale past a few dozen iterations. It is costly, introduces personal reviewer bias, and cannot block a broken pull request in a build pipeline.
2. Ad-Hoc LLM-as-a-Judge Scripts
Developers write quick helper scripts asking a model like gpt-4o to judge, “Is this answer relevant? Reply YES or NO.” While quick to sketch out, bespoke judge scripts rarely maintain consistent scoring thresholds across team members. Managing raw prompts for multiple evaluation criteria quickly becomes its own maintenance burden.
3. Dedicated Evaluation Frameworks (DeepEval)
DeepEval plugs production-grade evaluation metrics directly into Pytest. It gives you standardized metrics—like Hallucination, Answer Relevancy, Faithfulness, and Contextual Precision—complete with calibrated scoring thresholds. If a prompt tweak causes relevancy to drop below your designated threshold, the build fails just like an ordinary broken test.
Implementation: DeepEval Quality Gates in Practice
Let’s set up DeepEval, write targeted tests for relevancy and hallucination, and enforce these checks automatically on every pull request.
Step 1: Environment Setup
Create an isolated virtual environment and install DeepEval along with Pytest:
python3 -m venv venv
source venv/bin/activate
pip install deepeval==1.2.0 pytest==8.2.0
DeepEval uses an evaluation model (GPT-4o by default) to score output quality. Export your API credentials in your active shell:
export OPENAI_API_KEY="sk-proj-your-api-key"
Step 2: Testing Answer Relevancy
The AnswerRelevancyMetric scores whether a response answers the user’s explicit question without rambling or adding irrelevant claims.
Create tests/test_relevancy.py:
import pytest
from deepeval import assert_test
from deepeval.test_case import LLMTestCase
from deepeval.metrics import AnswerRelevancyMetric
def test_answer_relevancy():
user_input = "How do I reset my account password?"
model_output = (
"To reset your account password, click 'Forgot Password' on the login screen, "
"enter your registered email address, and follow the link delivered to your inbox."
)
# Score ranges from 0.0 to 1.0. 0.7 requires strong direct alignment.
metric = AnswerRelevancyMetric(threshold=0.7)
test_case = LLMTestCase(
input=user_input,
actual_output=model_output
)
assert_test(test_case, [metric])
Run the test through the DeepEval CLI runner:
deepeval test run tests/test_relevancy.py
DeepEval extracts statements from the completion, checks them against the query, and produces a score between 0 and 1. If the score misses the 0.7 bar, the test fails and prints the specific lines flagged as off-topic.
Step 3: Measuring and Catching Hallucinations
Hallucination testing compares your model’s answer against the source documents provided by your retrieval layer. HallucinationMetric checks whether the completion makes claims unsupported by those reference chunks.
Create tests/test_hallucination.py:
import pytest
from deepeval import assert_test
from deepeval.test_case import LLMTestCase
from deepeval.metrics import HallucinationMetric
def test_hallucination_guard():
retrieved_docs = [
"Our refund policy allows customers to request a full refund within 30 days of purchase.",
"Refunds are credited back to the original payment method within 5 to 7 business days."
]
user_input = "Can I get a cash refund in your physical store?"
# Simulated faulty completion introducing an unsubstantiated policy
faulty_output = (
"You can get a full refund within 30 days. "
"We also provide cash refunds at any retail store location."
)
# For HallucinationMetric, threshold is the MAXIMUM acceptable hallucination score.
# 0.0 demands zero ungrounded statements.
hallucination_metric = HallucinationMetric(threshold=0.1)
test_case = LLMTestCase(
input=user_input,
actual_output=faulty_output,
context=retrieved_docs
)
assert_test(test_case, [hallucination_metric])
Execute the test suite:
deepeval test run tests/test_hallucination.py
Because the completion fabricated an in-store cash refund policy not present in the context, DeepEval assigns a high contradiction score (typically ~0.50). This easily breaches the 0.1 threshold and causes the assertion to fail with a clear diagnostic log.
Step 4: Integrating DeepEval into GitHub Actions CI/CD
Manual testing during development is helpful, but continuous automation keeps quality steady. Add .github/workflows/llm_eval.yml to run these checks on every PR targeting main:
name: LLM Evaluation CI
on:
pull_request:
branches:
- main
jobs:
evaluate-llm:
runs-on: ubuntu-latest
steps:
- name: Check out repository
uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.11'
cache: 'pip'
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -r requirements.txt
pip install deepeval pytest
- name: Run DeepEval Quality Gates
env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
run: |
deepeval test run tests/ --json-report
Once this workflow runs, prompt edits are verified automatically alongside regular application code. If a teammate alters prompt instructions or swaps chunking parameters in a way that introduces hallucinations, GitHub Actions fails the check. Broken outputs get caught in code review—not in front of your customers.

