Dev.to
Building Production-Grade LLM Evaluation Pipelines: From Vibes to Metrics
Building Production-Grade LLM Evaluation Pipelines: From Vibes to Metrics
How we replaced "looks good to me" with automated evaluation catching 92% of hallucinations before deployment
The Problem: Why "Vibe Checks" Fail in Production
Three months ago, our team shipped a RAG-based customer support assistant. It worked great in testing — we'd ask it questions, read the answers, and say "yeah, that looks right."
Then it hit production.
A customer asked about their billing cycle. The assistant confidently cited a policy that didn't exist. Another asked about API rate limits and got numbers from a competitor's documentation. By the time we caught it, 500+ users had seen hallucinated responses.
The post-mortem was brutal: we had zero automated evaluation. Our test process was literally "ask 5 questions, read answers, thumbs up."
What Production Evaluation Actually Needs
Academic benchmarks (MMLU, HellaSwag) don't tell you if your system works for your use case. Production evaluation needs:
Domain-specific judges — Your criteria, not generic "helpfulness"
Speed — Evaluation must run in CI/CD, not overnight
Regression detection — Know immediately when a prompt change breaks things
CI/CD integration — Block merges that degrade quality
Golden dataset management — Versioned, stratified, growing test cases
Architecture: The Evaluation Pipeline
┌─────────────┐ ┌──────────────┐ ┌────────────────────┐ ┌──────────────┐
│ Test Cases │────▶│ LLM Under │────▶│ Judge Ensemble │────▶│ Metrics & │
│ (Golden Set)│ │ Test │ │ - Faithfulness │ │ Regression │
└─────────────┘ └──────────────┘ │ - Instruction F. │ │ Detection │
│ - JSON Schema │ └──────┬───────┘
│ - Custom LLM │ ▼
└────────────────────┘ ┌──────────────┐
│ Dashboard/ │
│ PR Comments │
└──────────────┘
Core Abstractions
# eval/base.py
@dataclass(frozen=True)
class TestCase:
id: str
input: dict[str, Any]
expected: dict[str, Any] | None = None
tags: list[str] = field(default_factory=list) # ["edge-case", "long-context"]
@dataclass(frozen=True)
class EvaluationResult:
test_case_id: str
judge_name: str
score: float
passed: bool
reasoning: str
class Judge(ABC):
@abstractmethod
async def evaluate(self, test_case: TestCase, response: Any) -> EvaluationResult: ...
class EvaluationHarness:
def __init__(self, judges: list[Judge]):
self.judges = judges
async def evaluate_all(self, test_cases, generate_fn, concurrency=10):
# Runs all cases through all judges with controlled concurrency
...
The Judge Ensemble: Beyond RAGAS
RAGAS gives you faithfulness and answer relevance. But production needs more:
Judge
Purpose
Type
Threshold
Faithfulness
Answer contradicts retrieved context?
LLM
0.8
Instruction Following
All prompt constraints satisfied?
LLM
0.9
JSON Schema
Valid structured output?
Deterministic
1.0
Safety
PII, harmful content, policy violations
LLM
1.0
Domain Expert
Medical/legal/financial accuracy
LLM (few-shot)
0.85
Custom LLM Judge with Few-Shot
# eval/judges.py
class LLMJudge(Judge):
def __init__(self, name, criteria, model="gpt-4o-mini", few_shot_examples=None):
self.name = name
self.criteria = criteria
self.model = model
self.few_shot = few_shot_examples or []
async def evaluate(self, test_case, response):
client = instructor.from_openai(AsyncOpenAI())
class Output(BaseModel):
score: float = Field(ge=0, le=1)
reasoning: str
passed: bool
result = await client.chat.completions.create(
model=self.model,
response_model=Output,
messages=[
{"role": "system", "content": self._system_prompt()},
*self._few_shot_messages(),
{"role": "user", "content": self._build_prompt(test_case, response)},
],
temperature=0.0,
)
return EvaluationResult(...)
Faithfulness Judge (Production-Ready)
def create_faithfulness_judge() -> LLMJudge:
return LLMJudge(
name="faithfulness",
criteria="""
Evaluate whether the ANSWER is faithful to the CONTEXT.
- Score 1.0: All claims in answer are directly supported by context
- Score 0.5: Some claims unsupported but not contradictory
- Score 0.0: Answer contains claims directly contradicted by context
""",
threshold=0.8,
few_shot_examples=[
{
"input": {
"context": "Company founded in 2019. Revenue $10M in 2023.",
"answer": "The company was founded in 2019 and reached $10M revenue in 2023."
},
"output": {"score": 1.0, "reasoning": "All claims supported", "passed": True}
},
{
"input": {
"context": "Product launched in January 2024.",
"answer": "The product launched in March 2024 after extensive beta testing."
},
"output": {"score": 0.0, "reasoning": "Contradicts launch date", "passed": False}
},
]
)
Golden Dataset Strategy
Don't start with 1000 cases. Start with 50 real production cases.
# eval/golden_set.jsonl
{"id": "support-001", "input": {"question": "How do I reset my password?", "context": "..."}, "expected": {"answer": "Use the 'Forgot Password' link..."}, "tags": ["basic", "auth"]}
{"id": "support-042", "input": {"question": "Why was I charged twice?", "context": "..."}, "expected": null, "tags": ["billing", "edge-case"]}
Stratification matters:
40% basic/happy-path
30% edge cases (ambiguous, multi-step)
20% adversarial (injection, off-topic)
10% multilingual/long-context
Version your dataset: Git-track it. Every production failure becomes a new test case.
Regression Detection That Works
def regression_report(self, baseline: dict[str, float]) -> dict[str, Any]:
current = self.summary()
report = {}
for judge_name, metrics in current.items():
if judge_name not in baseline:
continue
baseline_mean = baseline[judge_name]
current_mean = metrics["mean_score"]
diff = current_mean - baseline_mean
# Statistical test (simplified - use proper stats in prod)
report[judge_name] = {
"baseline": baseline_mean,
"current": current_mean,
"delta": diff,
"regressed": diff < -0.05, # 5% drop = regression
"improved": diff > 0.02,
}
return report
CI/CD Integration: GitHub Actions
# .github/workflows/llm-eval.yml
name: LLM Evaluation
on:
pull_request:
paths: ['prompts/**', 'eval/**']
schedule: ['0 2 * * *'] # Nightly
jobs:
evaluate:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with: {python-version: '3.11'}
- name: Install deps
run: pip install -e .[dev]
- name: Run evaluation
env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
run: |
python -m eval.run_suite --config config.yaml --output results.json
- name: Check regressions
run: |
python -m eval.check_regression --baseline baseline.json --current results.json
- name: Comment PR
if: github.event_name == 'pull_request'
uses: actions/github-script@v7
with:
script: |
const results = JSON.parse(fs.readFileSync('results.json'));
const body = `## LLM Evaluation Results
| Judge | Pass Rate | Mean Score |
|-------|-----------|------------|
${Object.entries(results.summary).map(([k,v]) =>
`| ${k} | ${(v.pass_rate*100).toFixed(1)}% | ${v.mean_score.toFixed(3)} |`).join('\n')}
`;
github.rest.issues.createComment({
issue_number: context.issue.number,
owner: context.repo.owner,
repo: context.repo.repo,
body
});
Results: 6 Months of Production Evaluation
Metric
Before
After
Change
Hallucination catch rate
~67% (human)
92% (auto)
+25%
Prompt iteration cycle
2 hours
15 minutes
8x faster
Production incidents
3/month
0.2/month
15x reduction
Regression detection
Manual (days)
Automated (minutes)
—
Open Source Tooling We Built
All MIT licensed, production-hardened:
llm-eval-harness — Core framework (this article's code)
prompt-registry — Git-based prompt versioning with eval history
eval-dashboard — Real-time monitoring with alerting
Getting Started Today (5 Minutes)
pip install llm-eval-harness
from eval.harness import EvaluationHarness
from eval.judges import create_faithfulness_judge, create_instruction_following_judge
from eval.base import TestCase
# 1. Define 10 real test cases from your logs
cases = [
TestCase(id="1", input={"question": "..."}, expected={"answer": "..."}, tags=["basic"]),
# ...
]
# 2. Build judge ensemble
judges = [
create_faithfulness_judge(),
create_instruction_following_judge(),
# Add your domain-specific judges
]
# 3. Run
harness = EvaluationHarness(judges)
results = await harness.evaluate_all(cases, your_llm_function)
print(harness.summary())
# 4. Save baseline
harness.save_baseline("baseline.json")
# 5. Add to CI — done
The Mental Shift
Evaluation is infrastructure, not afterthought.
Treat judges as first-class code (versioned, tested, reviewed)
Golden dataset = your most valuable IP (curate it religiously)
Every prompt change = eval run (enforced by CI)
Regression alerts = paging alerts (not email digests)
Your users don't care about your prompt engineering cleverness. They care that the answer is right. Automated evaluation is how you guarantee that at scale.
Code: github.com/yourname/llm-eval-harness |
Discussion: Hacker News |
Follow: @yourname
2 hours ago
