AI Evaluation & Testing Playbook

Unlike traditional software, Large Language Model (LLM) applications are non-deterministic, making them prone to regressions, hallucinations, and format drift. Building a production-grade AI system requires a robust evaluation framework to measure output quality, identify failures, and gate releases.

📐 1. Evaluation Architecture Overview

Evaluating LLM applications occurs across three primary environments: development (offline), continuous integration (CI/CD), and runtime production (online).

• Offline vs. Online Evaluation:
  • Offline (Pre-production): Running evaluations on static test suites (“Golden Datasets”) before deployment. This evaluates system changes (prompts, chunking, models) under controlled conditions.
  • Online (Production): Sampling live user interactions to calculate quality metrics, detect semantic drift, and catch hallucinations at runtime.
• Human Evaluation vs. LLM-as-a-Judge:
  • Human Evaluation: High-accuracy gold standard, but slow, expensive, and cannot scale for continuous integration.
  • LLM-as-a-Judge: Automated evaluation where frontier models (e.g., GPT-4o) act as judges by grading model outputs against structured rubrics.
• Continuous Evaluation Pipelines: Automating test executions on every code commit. Pull requests that lower safety or accuracy scores below predefined thresholds are automatically blocked.

Complete Evaluation Workflow

The diagram below details the continuous request-to-evaluation pipeline:

📊 2. Core Metrics

To isolate issues in either retrieval or generation, production pipelines measure specific semantic dimensions:

• Faithfulness (Groundedness): Measures if the generated output contains only claims supported by the retrieved context. This directly tracks hallucinations. \[ \text{Faithfulness} = \frac{\text{Number of Factual Claims Grounded in Context}}{\text{Total Claims in Generated Output}} \]
• Answer Relevancy: Measures how directly the output addresses the user’s input query. It penalizes redundant or off-topic generation.
• Context Precision: Evaluates the quality of retrieval by checking if the most relevant document chunks are ranked at the top of the context block.
• Context Recall: Measures if the retrieved chunks contain all the necessary information to synthesize the expected answer.
• Hallucination Rate: Percentage of outputs containing ungrounded statements or logic errors.
• Toxicity & Bias: Detects offensive language, stereotyping, or security leaks (e.g., PII leakage) before output delivery.

🔍 3. RAG Evaluation & RAGAS Framework

The RAGAS (Retrieval Augmented Generation Assessment) framework evaluates RAG systems by separating retrieval quality from generation quality using the RAG Triad:

• Retrieval Validation (Query ➔ Context): Evaluated via Context Recall and Context Precision. If these scores are low, engineers should optimize chunk size, overlapping strategies, metadata filters, or introduce re-ranking models.
• Generation Validation (Context ➔ Response): Evaluated via Faithfulness and Answer Relevancy. If these scores are low, the prompt needs better instruction alignment, output formats need schema enforcement, or the LLM is underperforming on reasoning tasks.

🤖 4. LLM-as-a-Judge

LLM-as-a-judge leverages advanced LLMs to evaluate candidate outputs. To achieve consistent grading:

1. G-Eval Methodology: A framework that generates custom evaluation steps from a natural language rubric. The judge model produces reasoning steps before outputting a final score (e.g., 1–5), which can be weighted by token probability to improve consistency.
2. Pairwise Comparison: The judge model compares two candidate completions (Model A vs. Model B) side-by-side. To prevent biases:
   • Position Bias: Swap the order of Model A and Model B in secondary evaluation cycles.
   • Verbosity Bias: Instruct the judge to ignore completion length and evaluate strictly based on correctness.
3. Rubric-Based Grading: Defining clear criteria for each score level (e.g., “Score 3 means the code is functionally correct but lacks comments; Score 5 means the code is correct, commented, and optimal”).

💻 5. DeepEval Integration

The DeepEval framework allows writing evaluations as standard Python unit tests, facilitating automation inside CI/CD pipelines.

Python Evaluation Test Case (test_evals.py)

import pytest
from deepeval import assert_test
from deepeval.test_case import LLMTestCase
from deepeval.metrics import HallucinationMetric, AnswerRelevancyMetric
 
def test_production_faq_routing():
    # 1. Instantiate the test payload
    test_case = LLMTestCase(
        input="How do I cancel my subscription?",
        actual_output="You can cancel your subscription inside the billing dashboard by clicking cancel subscription.",
        retrieval_context=[
            "To cancel your subscription, navigate to settings, open the billing dashboard, and click cancel subscription.",
            "Refunds are processed within 5-10 business days."
        ]
    )
 
    # 2. Configure metrics with target thresholds
    hallucination_metric = HallucinationMetric(threshold=0.3)
    relevancy_metric = AnswerRelevancyMetric(threshold=0.8)
 
    # 3. Assert test passes metrics requirements
    assert_test(test_case, [hallucination_metric, relevancy_metric])

GitHub Actions Workflow Config (.github/workflows/deepeval_ci.yml)

name: LLM Evaluation Run
 
on:
  pull_request:
    branches: [ main ]
 
jobs:
  run-evals:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Code
        uses: actions/checkout@v4
 
      - name: Setup Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.10'
 
      - name: Install Libraries
        run: |
          pip install pytest deepeval openai
 
      - name: Execute DeepEval Suite
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: |
          deepeval test run test_evals.py

🏛️ 6. The Evaluation Pyramid

Like traditional testing, AI validation employs a tiered pyramid testing model to balance cost, execution speed, and verification depth:

                  / \
                 /   \
                / Human\
               / Review \
              /----------\
             / LLM-as-a-  \
            /    Judge     \
           /----------------\
          /  Golden Dataset  \
         /     Benchmarks     \
        /----------------------\
       /    Unit / Integration  \
      /          Tests           \
     /____________________________\

1. Unit Tests (Base): Fast, programmatic validations checking JSON schema conformity, input bounds, or blocked keyword lists. Cheap and runs on every commit.
2. Golden Dataset Benchmarks: Evaluating changes against a representative test suite (50–500 cases) to catch regressions before branch merges.
3. LLM-as-a-Judge: Evaluating semantic quality (faithfulness, relevancy) at scale. Run weekly or on major model updates.
4. Human Review (Apex): Manual checking of failures or low-confidence boundary scores. Used to refine golden datasets.

🚨 7. Common Failure Modes & Mitigations

• Hallucinations: Model introduces facts not present in the reference context.
  • Mitigation: Apply strict guided decoding (JSON schemas), lower model temperature to 0.0, and add explicit “If the information is missing, say I don’t know” instructions.
• Context Misses: The relevant document is missing from the database or filtered out.
  • Mitigation: Implement hybrid search (combining sparse keyword TF-IDF/BM25 with dense vector embeddings) and increase chunk retrieval limits.
• Retrieval Failures (Noisy Chunks): Chunks containing irrelevant text dilute context.
  • Mitigation: Use cross-encoder re-ranking models (e.g., Cohere Rerank) to sort chunks, dropping chunks below a relevance threshold.
• Judge Model Bias: The LLM judge favors its own completions or bases scores on length (verbosity bias).
  • Mitigation: Shuffle position alignments, strip model identifiers, and enforce strict, numeric grading rubrics.
• Dataset Leakage: Evaluated context is trained into the base LLM, inflating accuracy metrics.
  • Mitigation: Continually generate synthetic test scenarios (Golden Datasets) using independent generators, and rotate test questions frequently.

🚦 8. Production Score Thresholds

To gate code releases, establish baseline validation metrics:

🎯 9. Evaluation Strategy by Application Type

RAG Systems

• Metrics Focus: Context Recall, Context Precision, Faithfulness.
• Testing Method: Running queries against a static knowledge base to check retrieval rankings.

AI Agents (Autonomous Loops)

• Metrics Focus: Tool Call Accuracy, State Sequence Validity, Goal Achievement.
• Testing Method: Mocking tool outputs (APIs, databases) to verify loop termination and retry strategies.

Chatbots (Conversational)

• Metrics Focus: Coherence, Tone/Style alignment, Conversational State Tracking.
• Testing Method: Running simulated multi-turn conversations using user personas.

Structured Extraction

• Metrics Focus: Schema Compliance, Field Extraction Accuracy, Data Completeness.
• Testing Method: Automated programmatic verification (Pydantic model validation) on generated outputs.

Code Generation

• Metrics Focus: Syntax correctness, Execution success, Test case coverage.
• Testing Method: Executing generated code inside isolated sandbox containers and running verification test suites.

🛠️ Platform Comparison Matrix