# Week 7: Evaluation and LLM-as-a-judge

Applied AI Course — Sawan Ruparel

Public self-study edition. See https://applied-ai.sawanruparel.com/updates/ for version notes.

# Evaluation & LLM-as-a-Judge

## Measuring What Matters

Week 7 | Applied AI Course | Spring 2026

---

> "If you can't measure it, you can't improve it." -- adapted from Lord Kelvin

Your RAG pipeline is only as good as your ability to prove it works.

---

# Why Evaluation Matters

## The "Vibes-Based Testing" Problem

- You build a RAG pipeline, try 5 queries, it "looks good" -- ship it
- Two weeks later: hallucinations in production, angry users, broken trust
- Manual spot-checking doesn't scale beyond a handful of examples

## What Goes Wrong Without Systematic Eval

- **Silent regressions**: a prompt tweak improves one case, breaks ten others
- **Inconsistent quality**: works for common queries, fails on edge cases
- **No baseline**: you can't tell if v2 is actually better than v1
- **False confidence**: cherry-picked examples create an illusion of quality

## The Cost of Getting It Wrong

| Failure Mode | Impact |
|---|---|
| Hallucinated facts | Legal liability, misinformation |
| Missed relevant context | Incomplete or wrong answers |
| Prompt injection bypass | Security vulnerabilities |
| Degraded latency | User abandonment |

## The Goal

Move from "it seems to work" to "we have quantitative evidence that it works, and we'll know immediately when it stops working."

---

# The Evaluation Landscape

## Benchmarks

Static test suites with known answers (MMLU, HumanEval, TruthfulQA). Good for comparing base models. Limited for production app evaluation -- your domain isn't on the benchmark. MMLU is now effectively **saturated** (88%+ from frontier models); the community has shifted to harder live-difficulty signals like **GPQA Diamond** (94.3% by Gemini 3.1 Pro) and **ARC-AGI-2** (85% by GPT-5.5).

## Human Evaluation

Gold standard for subjective quality. Expensive ($5-15 per judgment), slow (days-weeks), hard to reproduce. Essential for calibration but not for CI/CD.

## Automated Metrics

Traditional NLP metrics: BLEU, ROUGE, BERTScore. Fast and cheap. Often poorly correlated with actual quality for open-ended generation tasks.

## LLM-as-a-Judge

Use a capable LLM (GPT-4, Claude) to score outputs against rubrics. 80-90% agreement with human raters. Scales to thousands of evaluations per hour at ~$0.01-0.05 each.

## Sources

- [MMLU: Measuring Massive Multitask Language Understanding (Hendrycks et al., 2020)](https://arxiv.org/abs/2009.03300)
- [HumanEval: Evaluating Large Language Models Trained on Code (Chen et al., 2021)](https://arxiv.org/abs/2107.03374)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods (Lin et al., 2021)](https://arxiv.org/abs/2109.07958)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (Zheng et al., 2023)](https://arxiv.org/abs/2306.05685)
- [2026 LLM Evaluation & Benchmarking: MMLU Saturation, GPQA Diamond, ARC-AGI-2 (Zylos)](https://zylos.ai/research/2026-01-16-llm-evaluation-benchmarking)

---

# The Three-Layer Evaluation Architecture

Modern evaluation stacks are layered: each layer is broader and cheaper than the next, and each one's failures feed deeper, more expensive review.

```mermaid
flowchart TB
    L1["Layer 1: Automated benchmarks<br/>broad, cheap, fast"] -->|failures| L2["Layer 2: LLM-as-Judge<br/>moderate depth, low cost"]
    L2 -->|failures| L3["Layer 3: Human evaluation<br/>deep, expensive"]
```

- **Layer 1 -- Automated benchmarks**: broad and cheap. Run on every change; catch obvious regressions across many cases.
- **Layer 2 -- LLM-as-Judge**: moderate depth at low cost. Score the cases that pass Layer 1 against rubrics; surface subtler quality issues.
- **Layer 3 -- Human evaluation**: deep but expensive. Reserve for the ambiguous, high-stakes, or judge-disagreement cases that Layer 2 flags.

Each layer feeds its failures to the next, so expensive human attention is spent only where cheaper layers cannot decide.

## From Model-Level to System-Level Evaluation

2026 frameworks have shifted from scoring individual responses to scoring **full agent pipelines end-to-end**:

- **End-to-end task success**: did the system actually complete the user's goal?
- **Tool-call correctness**: were the right tools called with the right arguments?
- **Latency and cost**: operational quality is now a first-class eval metric, not an afterthought

A model that produces great individual answers can still fail as a system -- wrong tool, too slow, too expensive. Evaluate the pipeline, not just the response.

## Sources

- [LLM Evaluation Frameworks: 2025 vs 2026 -- The Three-Layer Architecture (MLAI Digital)](https://www.mlaidigital.com/blogs/llm-evaluation-frameworks-2025-vs-2026-what-matters-now-2026)
- [The Best LLM Evaluation Tools of 2026 -- System-Level Evaluation (Online Inference)](https://medium.com/online-inference/the-best-llm-evaluation-tools-of-2026-40fd9b654dce)

---

# What to Measure

## The Five Pillars of LLM Evaluation

1. **Accuracy / Correctness**
   - Does the output contain factually correct information?
   - Can it be verified against a ground truth or reference?

2. **Faithfulness / Groundedness**
   - Is the output grounded in the provided context?
   - Does it avoid adding information not present in the source?

3. **Relevance**
   - Does the output actually answer the question asked?
   - Is the retrieved context relevant to the query?

4. **Helpfulness / Completeness**
   - Is the answer thorough enough to be useful?
   - Does it address all parts of a multi-part question?

5. **Safety / Harmlessness**
   - Does the output avoid toxic, biased, or harmful content?
   - Does it refuse appropriately when it should?

## Choosing Your Metrics

- **RAG systems**: prioritize faithfulness and context quality
- **Chat assistants**: prioritize helpfulness and safety
- **Code generation**: prioritize correctness and executability
- **Summarization**: prioritize faithfulness and completeness

Not every metric matters equally for every application. Define what "good" means for YOUR use case first.

---

# RAG-Specific Metrics

RAG evaluation requires measuring TWO distinct stages independently.

```mermaid
flowchart LR
    Q[Query] --> Ret[Retriever]
    Ret -.->|measure| RM["Retrieval metrics<br/>precision, recall, MRR, NDCG"]
    Ret --> Gen[Generator]
    Gen -.->|measure| GM["Generation metrics<br/>faithfulness, relevancy,<br/>correctness"]
    Gen --> Ans[Answer]
```

## Retrieval Quality

**Did we find the right documents?**

- **Context Precision**: What fraction of retrieved chunks are actually relevant?
  - `Precision = relevant_retrieved / total_retrieved`
- **Context Recall**: Did we retrieve all the relevant chunks?
  - `Recall = relevant_retrieved / total_relevant`
- **Mean Reciprocal Rank (MRR)**: Is the most relevant chunk ranked first?
  - `MRR = 1/rank_of_first_relevant`
- **NDCG@k**: Are relevant chunks ranked higher than irrelevant ones?

## Generation Quality

**Given the retrieved context, is the answer good?**

- **Faithfulness**: Is every claim in the answer supported by the context?
  - `Faithfulness = supported_claims / total_claims`
- **Answer Relevancy**: Does the answer address the original question?
- **Answer Correctness**: Does the answer match a known ground truth?
  - Combines factual similarity + semantic similarity
- **Hallucination Rate**: What fraction of claims are fabricated?

---

**Key insight**: A bad answer can come from bad retrieval OR bad generation. You must measure both to know where to fix.

---

# The RAGAS Framework

## Retrieval Augmented Generation Assessment

RAGAS (Es et al., 2023) provides a standardized set of metrics specifically designed for RAG pipeline evaluation.

## Core Metrics

| Metric | What It Measures | Requires Ground Truth? |
|---|---|---|
| **Faithfulness** | Is the answer grounded in the context? | No |
| **Answer Relevancy** | Does the answer address the question? | No |
| **Context Precision** | Are retrieved chunks relevant? | Yes |
| **Context Recall** | Did we retrieve all needed info? | Yes |
| **Answer Correctness** | Does the answer match expected output? | Yes |

## Why RAGAS?

- **Reference-free metrics**: Faithfulness and answer relevancy don't need ground truth labels -- you can run them on production traffic
- **LLM-based scoring**: Uses an LLM to decompose answers into claims and verify each one
- **Component-level diagnosis**: Pinpoints whether failures come from retrieval or generation
- **Framework integrations**: Works with LangChain, LlamaIndex, Haystack out of the box

## The RAGAS Score

A single composite score combining the core metrics:

```
RAGAS Score = Harmonic Mean(Faithfulness, Answer Relevancy,
                            Context Precision, Context Recall)
```

Typical production targets: RAGAS Score > 0.7 for general QA, > 0.85 for high-stakes domains.

## Sources

- [RAGAS: Automated Evaluation of Retrieval Augmented Generation (Es et al., 2023)](https://arxiv.org/abs/2309.15217)
- [RAGAS Documentation](https://docs.ragas.io)

---

# Faithfulness

## Is the Answer Grounded in the Retrieved Context?

Faithfulness is the single most important metric for RAG systems. A high faithfulness score means the LLM is not hallucinating beyond the provided context.

## How RAGAS Computes Faithfulness

```mermaid
flowchart LR
    Ans[Generated answer] --> Decomp["1. Decompose<br/>into atomic claims"]
    Decomp --> Verify["2. Verify each claim<br/>against context"]
    Ctx[Retrieved context] --> Verify
    Verify --> Score["3. score =<br/>supported / total"]
```

**Step 1: Claim Decomposition**
The LLM breaks the generated answer into individual atomic claims.

```
Answer: "Paris is the capital of France. It has a population
         of 2.1 million and was founded in the 3rd century BC."

Claims: [
  "Paris is the capital of France",
  "Paris has a population of 2.1 million",
  "Paris was founded in the 3rd century BC"
]
```

**Step 2: Claim Verification**
Each claim is checked against the retrieved context for support.

```
Context mentions: capital status (YES), population (YES),
                  founding date (NO -- not in context)

Supported: 2 / 3
```

**Step 3: Score Calculation**

```
Faithfulness = number_of_supported_claims / total_claims
Faithfulness = 2 / 3 = 0.67
```

## Interpreting Faithfulness Scores

- **1.0**: Every claim in the answer is supported by context (ideal)
- **0.7-0.9**: Mostly grounded, some minor additions
- **< 0.5**: Significant hallucination -- the model is inventing information

## Common Causes of Low Faithfulness

- Context is retrieved but the model ignores it and relies on parametric knowledge
- The model over-elaborates with details not in the retrieved chunks
- Incorrect summarization or paraphrasing that changes meaning

## Sources

- [RAGAS: Automated Evaluation of Retrieval Augmented Generation (Es et al., 2023)](https://arxiv.org/abs/2309.15217)
- [RAGAS Documentation — Faithfulness](https://docs.ragas.io)

---

# Context Precision & Recall

Did we retrieve the right information?

## Context Precision

**Are the retrieved chunks relevant to the question?**

Measures signal-to-noise ratio in your retrieval results.

```
Precision@k =
  (relevant chunks in top-k) / k
```

**Weighted precision** (RAGAS version):

```
CP = (1/k) * SUM[ (precision@i * rel_i) ]
     for i = 1..k where rel_i in {0,1}
```

Penalizes relevant docs ranked lower.

**Low precision means**: retrieving too much irrelevant context, which can confuse the generator and increase latency/cost.

## Context Recall

**Did we retrieve all the information needed to answer?**

Requires ground truth answers for comparison.

```
Recall = (GT statements attributable
          to context) / (total GT statements)
```

RAGAS decomposes the ground truth answer into statements, then checks if each can be attributed to the retrieved context.

**Low recall means**: missing critical information. The generator can't produce a correct answer if the right chunks were never retrieved.

---

**The precision-recall tradeoff**: Retrieving more chunks (higher k) tends to improve recall but hurt precision. Typical sweet spot: k=3-5 for most RAG applications.

## Sources

- [RAGAS: Automated Evaluation of Retrieval Augmented Generation (Es et al., 2023)](https://arxiv.org/abs/2309.15217)
- [RAGAS Documentation](https://docs.ragas.io)

---

# Answer Relevancy & Correctness

## Is the Answer Actually Useful?

## Answer Relevancy

Measures whether the generated answer addresses the original question (not whether it's factually correct).

**How it works (RAGAS):**
1. Given the answer, use an LLM to generate N hypothetical questions that the answer could be responding to
2. Compute cosine similarity between each generated question and the original question
3. Average the similarities

```
Answer Relevancy = (1/N) * SUM[ cosine_sim(q_original, q_generated_i) ]
```

**Example:**
- Question: "What is the return policy?"
- Answer: "Our company was founded in 1995 in Seattle..."
- Generated questions would be about company history, NOT return policies
- Low similarity with original question = low relevancy score

## Answer Correctness

Combines factual overlap with semantic similarity against a ground truth reference.

```
Answer Correctness = w1 * F1_factual + w2 * semantic_similarity
```

Where `F1_factual` is computed by:
- Decomposing both answer and ground truth into claims
- Computing TP (matching claims), FP (extra claims), FN (missing claims)
- `F1 = 2 * TP / (2*TP + FP + FN)`

Default weights: `w1 = 0.75, w2 = 0.25` (factual accuracy weighted higher).

## When to Use Each

- **Answer Relevancy**: Always -- doesn't require ground truth, catches off-topic answers
- **Answer Correctness**: When you have labeled test sets with expected answers

## Sources

- [RAGAS: Automated Evaluation of Retrieval Augmented Generation (Es et al., 2023)](https://arxiv.org/abs/2309.15217)
- [RAGAS Documentation](https://docs.ragas.io)

---

# End-to-End RAG Evaluation Pipeline

```mermaid
flowchart TB
    DS["Eval dataset<br/>(question, ground_truth, ref_docs)"] --> Pipe
    subgraph "Your RAG pipeline"
      Pipe[Retriever] --> Chunks[Retrieved chunks]
      Chunks --> Gen[Generator]
      Gen --> Ans[Answer]
    end
    Ans --> Ragas
    Chunks --> Ragas
    DS --> Ragas
    subgraph "RAGAS evaluation"
      Ragas{Has ground truth?}
      Ragas -->|no| NoGT["Faithfulness<br/>Answer relevancy"]
      Ragas -->|yes| GT["Context recall/precision<br/>Answer correctness"]
    end
    NoGT --> Results["Results & diagnostics<br/>(per-sample + aggregate)"]
    GT --> Results
```

**Key point**: You need four columns of data -- question, answer, contexts, and (optionally) ground_truth -- to run the full RAGAS evaluation suite.

## Sources

- [RAGAS: Automated Evaluation of Retrieval Augmented Generation (Es et al., 2023)](https://arxiv.org/abs/2309.15217)
- [RAGAS Documentation](https://docs.ragas.io)

---

# Building Evaluation Datasets

Your evaluation is only as good as your dataset. Here's how to build one.

```mermaid
flowchart LR
    S1["1. Manual curation<br/>50-100 expert pairs"] --> S2["2. Synthetic generation<br/>RAGAS TestsetGenerator"]
    S2 --> S3["3. Production logging<br/>sample real traffic"]
    S3 --> S4["4. Continuous maintenance<br/>quarterly review"]
    S4 -.->|new failures| S1
```

## Step 1: Manual Curation (Start Here)

- Domain experts write 50-100 question-answer pairs covering key use cases
- Include easy questions, hard questions, edge cases, and adversarial inputs
- Label which source documents contain the answer
- This is your "golden set" -- treat it like test fixtures in software

**Tip**: Categorize questions by type (factual lookup, multi-hop reasoning, comparison, summarization) to diagnose failures.

## Step 2: Synthetic Generation (Scale Up)

- Use an LLM to generate question-answer pairs from your document corpus
- RAGAS `TestsetGenerator` can create diverse question types automatically
- Filter and validate synthetic examples with human review (keep ~60-70%)
- Target: 200-500 examples for meaningful statistical power

```python
from ragas.testset import TestsetGenerator
generator = TestsetGenerator.from_langchain(generator_llm, critic_llm)
testset = generator.generate_with_langchain_docs(documents, test_size=200)
```

## Step 3: Production Logging (Stay Current)

- Log real user queries, retrieved contexts, and generated answers
- Sample and annotate production traffic weekly (thumbs up/down + corrections)
- Add interesting failures to your golden set
- Track distribution shift: are users asking questions you didn't anticipate?

## Step 4: Continuous Maintenance

- Review and update eval datasets quarterly
- Remove stale questions when source documents change
- Add new categories as your application evolves
- Version your eval datasets alongside your code

---

**Target composition**: 30% curated golden set, 40% validated synthetic, 30% annotated production samples.

## Sources

- [RAGAS Documentation — Test Set Generation](https://docs.ragas.io)

---

# LLM-as-a-Judge

## Using One LLM to Evaluate Another

## The Core Idea

Instead of writing heuristic scoring functions or hiring human annotators, use a capable LLM to read an output and score it against defined criteria.

```mermaid
flowchart LR
    In["question + answer<br/>+ context + rubric"] --> Judge["Judge LLM<br/>(GPT-4o, Claude)"]
    Judge --> Out["score (1-5)<br/>+ reasoning"]
```

## Why This Works

- Large LLMs have internalized human preferences from RLHF training
- They can follow detailed rubrics consistently across thousands of examples
- They provide natural language explanations for their scores (auditable)
- They can evaluate subjective qualities (tone, helpfulness) that rules can't capture

## The Key Insight

**RAGAS already does this under the hood.** Every RAGAS metric uses an LLM as a judge to decompose claims, verify support, and assess relevance. LLM-as-a-Judge is the engine powering modern automated evaluation.

## When to Use LLM-as-a-Judge

- Evaluating open-ended generation where there's no single "right answer"
- Assessing subjective qualities: helpfulness, tone, clarity, safety
- Scaling evaluation beyond what human reviewers can handle
- Running evaluation in CI/CD pipelines on every code change

## Sources

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (Zheng et al., 2023)](https://arxiv.org/abs/2306.05685)
- [RAGAS: Automated Evaluation of Retrieval Augmented Generation (Es et al., 2023)](https://arxiv.org/abs/2309.15217)

---

# Why LLM-as-a-Judge Works

## High Correlation with Human Judgment at Lower Cost

## Benchmark Evidence

Research consistently shows strong agreement between LLM judges and human evaluators:

| Study | Judge Model | Human Agreement | Task |
|---|---|---|---|
| MT-Bench (Zheng et al., 2023) | GPT-4 | 85% pairwise agreement | Multi-turn chat |
| AlpacaEval 2 | GPT-4-Turbo | 69.2% Spearman correlation | Instruction following |
| G-Eval (Liu et al., 2023) | GPT-4 | 0.514 Spearman (summarization) | Coherence, fluency |
| RAGAS (Es et al., 2023) | GPT-3.5+ | Significant correlation | RAG faithfulness |
| Prometheus (Kim et al., 2024) | Prometheus-2 | 0.59 Pearson | Open-ended scoring |

## Cost and Speed Comparison

| Method | Cost per Evaluation | Turnaround | Scalability |
|---|---|---|---|
| Expert human review | $5 - $15 | Days to weeks | Low (10-50/day/person) |
| Crowdsourced review | $0.50 - $2 | Hours to days | Medium (100s/day) |
| LLM judge (GPT-4o) | $0.01 - $0.05 | Seconds | High (1000s/hour) |
| LLM judge (GPT-4o-mini) | $0.001 - $0.005 | Seconds | Very high |

## The Practical Tradeoff

- LLM judges are **not perfect** -- they reach 80-90% agreement with humans at 500-5000x lower cost
- But human annotators only agree with EACH OTHER ~80-85% of the time (inter-annotator agreement)
- LLM judges are **consistent** -- same input always gets similar scores (no bad days, no fatigue)
- Use human eval to calibrate and validate your LLM judge, then scale with the LLM

## 2026 Best Practice

- Prefer an **ensemble of judge models** over a single judge -- it averages out individual model biases (see Multi-Judge Panels)
- **Calibrate judges against human ratings** on a held-out set before trusting them in production; report the agreement number, don't assume it

## Sources

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (Zheng et al., 2023)](https://arxiv.org/abs/2306.05685)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment (Liu et al., 2023)](https://arxiv.org/abs/2303.16634)
- [RAGAS: Automated Evaluation of Retrieval Augmented Generation (Es et al., 2023)](https://arxiv.org/abs/2309.15217)
- [Prometheus: Inducing Fine-Grained Evaluation Capability in Language Models (Kim et al., 2023)](https://arxiv.org/abs/2310.08491)
- [Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators (Dubois et al., 2024)](https://arxiv.org/abs/2404.04475)
- [LLM Evaluation in 2026: Judge Ensembles & Human Calibration (Meta-Intelligence)](https://www.meta-intelligence.tech/en/insight-llm-evaluation)

---

# Judge Patterns

Three fundamental approaches to LLM-based evaluation.

## Pairwise Comparison

Given a question, compare Answer A vs Answer B. The judge picks a winner or declares a tie.

**Pros**: Easier for LLMs than absolute scoring. Powers Chatbot Arena, AlpacaEval, MT-Bench.

**Prompt**: "Which response better answers the question? Choose A, B, or Tie. Explain your reasoning."

**Best for**: Model comparison, A/B testing prompt variants.

## Pointwise Scoring

Rate a single answer on a numeric scale (1-5 or 1-10) against defined criteria.

**Pros**: Simpler to implement, gives absolute scores you can threshold on. Used by G-Eval, RAGAS internally.

**Prompt**: "Rate this response from 1-5 on [criterion]. 1 = completely fails, 5 = excellent. Provide your score and reasoning."

**Best for**: CI/CD gating, regression testing, monitoring dashboards.

## Reference-Based Grading

Compare the answer against a known correct reference answer.

**Pros**: Most objective -- the judge checks factual overlap, not just vibes. Used for answer correctness in RAGAS.

**Prompt**: "Given the reference answer, score the candidate answer on factual accuracy and completeness (1-5)."

**Best for**: QA systems with known answers, exam-style evaluation.

## Sources

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (Zheng et al., 2023)](https://arxiv.org/abs/2306.05685)
- [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference (Chiang et al., 2024)](https://arxiv.org/abs/2403.04132)
- [Length-Controlled AlpacaEval (Dubois et al., 2024)](https://arxiv.org/abs/2404.04475)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment (Liu et al., 2023)](https://arxiv.org/abs/2303.16634)
- [RAGAS: Automated Evaluation of Retrieval Augmented Generation (Es et al., 2023)](https://arxiv.org/abs/2309.15217)

---

# Designing Judge Prompts

## Rubrics, Criteria, and Scoring Scales

## Anatomy of a Good Judge Prompt

A well-designed judge prompt has five components:

1. **Role**: "You are an expert evaluator assessing the quality of AI-generated responses."
2. **Criteria**: Exactly what dimension(s) to evaluate (one at a time is best)
3. **Rubric**: Concrete descriptions for each score level
4. **Input format**: Where to find the question, answer, context, reference
5. **Output format**: Structured scoring with mandatory reasoning before the score

## Example: Faithfulness Rubric

```
Score 5: Every claim is directly supported by the provided context.
Score 4: Nearly all claims supported; one minor unsupported detail.
Score 3: Most claims supported, but 1-2 significant unsupported claims.
Score 2: Multiple unsupported claims that materially affect accuracy.
Score 1: The response contains fabricated information not in the context.
```

## Best Practices

- **One criterion per prompt**: Evaluating helpfulness AND safety in one call degrades both
- **Chain-of-thought first**: Require the judge to explain reasoning BEFORE giving a score (reduces random scoring)
- **Anchor with examples**: Include 1-2 scored examples in the prompt to calibrate
- **Use odd scales**: 1-5 works better than 1-4 (avoids forced middle) or 1-10 (too granular, inconsistent)
- **Structured output**: Use JSON output format for reliable parsing

```json
{
  "reasoning": "The answer correctly states X and Y from the context,
                but the claim about Z is not supported...",
  "score": 3
}
```

---

# Known Biases in LLM Judges

## Understanding the Failure Modes

LLM judges are powerful but systematically biased. You must know these biases to build reliable evaluation.

## Position Bias

- In pairwise comparisons, LLMs prefer the response presented **first** (or sometimes last)
- GPT-4 shows ~10-15% position bias in MT-Bench experiments
- The same response can win or lose depending on whether it's shown as "A" or "B"

## Verbosity Bias

- LLMs tend to prefer **longer, more detailed** responses regardless of accuracy
- A verbose but partially wrong answer may score higher than a concise correct one
- Measured at 5-10% bias toward longer outputs in controlled studies

## Self-Preference Bias (Self-Enhancement)

- LLMs rate their own outputs ~10% higher than outputs from other models
- GPT-4 judging GPT-4 outputs vs Claude outputs shows measurable self-preference
- This makes same-model evaluation unreliable for model comparison

## Formatting Bias

- Responses with markdown formatting, bullet points, and headers score higher
- Well-structured wrong answers can outscore poorly-formatted correct answers

## Authority/Confidence Bias

- Responses that sound confident and authoritative score higher
- Hedging language ("I think", "it might be") is penalized even when appropriate

## Sensitivity to Prompt Wording

- Small changes in the judge prompt can shift scores by 0.5-1.0 points on a 5-point scale
- The order of criteria in the rubric affects which criteria get weighted more heavily

## Sources

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (Zheng et al., 2023)](https://arxiv.org/abs/2306.05685)

---

# Mitigating Judge Bias

## Practical Techniques for More Reliable LLM Evaluation

## 1. Swap Positions (for Pairwise)

Run every comparison twice with A and B swapped. Only count a win if the same response wins in both orderings. Ties when results conflict.

```python
result_1 = judge(question, answer_a=X, answer_b=Y)  # X first
result_2 = judge(question, answer_a=Y, answer_b=X)  # Y first
final = X_wins if both agree else Tie
```

Reduces position bias from ~15% to ~3%.

## 2. Use Detailed Rubrics

Vague criteria ("rate quality 1-5") amplify all biases. Specific rubrics with concrete examples for each score level reduce variance by 20-30%.

## 3. Calibrate with Human Labels

- Score 100+ examples with both human raters and the LLM judge
- Compute Cohen's kappa or Spearman correlation
- If agreement < 0.6, revise your rubric or switch judge models
- Re-calibrate quarterly as models update

## 4. Use a Different Model as Judge

Avoid self-preference bias by using a different model family as judge. Common pattern: use GPT-4o to judge Claude outputs, or vice versa.

## 5. Multi-Metric Decomposition

Instead of one holistic "quality" score, evaluate multiple specific dimensions separately. Aggregating 4-5 narrow scores is more reliable than one broad score.

## 6. Control for Length

Add explicit instructions: "Do not prefer longer responses. A concise correct answer should score higher than a verbose partially-correct one."

## Validation Checklist

- [ ] Position bias test: < 5% swing when swapping A/B
- [ ] Length correlation: Spearman between score and length < 0.3
- [ ] Human agreement: Cohen's kappa > 0.6
- [ ] Test-retest reliability: same inputs produce same scores (>90%)

## Sources

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (Zheng et al., 2023)](https://arxiv.org/abs/2306.05685)

---

# Multi-Judge Panels

## Aggregate Multiple LLM Judges for Robustness

## The Idea

Just as academic papers use multiple reviewers, use multiple LLM judges and aggregate their scores. This reduces individual model biases and increases reliability.

```mermaid
flowchart LR
    In[Answer + rubric] --> J1[GPT-4o judge]
    In --> J2[Claude judge]
    In --> J3[Gemini judge]
    J1 --> Agg["Aggregate<br/>(median / majority vote)"]
    J2 --> Agg
    J3 --> Agg
    Agg --> Score[Final score]
```

## Panel Configurations

**Same model, different prompts:**
```python
scores = [
    judge(rubric_v1, answer),   # Rubric focused on accuracy
    judge(rubric_v2, answer),   # Rubric focused on completeness
    judge(rubric_v3, answer),   # Rubric focused on clarity
]
final_score = median(scores)
```

**Different models, same prompt:**
```python
scores = [
    gpt4o_judge(rubric, answer),      # GPT-4o
    claude_judge(rubric, answer),      # Claude 3.5 Sonnet
    gemini_judge(rubric, answer),      # Gemini Pro
]
final_score = majority_vote(scores)  # or median
```

## Aggregation Strategies

| Strategy | When to Use | Robustness |
|---|---|---|
| Majority vote | Pairwise comparisons | High -- eliminates outlier judges |
| Mean | Pointwise scores, smooth distribution | Medium -- sensitive to outliers |
| Median | Pointwise scores, noisy judges | High -- robust to outliers |
| Weighted average | When you know which judge is best | Highest -- requires calibration data |

## Practical Considerations

- **Cost**: 3 judges = 3x cost. Budget ~$0.03-0.15 per evaluation
- **Latency**: Run judges in parallel to avoid 3x latency
- **Diminishing returns**: 3 judges captures most of the benefit; 5+ judges rarely worth the cost
- **Disagreement signal**: When judges disagree strongly, flag for human review

## The PoLL Protocol (Verga et al., 2024)

Panel of LLM Evaluators showed that a panel of smaller, diverse models outperforms a single large model judge. Three GPT-3.5-level models in a panel achieved higher human agreement than a single GPT-4 judge, at lower cost.

## Sources

- [Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models (Verga et al., 2024)](https://arxiv.org/abs/2404.18796)

---

# RAGAS Deep Dive

## Setup, Metrics, and Integration

## Installation and Basic Usage

```python
pip install ragas langchain-openai

from ragas import evaluate
from ragas.metrics import (
    faithfulness, answer_relevancy,
    context_precision, context_recall,
    answer_correctness
)
from datasets import Dataset

# Prepare your evaluation data
eval_data = {
    "question": ["What is the refund policy?", ...],
    "answer": ["You can return items within 30 days...", ...],
    "contexts": [["Policy doc: Returns accepted within 30 days..."], ...],
    "ground_truth": ["Items can be returned within 30 days for a full refund.", ...]
}
dataset = Dataset.from_dict(eval_data)

# Run evaluation
results = evaluate(
    dataset,
    metrics=[faithfulness, answer_relevancy,
             context_precision, context_recall],
)
print(results)  # {'faithfulness': 0.82, 'answer_relevancy': 0.91, ...}
```

## LangChain Integration

```python
from ragas.integrations.langchain import EvaluatorChain

faithfulness_chain = EvaluatorChain(metric=faithfulness)
result = faithfulness_chain.invoke({
    "query": question, "result": answer, "source_documents": docs
})
```

## Configuration Options

- **LLM choice**: Default is GPT-4o. Switch to Claude or open-source models via `LangchainLLMWrapper`
- **Embeddings**: Default is OpenAI. Swap for local embeddings to reduce cost
- **Batch processing**: Use `evaluate()` with `raise_exceptions=False` for robust batch runs
- **Async support**: `evaluate()` runs LLM calls concurrently for 5-10x speedup

## Cost Estimation

| Metric | LLM Calls per Sample | Approx Cost (GPT-4o) |
|---|---|---|
| Faithfulness | 2-3 (decompose + verify) | $0.02 - $0.04 |
| Answer Relevancy | 2 (generate questions + embed) | $0.01 - $0.02 |
| Context Precision | 1-2 | $0.01 - $0.02 |
| Context Recall | 1-2 | $0.01 - $0.02 |
| Full suite (5 metrics) | 8-12 | $0.06 - $0.12 |

**Budget**: ~$12-24 to evaluate 200 samples across all metrics.

## Sources

- [RAGAS: Automated Evaluation of Retrieval Augmented Generation (Es et al., 2023)](https://arxiv.org/abs/2309.15217)
- [RAGAS Documentation](https://docs.ragas.io)

---

# DeepEval

## Unit Testing for LLM Applications

## Philosophy

DeepEval treats LLM evaluation like software testing: write test cases, define assertions, run them in CI/CD, get pass/fail results.

## Core Concepts

```python
pip install deepeval

from deepeval import assert_test
from deepeval.test_case import LLMTestCase
from deepeval.metrics import (
    FaithfulnessMetric,
    AnswerRelevancyMetric,
    HallucinationMetric,
    GEval
)

# Define a test case
test_case = LLMTestCase(
    input="What is our vacation policy?",
    actual_output="Employees get 20 days of PTO per year.",
    retrieval_context=["HR Policy: Full-time employees receive 20 days PTO."],
    expected_output="Full-time employees receive 20 days of paid time off."
)

# Define metrics with thresholds
faithfulness = FaithfulnessMetric(threshold=0.7)
relevancy = AnswerRelevancyMetric(threshold=0.7)

# Assert -- fails the test if below threshold
assert_test(test_case, [faithfulness, relevancy])
```

## Integration with pytest

```python
# test_rag.py -- runs with: deepeval test run test_rag.py
import pytest
from deepeval import assert_test

@pytest.mark.parametrize("test_case", eval_dataset)
def test_faithfulness(test_case):
    assert_test(test_case, [FaithfulnessMetric(threshold=0.7)])
```

## G-Eval: Custom Criteria

```python
correctness = GEval(
    name="Correctness",
    criteria="Determine if the output is factually correct.",
    evaluation_params=[LLMTestCaseParams.ACTUAL_OUTPUT,
                       LLMTestCaseParams.EXPECTED_OUTPUT],
    threshold=0.5
)
```

## Key Features

- **Confident AI dashboard**: Free hosted platform to track eval results over time
- **50+ research-backed metrics**: deterministic and LLM-as-Judge -- faithfulness, hallucination, bias, toxicity, summarization, and more
- **CI integration**: pass/fail gates that run on every code change, like a unit-test suite
- **Synthesizer**: Generate test datasets from documents (similar to RAGAS TestsetGenerator)
- **Conversational metrics**: Evaluate multi-turn conversations, not just single QA pairs

> Together with **RAGAS** and **TruLens**, DeepEval forms the standard open-source evaluation triad recommended for production teams.

## Sources

- [DeepEval GitHub Repository](https://github.com/confident-ai/deepeval)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment (Liu et al., 2023)](https://arxiv.org/abs/2303.16634)
- [LLM Evaluation Frameworks Compared — DeepEval, RAGAS, TruLens (Atlan)](https://atlan.com/know/llm-evaluation-frameworks-compared/)

---

# Phoenix / Arize

## Observability-Driven Evaluation and Tracing

## The Observability Approach

Phoenix (by Arize AI, open-source) combines **tracing** and **evaluation** -- you instrument your RAG pipeline to capture every step, then evaluate the captured traces.

## Tracing: See What Your Pipeline Actually Does

```python
pip install arize-phoenix opentelemetry-instrumentation

import phoenix as px
from phoenix.otel import register
from openinference.instrumentation.langchain import LangChainInstrumentor

# Launch Phoenix UI
px.launch_app()

# Auto-instrument LangChain
tracer_provider = register(project_name="my-rag-app")
LangChainInstrumentor().instrument(tracer_provider=tracer_provider)

# Now every LangChain call is traced automatically
result = rag_chain.invoke("What is the return policy?")
# Phoenix UI shows: query -> retriever -> reranker -> LLM -> response
```

## Evaluation on Traces

```python
from phoenix.evals import llm_classify, RAG_RELEVANCY_PROMPT

# Pull traces from Phoenix
trace_df = px.Client().get_spans_dataframe()

# Run LLM-as-judge evaluation on retrieved documents
results = llm_classify(
    dataframe=trace_df,
    template=RAG_RELEVANCY_PROMPT,
    model=OpenAIModel(model="gpt-4o"),
    rails=["relevant", "irrelevant"]
)
```

## Why Phoenix?

- **Zero-code instrumentation**: Auto-instruments LangChain, LlamaIndex, OpenAI, and more
- **Visual trace explorer**: See exactly which chunks were retrieved, what the prompt looked like, and what the LLM generated
- **Built-in eval templates**: Relevancy, hallucination, QA correctness, toxicity
- **Local-first**: Runs on your machine, no data leaves your environment
- **Production monitoring**: Track metrics over time, set up alerts for quality degradation

## Sources

- [Phoenix GitHub Repository (Arize AI)](https://github.com/Arize-ai/phoenix)
- [Phoenix Documentation](https://phoenix.arize.com/)

---

# Braintrust, Promptfoo & Other Platforms

## The Eval Platform Landscape

## Braintrust

- **Focus**: Experiment tracking and scoring for LLM applications
- **Key feature**: Side-by-side comparison of prompt variants with automatic scoring
- **Differentiator**: Strong logging and dataset management, team collaboration
- **Pricing**: Free tier available, paid for teams

```python
import braintrust
experiment = braintrust.init(project="my-rag", experiment="v2-prompt")
for case in eval_dataset:
    output = rag_pipeline(case.question)
    experiment.log(input=case.question, output=output,
                   expected=case.ground_truth,
                   scores={"faithfulness": score})
```

## Promptfoo

- **Focus**: CLI-first eval tool for prompt engineering and red-teaming
- **Key feature**: YAML-based test configuration, model comparison grids
- **Differentiator**: Excellent for comparing multiple models/prompts simultaneously
- **Pricing**: Open-source core, cloud features paid

```yaml
# promptfoo config.yaml
prompts:
  - "Answer based on context: {{context}}\nQ: {{question}}"
  - "You are a helpful assistant. Context: {{context}}\nQ: {{question}}"
providers: [openai:gpt-4o, anthropic:claude-3.5-sonnet]
tests:
  - vars: { question: "What is the refund policy?", context: "..." }
    assert:
      - type: llm-rubric
        value: "Answer accurately describes the refund policy"
```

## Other Notable Tools

| Tool | Best For | Open Source? |
|---|---|---|
| **LangSmith** | LangChain-native tracing + eval | No (cloud) |
| **Weights & Biases Weave** | ML-team-friendly experiment tracking | Partial |
| **Patronus AI** | Enterprise compliance and safety eval | No |
| **Galileo** | Hallucination detection and monitoring | No |
| **OpenAI Evals** | Evaluating OpenAI models specifically | Yes |

## How to Choose

- **Just starting**: RAGAS + DeepEval (free, Python-native, fast to set up)
- **Prompt engineering focus**: Promptfoo (great CLI, model comparison)
- **Production observability**: Phoenix/Arize (tracing + eval in one tool)
- **Team collaboration**: Braintrust or LangSmith (dashboards, sharing)

## Sources

- [Braintrust](https://www.braintrust.dev)
- [Promptfoo GitHub Repository](https://github.com/promptfoo/promptfoo)
- [LangSmith Documentation](https://docs.smith.langchain.com)
- [Weights & Biases Weave](https://github.com/wandb/weave)
- [OpenAI Evals GitHub Repository](https://github.com/openai/evals)

---

# Building Custom Evaluation Pipelines

## When off-the-shelf metrics don't cover your domain requirements.

```mermaid
flowchart TB
    Crit["1. Define criteria<br/>compliance, brand voice,<br/>citation, terminology"] --> Rub["2. Build rubrics<br/>concrete score levels"]
    Rub --> Score["3. Implement scoring"]
    Score --> Rule[Rule-based checks<br/>regex, blocklist, format]
    Score --> LLMJ[LLM judge<br/>rubric + CoT]
    Rule --> Val["4. Validate & calibrate<br/>human-label 100 samples<br/>target kappa > 0.6"]
    LLMJ --> Val
    Val --> Int["5. Integrate<br/>CI/CD gate, dashboard,<br/>alerting"]
```

**Template for custom LLM judge metric:**

```python
CUSTOM_JUDGE_PROMPT = """
You are evaluating a {domain} AI assistant.
Criterion: {criterion_name}
Definition: {criterion_definition}
Rubric:
  5 - {score_5_description}
  3 - {score_3_description}
  1 - {score_1_description}

Question: {question}
Context: {context}
Answer: {answer}

First explain your reasoning, then provide your score (1-5).
Output JSON: {"reasoning": "...", "score": N}
"""
```

---

# Offline vs Online Evaluation

## Two complementary strategies for different stages of the development lifecycle.

```mermaid
flowchart LR
    Dev[Dev change] --> Offline["Offline eval<br/>golden set + synthetic"]
    Offline -->|pass regression| Deploy[Deploy]
    Offline -->|fail| Dev
    Deploy --> Online["Online eval<br/>sample prod traces"]
    Online -->|new failures| Golden[Add to golden set]
    Golden --> Offline
```

## Offline (Pre-Deployment)

**When**: Before shipping changes.

**What you evaluate**:
- Fixed test datasets (golden set + synthetic)
- Known question-answer pairs with ground truth
- Controlled conditions, reproducible results

**How**:
- RAGAS / DeepEval on curated eval sets
- Regression tests in CI/CD
- A/B comparisons of prompt variants
- Red-teaming for safety and edge cases

**Strengths**:
- Reproducible and comparable across versions
- Catches regressions before users see them
- Can test adversarial and edge cases safely

**Weakness**:
- Doesn't reflect real user behavior
- Dataset can become stale

## Online (Production Monitoring)

**When**: After deployment, continuously.

**What you evaluate**:
- Real user queries and actual system responses
- Production traffic patterns and edge cases
- User satisfaction signals

**How**:
- Sample and score production traces with LLM judges
- Collect implicit feedback (thumbs up/down, follow-ups)
- Monitor retrieval quality metrics on live queries
- Track latency, error rates, and fallback rates

**Strengths**:
- Reflects actual user experience
- Catches issues offline eval missed
- Discovers new query patterns

**Weakness**:
- Users see failures before you catch them
- Noisier signals, harder to debug

---

**Best practice**: Gate deployments on offline eval (must pass regression tests), then monitor online eval for the first 24-48 hours after every release.

---

# Regression Testing

## Ensuring Changes Don't Break Existing Behavior

## The Problem

You improve your prompt for one type of question. Three other question types silently degrade. Without regression testing, you won't know until users complain.

```mermaid
flowchart LR
    PR[Pull request] --> CI[CI runs eval]
    CI --> Golden[Golden set + new test cases]
    Golden --> Score[Score with metrics]
    Score --> Gate{Above thresholds?}
    Gate -->|yes| Merge[Merge]
    Gate -->|no| Block[Block PR]
```

## Setting Up Regression Tests

```python
# eval/test_regression.py
import json
from deepeval import assert_test
from deepeval.metrics import FaithfulnessMetric, AnswerRelevancyMetric

GOLDEN_SET = json.load(open("eval/golden_set.json"))

METRICS = [
    FaithfulnessMetric(threshold=0.7),
    AnswerRelevancyMetric(threshold=0.7),
]

def test_regression_suite():
    for case in GOLDEN_SET:
        output = rag_pipeline(case["question"])
        test_case = LLMTestCase(
            input=case["question"],
            actual_output=output["answer"],
            retrieval_context=output["contexts"],
            expected_output=case.get("ground_truth")
        )
        assert_test(test_case, METRICS)
```

## CI/CD Integration

```yaml
# .github/workflows/eval.yml
eval-regression:
  runs-on: ubuntu-latest
  steps:
    - uses: actions/checkout@v4
    - run: pip install -r requirements.txt
    - run: deepeval test run eval/test_regression.py
      env:
        OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
    - run: python eval/compare_to_baseline.py
```

## Tracking Metrics Over Time

- Store eval results as JSON artifacts alongside each commit/PR
- Compare current scores against the last release baseline
- Set **absolute thresholds** (faithfulness must be > 0.7) AND **relative thresholds** (faithfulness must not drop > 5% from baseline)
- Alert on regressions, block merges that fail eval

## What to Include in Your Golden Set

- **High-frequency queries**: The 50 most common real user questions
- **Known failure cases**: Questions that broke in the past (prevent re-occurrence)
- **Edge cases**: Ambiguous queries, multi-hop questions, out-of-scope questions
- **Safety tests**: Prompt injection attempts, requests for harmful content
- **Format diversity**: Short questions, long questions, multi-part questions

## Sources

- [DeepEval GitHub Repository](https://github.com/confident-ai/deepeval)

---

# Human Evaluation

## When You Still Need Humans in the Loop

## LLM Judges Are Not Enough

LLM judges fail or are unreliable for:

- **Novel domains**: The judge LLM may lack domain expertise (medical, legal, scientific)
- **Subjective quality**: Tone, brand voice, cultural appropriateness
- **Safety edge cases**: Subtle harmful content that LLMs don't flag
- **Calibration**: You need human labels to validate that your LLM judge is actually working
- **High-stakes decisions**: When wrong answers have legal or financial consequences

## Designing Human Evaluation

### Annotation Schema
- Use the SAME rubrics you give the LLM judge (enables direct comparison)
- Binary labels (good/bad) for quick passes; Likert scales (1-5) for nuanced analysis
- Always include "unsure" option to avoid forced guessing

### Sample Size Guidelines
| Confidence Level | Margin of Error | Required Samples |
|---|---|---|
| 90% | +/- 5% | ~270 |
| 95% | +/- 5% | ~385 |
| 95% | +/- 3% | ~1,068 |

### Inter-Annotator Agreement
- Have 2-3 annotators label the same samples
- Compute Cohen's kappa (2 raters) or Fleiss' kappa (3+ raters)
- Target kappa > 0.6 (substantial agreement) before trusting labels
- If kappa < 0.4, your rubric is ambiguous -- revise it

## The Hybrid Approach

1. **Calibration set**: 200 human-labeled examples
2. **Validate LLM judge**: Compare LLM scores to human labels (target agreement > 80%)
3. **Scale with LLM**: Use the validated LLM judge for the remaining thousands of examples
4. **Ongoing audit**: Human-review 5-10% of LLM-judged samples weekly to detect drift
5. **Escalation**: Route low-confidence LLM judgments to human reviewers

---

# Eval-Driven Development

## Write Evals Before Writing Prompts

## The Methodology

Eval-driven development (EDD) is test-driven development (TDD) for LLM applications. Just as TDD says "write the test first, then write the code," EDD says:

> **Write the eval first, then write the prompt.**

## The EDD Loop

```mermaid
flowchart LR
    Def[1. Define good] --> Data[2. Build eval set]
    Data --> Score[3. Set up scoring]
    Score --> Prompt[4. Write prompt<br/>establish baseline]
    Prompt --> Iter[5. Iterate using<br/>eval scores]
    Iter --> Add[6. Add failures<br/>to eval set]
    Add --> Prompt
```

1. **Define what "good" looks like** for your use case (criteria + rubrics)
2. **Build your eval dataset** (50-100 examples covering key scenarios)
3. **Set up automated scoring** (RAGAS / DeepEval / custom judge)
4. **Write your first prompt** and run evals -- establish a baseline
5. **Iterate on the prompt** using eval scores as your guide
6. **Add failure cases** to the eval set as you discover them
7. **Repeat** for every prompt change, model swap, or pipeline modification

## Why EDD Works

- **Prevents overfitting to anecdotes**: You optimize for aggregate performance, not cherry-picked examples
- **Makes tradeoffs visible**: Improving one dimension while degrading another shows up immediately
- **Enables confident refactoring**: Swap models, change chunking strategies, restructure prompts -- evals tell you if it's better
- **Compounds over time**: Your eval set becomes a living specification of your system's requirements

## Anti-Patterns to Avoid

- **Eval after the fact**: Building evals only after problems appear in production
- **Optimizing for the eval**: Overfitting prompts to pass specific test cases rather than genuinely improving quality
- **Stale datasets**: Never updating your eval set as your application evolves
- **Single-metric obsession**: Optimizing one metric (e.g., faithfulness) while ignoring others (e.g., helpfulness)
- **No baseline**: Making changes without recording the previous eval scores for comparison

## The EDD Manifesto

- Evals are **first-class artifacts**, versioned and reviewed like code
- Every prompt change requires an **eval run** before merge
- Eval results are **shared and visible** to the whole team
- **Failures are features**: every production bug becomes a new test case

---

# Key Takeaways

## Measure Both Stages

RAG evaluation requires separate metrics for retrieval quality (precision, recall) and generation quality (faithfulness, relevancy). Diagnose which component is failing before optimizing.

## Use RAGAS as Your Starting Point

RAGAS provides battle-tested metrics that work out of the box. Start with faithfulness and answer relevancy (no ground truth needed), then add context metrics as you build labeled datasets.

## LLM Judges Scale Evaluation

LLM-as-a-Judge achieves 80-85% human agreement at 100x lower cost. But know the biases (position, verbosity, self-preference) and mitigate them with position swapping, rubrics, and calibration.

## Eval-Driven Development

Write evals before prompts. Gate deployments on regression tests. Monitor production quality continuously. Your eval dataset is a living specification that grows with your application.

---

# Key Papers & References

## Essential Reading for LLM Evaluation

| Paper | Authors | Year | Key Contribution |
|---|---|---|---|
| [**RAGAS: Automated Evaluation of Retrieval Augmented Generation**](https://arxiv.org/abs/2309.15217) | Es et al. | 2023 | Framework for reference-free RAG evaluation; faithfulness and relevancy metrics |
| [**Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena**](https://arxiv.org/abs/2306.05685) | Zheng et al. | 2023 | Systematic study of LLM judge biases; MT-Bench benchmark; 85% GPT-4/human agreement |
| [**G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment**](https://arxiv.org/abs/2303.16634) | Liu et al. | 2023 | Chain-of-thought evaluation with probability-weighted scoring; state-of-the-art correlation with humans |
| [**Prometheus: Inducing Fine-Grained Evaluation Capability in Language Models**](https://arxiv.org/abs/2310.08491) | Kim et al. | 2023 | Open-source judge model; fine-tuned for evaluation with rubric-following |
| [**Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models (PoLL)**](https://arxiv.org/abs/2404.18796) | Verga et al. | 2024 | Panel of smaller models outperforms single large judge; cost-effective evaluation |
| [**Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences**](https://arxiv.org/abs/2404.12272) | Shankar et al. | 2024 | Meta-evaluation of LLM judges; calibration and validation methodology |

## Benchmarks to Know

| Benchmark | Purpose | Size |
|---|---|---|
| [**MT-Bench**](https://arxiv.org/abs/2306.05685) | Multi-turn conversation quality | 80 questions, 8 categories |
| [**AlpacaEval**](https://arxiv.org/abs/2404.04475) | Instruction-following quality | 805 instructions |
| [**TruthfulQA**](https://arxiv.org/abs/2109.07958) | Truthfulness and factuality | 817 questions |
| [**MMLU**](https://arxiv.org/abs/2009.03300) | Broad knowledge and reasoning | 14,042 questions, 57 subjects |
| [**Chatbot Arena**](https://arena.ai/) | Head-to-head model comparison | 1M+ human votes (crowd-sourced) |

## Recommended Further Reading

- [Hamel Husain's blog: "Your AI Product Needs Evals"](https://hamel.dev/blog/posts/evals/) (practical eval methodology)
- [Eugene Yan's "Patterns for Building LLM-based Systems & Products"](https://eugeneyan.com/writing/llm-patterns/) (eval section)
- [Anthropic's "Challenges in Evaluating AI Systems"](https://www.anthropic.com/research/evaluating-ai-systems) (safety-focused evaluation)

---

# Week 7 Wrap-Up

## What We Covered

RAG evaluation with RAGAS -- LLM-as-a-Judge patterns and biases -- Evaluation frameworks and tooling -- Production eval workflows

## The One Thing to Remember

> "You can't improve what you can't measure. In the world of LLMs, measurement itself is an LLM problem."

## Next Week

Week 8: Model Context Protocol — tools, resources, prompts, transport, and authorization.

## Assignment

Build a RAGAS evaluation pipeline for your Week 5/6 RAG project. Report faithfulness, answer relevancy, context precision, and context recall on a 50-question eval set. Compare at least two prompt or retrieval variants.