# How to evaluate an LLM application

Updated: 2026-11-13. https://thegccengineer.com/guides/llm-evaluation-basics

**Quick answer:** To evaluate an LLM application, build a golden set of representative inputs with expected results, score the application's outputs against it using code checks, human review or a validated LLM judge, and rerun that set on every change to catch regressions. Then confirm offline scores against online signals from real users. LLM evaluation measures your whole application on your task, not the model in general.

You evaluate an LLM application by running it against a fixed set of representative examples, scoring the outputs with checks you trust, and repeating that every time anything changes. The set is called a golden set, the scoring is a mix of code, human review and model-based judging, and the repetition is regression testing. LLM evaluation is the difference between "it looked good when I tried it" and knowing whether a change made the system better or worse.

Evaluation is the skill that most clearly separates a demo from a product, which is why it is a core item on the [AI engineer roadmap](https://thegccengineer.com/guides/ai-engineer-roadmap) for GenAI and ML engineers. It builds directly on the ideas in [machine learning fundamentals](https://thegccengineer.com/guides/machine-learning-fundamentals): held-out data, baselines and metrics chosen for the cost of errors.

## Why is evaluating an LLM application hard?

Evaluating an LLM application is hard because there is rarely a single correct output, and the system changes behaviour in ways traditional tests do not expect. Four properties cause most of the difficulty:

- **Many valid answers.** Two differently worded replies can both be right, so exact-match comparison fails.
- **Non-determinism.** The same input can produce different outputs on different runs.
- **Silent failure.** A wrong answer is fluent and confident; nothing crashes.
- **Coupled changes.** Editing a prompt to fix one case can quietly break five others.

One distinction clears up a lot of confusion. Model evaluation compares models on public benchmarks. Application evaluation measures your prompts, retrieval, tools and model together, on your task. A model that tops a leaderboard can still perform poorly in your application, so the second kind is the one you must do yourself.

## What is a golden set and how do you build one?

A golden set is a curated collection of inputs, each paired with what a good output must contain or satisfy. It is the fixed yardstick every version of the application is measured against.

**Building a golden set**

1. **Collect real inputs**: Use actual user questions or documents where you can. Invented examples are tidier than reality.
2. **Cover the range**: Include common cases, rare cases, ambiguous inputs and inputs the system should refuse or decline.
3. **Write the expectation**: Record required facts, the correct label or a rubric, not one exact wording.
4. **Review with an expert**: Have someone who knows the domain confirm the expectations and the criteria.
5. **Keep it separate**: Do not tune prompts against every example. Hold some back so the score stays honest.
6. **Grow it from failures**: Every bug found in production becomes a new test case.

The fifth step is the one people skip. If you adjust a prompt until every golden example passes, the set no longer tells you how the application handles new inputs. That is overfitting, the same problem as tuning a classical model on its test set.

## Which metrics should you use?

Use the cheapest metric that actually measures what you care about, and prefer checks that code can verify. Scoring methods form a ladder from cheap and exact to expensive and subjective.

| Method | What it checks | Strength | Weakness |
| --- | --- | --- | --- |
| Code assertions | Valid JSON, required fields, length, forbidden terms, exact labels | Fast, free, deterministic | Cannot judge meaning |
| Reference comparison | Required facts present, answer matches a known value | Objective for factual tasks | Needs references; brittle on wording |
| LLM-as-judge | Faithfulness, relevance, tone, rubric adherence | Scales to subjective criteria | Biased and must be validated |
| Human review | Anything, including subtle domain correctness | The most trustworthy | Slow and costly |

*Scoring methods for LLM outputs, cheapest first.*

Design the task so that more of it lands on the upper rows. If the application classifies, extracts or routes, the output can be compared exactly. If it must return structured data, validate the structure in code. Save judgement-based scoring for free-text quality.

For retrieval-based systems, score the two halves separately: whether the right passages were retrieved, and whether the answer is faithful to them. [How RAG works](https://thegccengineer.com/guides/rag-explained) covers the retrieval metrics.

## What is LLM-as-judge and where does it fail?

LLM-as-judge is the practice of using a language model to score another model's output against written criteria. It makes subjective evaluation affordable at scale, and it is useful only after you have checked that the judge agrees with human reviewers.

Known weaknesses to plan for:

- **Position and length bias.** When comparing two answers, judges can favour the first one shown or the longer one.
- **Leniency.** Judges tend to rate fluent text highly, including fluent text that is wrong.
- **Self-preference.** A judge may favour output written in the style of its own model family.
- **Vague criteria.** "Rate the quality from one to ten" produces noisy, drifting scores.
- **Drift.** Changing the judge model or its prompt changes every score, so past results are no longer comparable.

Practices that make a judge dependable: ask one narrow question per call with a yes or no answer, such as "is every claim in the answer supported by the context?"; give the judge the criteria and a few labelled examples; have it state its reasoning before the verdict; and measure its agreement with human labels on a sample before trusting it. Recheck that agreement whenever the judge changes.

> **A judge is a model too:** An unvalidated judge gives you numbers, not measurements. If the judge and human reviewers disagree often on a sample, fix the rubric or the judge prompt before using its scores to make decisions.

## What is the difference between offline and online evaluation?

Offline evaluation runs the application on a fixed dataset before release; online evaluation measures it on real traffic after release. Offline tells you whether a change is safe to ship. Online tells you whether it helped users.

**Offline and online evaluation**

**Offline**

- Golden set, run before release
- Fast, repeatable, comparable across versions
- Catches regressions early
- Limited to the cases you thought to include

**Online**

- Real users and real inputs, after release
- Signals: feedback, edits, retries, escalations, task completion
- Reveals inputs nobody anticipated
- Slower, noisier and needs careful experiment design

The two feed each other. Online monitoring surfaces failures and new kinds of input; those become golden set cases; the offline score then guards against them returning. Where traffic allows, an A/B test between the old and new version is the strongest evidence that an offline improvement is real.

## How do you set up regression testing for prompts?

Set up regression testing by treating prompts, model settings and retrieval configuration as code: keep them in version control, and run the golden set automatically whenever they change. A release goes ahead only if the scores have not dropped beyond an agreed margin.

```python
import json


def load_cases(path: str) -> list[dict]:
    with open(path, encoding="utf-8") as f:
        return [json.loads(line) for line in f if line.strip()]


def has_required_facts(case: dict, output: str) -> bool:
    return all(fact.lower() in output.lower() for fact in case["must_include"])


def pass_rate(cases: list[dict], answer) -> float:
    passed = sum(has_required_facts(case, answer(case["input"])) for case in cases)
    return passed / len(cases)


def test_no_regression(answer_fn, baseline_rate: float) -> None:
    cases = load_cases("golden.jsonl")
    assert pass_rate(cases, answer_fn) >= baseline_rate
```

In that sketch, `answer_fn` is your application and `baseline_rate` is the score of the version currently in production; both would be supplied as test fixtures. Three habits make the approach work. Compare against the previous version, not against perfection. Look at which cases changed, not only the total, because a flat score can hide five fixes and five new failures. And record the model version with every run, since a provider update can shift results with no change on your side.

Agents need the same discipline applied to whole tasks, with step counts and forbidden actions checked as well as the final answer; see [AI agents explained](https://thegccengineer.com/guides/ai-agents-explained).

## How do interviews test LLM evaluation?

Interviews test LLM evaluation by asking how you would know that a system works. After you describe any LLM project or design, expect "how did you measure quality?" and "what happened when you changed the prompt?". Candidates who answer with a golden set, a named metric and a specific regression they caught stand out, because most can only say the outputs looked reasonable.

Worked question patterns are in [GenAI interview questions on RAG, agents and evaluation](https://thegccengineer.com/guides/genai-interview-questions), and evaluation is weighted heavily in the [GenAI and LLM engineer role](https://thegccengineer.com/guides/genai-llm-engineer-role). An evaluation harness is also the single most persuasive part of an [AI portfolio project](https://thegccengineer.com/guides/ai-portfolio-projects).

To find out how your evaluation skills compare with what your target role expects, take the [free readiness check](https://thegccengineer.com/signup).

## Key takeaways

- Evaluate the application on your own task; public model benchmarks say little about your use case.
- A golden set of real, varied, hard examples is the foundation, and it grows from production failures.
- Prefer checks that code can verify; use an LLM judge only for what code cannot assess, and validate it against human labels.
- Rerun the evaluation on every prompt, model or retrieval change, because fixes in one place cause regressions in another.
- Offline evaluation predicts quality; online evaluation confirms it with real users.

## Frequently asked questions

### How many examples does a golden set need?

Enough to cover the main kinds of input and the known hard cases, and enough that one or two changed results do not swing the score. It is reasonable to start small, with carefully chosen examples, and grow the set over time. Coverage and quality of the examples matter more than raw count, and a small set you trust beats a large one you do not.

### Are metrics like BLEU and ROUGE useful for LLM applications?

Rarely. They measure word overlap with a reference text, which suits tasks with one correct wording, such as some translation work. LLM applications usually have many acceptable answers, so a correct reply phrased differently scores badly and a fluent wrong one can score well. Task-specific checks and rubric-based judging are more informative.

### How do you evaluate an LLM application when outputs are non-deterministic?

Fix what you can and measure the rest. Pin the model version and settings, then run each test case several times and report a pass rate instead of a single pass or fail. Assert on properties of the output, such as containing required facts or valid structure, instead of exact wording.

### Who should label the examples in a golden set?

Someone who knows what a good answer looks like for the task, ideally a domain expert, with written criteria. Two people labelling the same sample and comparing disagreements exposes vague criteria quickly. Engineers can label formats and obvious errors, but correctness in specialised fields such as finance or medicine needs expert review.

### What is the difference between evaluation and guardrails?

Evaluation measures quality before and after release, across many examples, to guide changes. Guardrails act at run time on a single request, blocking or altering an unsafe input or output. Evaluation tells you how often a problem occurs; guardrails limit the damage when it does. A production system needs both.