Eval Harness
Also known as: LLM evals, evaluation suite
An eval harness is the automated test suite for a non-deterministic system: a fixed dataset of inputs, a scoring method per case, and a scoreboard run on every change. It is what turns prompt and model changes from vibes into a measurable regression signal.
Last reviewed · Part of the Architecture Glossary
In practice
Scoring methods, cheapest and most reliable first:
| Method | Use for | Caveat |
|---|---|---|
| Exact match / regex | Extraction, classification, JSON fields | Only for deterministic outputs |
| Programmatic assertion | Schema validity, tool called, code compiles | Cheap, deterministic, underused |
| Retrieval metrics | recall@k, MRR | Isolates the retrieval half |
| LLM-as-judge | Open-ended quality, faithfulness | Needs its own calibration against human labels |
| Human review | The final arbiter | Expensive; reserve for the judge's calibration set |
Build the dataset from production failures. Every bug report becomes a case; the suite grows into a map of exactly how your system fails. Fifty real cases beat a thousand synthetic ones.
Report per-category pass rates rather than one aggregate. A change that lifts the mean while breaking every date-arithmetic case is a regression the average hides.
When it matters
Before the second prompt change, and certainly before any model upgrade — the harness is what makes "should we move to the new model" a one-hour question instead of a quarter-long argument.
Common mistake
Running evals manually before releases. If it is not in CI with a threshold, it decays into a script nobody has run since the person who wrote it changed teams.
See also
- HallucinationA hallucination is model output that is fluent and confident but not supported by the input or by fact — an invented citation, a non-existent API method, a fabricated figure.
- RAG (Retrieval-Augmented Generation)RAG is the pattern of retrieving relevant documents at query time and placing them in the model's context so the answer is grounded in your data rather than in the model's parameters.
- Fitness FunctionAn architectural fitness function is an automated, objective test of a non-functional requirement — coupling, latency, security posture, cost — run continuously in CI.