Hallucination
Also known as: confabulation, fabrication
A hallucination is model output that is fluent and confident but not supported by the input or by fact — an invented citation, a non-existent API method, a fabricated figure. It is a property of how generative models work, not a bug to be patched out, so systems must be designed to contain it.
Last reviewed · Part of the Architecture Glossary
In practice
Separate the two causes before choosing a fix:
- Ungrounded generation — nothing relevant was in the context, so the model filled the gap. The fix is retrieval, plus an explicit instruction and an easy escape hatch ("if the context does not contain the answer, say so").
- Unfaithful generation — the right context was present and the answer still contradicts it. The fix is the generation stage: a stronger model, a citation requirement, or a verification pass.
Containment patterns that work in production:
- Force citations. Require every claim to reference a retrieved chunk ID, then verify the IDs exist. Unverifiable claims get stripped, not shipped.
- Constrain the output space. Structured output and enums cannot hallucinate a category that is not in the schema.
- Verify externally. Compile the code, run the query against a real schema, check the URL resolves. Any deterministic check beats asking the model whether it was right.
- Make the confidence visible. Showing sources lets a human catch what automation missed.
When it matters
Anywhere output is used without review: customer-facing answers, automated actions, generated code that merges.
Common mistake
Adding "do not hallucinate" to the system prompt and treating the problem as addressed. Measure it with an eval harness and a faithfulness score; a prompt instruction is not a control.
See also
- RAG (Retrieval-Augmented Generation)RAG is the pattern of retrieving relevant documents at query time and placing them in the model's context so the answer is grounded in your data rather than in the model's parameters.
- Eval HarnessAn eval harness is the automated test suite for a non-deterministic system: a fixed dataset of inputs, a scoring method per case, and a scoreboard run on every change.
- Prompt InjectionPrompt injection is an attack in which instructions embedded in content the model processes — a web page, a document, a ticket, a tool result — are followed as if they came from the operator or user.