Hallucination is the failure that makes AI risky in a business. The output is fluent, plausible and wrong — which means the ordinary signals people use to detect a mistake do not fire. A person reading a badly written answer is alert; a person reading a confident, well-formed one is not.

This guide covers what hallucination is, where it appears most, the verification steps that catch it, and how to record them so the control is real. It is part of the Governance, Risk & Data pillar.

Fluency is not evidence. The check that matters is whether someone opened the source.

What hallucination actually is

A large language model generates text that is statistically plausible given its input. It does not retrieve verified facts and it does not check itself. When it produces a fact, a figure or a citation, it is producing something that fits — which may or may not be true.

That mechanism explains the pattern of failures:

  • Fabricated citations. A reference that reads exactly like a real one, with a plausible author, journal and year.
  • Fabricated figures. A number in the right range, formatted correctly, sourced to nothing.
  • Confident synthesis. Two real facts combined into a third that is false.
  • Out-of-date claims. Correct at the model’s training point, and no longer correct.
  • Invented quotations. A paraphrase presented as a direct quote, with a real person’s name attached.

Every one of these is fluent. None of them announces itself.

How common it is

The evidence is broad rather than precise, and the imprecision is itself informative. Reported hallucination rates in large language models range widely — roughly 22% to 94% depending on the task and the measurement method — with one benchmark finding 13.6% of responses grounded (Stanford HAI, AI Index 2026, with Vectara’s hallucination leaderboard and OpenAI model documentation). Published evaluations have also reported model-specific rates around 33% and 48% on particular benchmarks.

The spread reflects genuine difficulty in measuring the phenomenon, and it should be read as follows: no model output can be treated as verified without a check, and no benchmark number should be used to justify skipping one.

Where it hurts most

Hallucination is unevenly dangerous. The risk is highest where three conditions coincide.

  • The output is fluent and structured, so nothing looks wrong.
  • The reader cannot check it cheaply, so they accept it.
  • The consequence of error is high — a client, a filing, a decision.

Research, citations and comparative claims sit in all three. Figures presented as facts sit in all three. Quotations and references sit in all three. Summaries of your own documents are far safer, because the source is in front of you.

The verification steps that catch it

Four checks catch the overwhelming majority of hallucinated content.

1. Open every citation. Confirm the source exists, says what it is claimed to say, and is current. A citation that cannot be opened is removed, not reworded.

2. Trace every figure to a named source. If a number cannot be traced to your system of record or an identified external source, it does not go in.

3. Check quotations verbatim. A quote must match the source word for word, and the source must be real. Paraphrase presented as a quote is a common and damaging failure.

4. Ask what would change the claim. For any substantive assertion, ask which source supports it. Where the answer is vague, the claim is a synthesis rather than a fact.

The fourth check is the fastest filter and the one most resistant to being gamed, because it asks the author to locate evidence rather than to confirm an impression.

Making verification a habit

Design beats intention. Four design choices turn the check into something that happens.

  • Put the check in the workflow, as a named step with an owner, not as a value statement.
  • Give it a place to be recorded — a tick, a version note, an approval in the system.
  • Practise it in training, on outputs containing a seeded fabricated citation and a plausible wrong figure.
  • Thank the person who catches one. The behavior you want repeated is the behavior you recognize.

The counter-productive approach is to warn staff about hallucination without giving them the drill. People remember a standard they have practised and forget a risk they have been told about.

A worked catch

A consultancy drafts a market briefing with AI assistance, using the bounded-source workflow.

  • The draft cites a named industry survey reporting that “68% of mid-market firms now use AI in their finance function”, with a publication and a year.
  • The check requires the researcher to open the citation. The publication exists, and the year is right, but the survey says 48%, and it is about a different group of firms entirely.
  • The response removes the claim rather than correcting the number, because the source does not support the sentence the briefing needed. The researcher finds a second source that does support a narrower claim, and cites that instead.
  • The record notes the correction and its cause, which was a fabricated figure attached to a real citation — the most difficult form of hallucination to spot, because the reference checks out at a glance.

Every hallucination caught this way costs minutes. The same claim, unchecked, would have gone to a client with a source attached, which makes it far more damaging than a figure with no source at all.

Common mistakes

  • Trusting fluency. A well-written answer feels checked; it is not.
  • Softening an unsupported claim instead of removing it. “Widely regarded as” is still unverified.
  • Accepting a citation that was never opened. The most expensive error in the category.
  • Verifying without recording. The control exists as practice and not as evidence.
  • Treating one model’s benchmark as a guarantee. Rates vary by task, by prompt and by measurement method, so no single number describes your exposure.
  • Warning without practising. The check must be a drill, not a lecture, because the behavior required is a habit rather than a piece of knowledge.

Frequently asked questions

What is an AI hallucination?

Output that is fluent and plausible but false — a fabricated citation, figure, quotation or claim — produced because a model generates statistically plausible text rather than retrieving verified facts.

How common are AI hallucinations?

Reported rates range widely, roughly 22% to 94% depending on the task and the measurement method, with one benchmark finding 13.6% of responses grounded (Stanford HAI, AI Index 2026). The spread reflects measurement difficulty and supports the same conclusion: verification is not optional.

How do we catch hallucinations?

Open every citation, trace every figure to a named source, check quotations verbatim, and ask what evidence supports each substantive claim. Where a source cannot be found, remove the claim.

Should we stop using AI because of hallucination?

No. Treat output as a draft, keep facts in your systems rather than the model, and require a human check on anything that leaves the team. That is how the risk is managed without losing the benefit.

Can we tell when AI is hallucinating?

Not reliably from the text alone, which is the whole difficulty. The reliable method is external verification against a source, not an assessment of how the answer reads.

Does a better model remove the risk?

It reduces the rate. It does not remove the need for verification on output that reaches a client, a filing or a decision — and the check is cheap relative to the cost of being wrong.

What about AI tools with built-in citations?

They reduce fabrication, and they do not eliminate it. The rule is unchanged: open the source. A generated citation is a lead, and a lead becomes evidence only when a person has confirmed it.

How do we handle an AI error we only discover later?

Treat it as an incident: establish what was affected, correct it, disclose proportionate to the impact, and fix the cause in the workflow. The later an error is found, the more the process rather than the person matters.

What about AI-generated images, charts or diagrams?

The same applies. A generated visual can present a figure or a relationship that does not exist, and it carries the extra risk of looking authoritative to a reader who would question a sentence.

Next step

Adopt the four checks, put them in the workflow with a named verifier, and record that the check happened. Use the Human-Verification Checklist, see Training Staff to Verify AI Output, or book an AI adoption call to install the check in your process.

Sources

  • Stanford HAI, AI Index 2026, with Vectara’s hallucination leaderboard and OpenAI model documentation (2025–2026): hallucination rates reported between roughly 22% and 94% depending on task and measurement, with one benchmark finding 13.6% of responses grounded; model-specific rates of around 33% and 48% on named benchmarks.

Figures are cited from their sources and dated. Where a source is a vendor benchmark, the sample size is stated where published.