CyberFreezeDevMoney and compliance
Invented tax rates, deadlines or totals lead to filing errors and penalties.
Read more about Money and compliance
Invented tax rates, deadlines or invoice totals cause filing errors and penalties.
- Home
- AI glossary
- Hallucination (AI)
What is AI hallucination?
A fluent, confident answer from an AI model that is simply made up.
The plain definition
A hallucination is output not backed by the input or by reality, and it reads just as confidently as a correct answer.
Read more about The plain definition
A hallucination is when a model outputs something that is not supported by its input or by reality: a fake case citation, a wrong GST rate, a product feature that does not exist, a total that does not add up. The model gives no signal that it is guessing; it reads just as confidently as a correct answer.
A well-known case: in Mata v. Avianca (US District Court, Southern District of New York, 2023), lawyers were sanctioned after filing a brief with court decisions that ChatGPT had invented.
Why it happens
-
01 Prediction, not lookup
A model predicts the likeliest next token; nothing in it checks facts.
Read more about Prediction, not lookup
A large language model generates the most probable next token. It has no built-in step that checks facts.
-
02 Gaps in knowledge
Where training data was thin, the model fills the gap with something plausible.
Read more about Gaps in knowledge
When the answer was rare or absent in training data, the model fills the gap with something that fits the pattern.
-
03 Pressure to answer
Prompts with no room for "I don't know" push the model to invent an answer.
Read more about Pressure to answer
Prompts that demand an answer, with no permitted "I don't know", push the model to invent one.
-
04 Missing context
If retrieval returns the wrong passages, the model may answer from memory.
Read more about Missing context
If retrieval returns the wrong passages, the model may answer from memory instead.
Where it matters in business
Customer promises
A bot that invents a refund policy creates an obligation or a complaint.
Read more about Customer promises
A support bot that invents a refund policy creates an obligation or a complaint.
Legal and medical text
Citations, dosages and clauses must be checked against the source every time.
Read more about Legal and medical text
Citations, dosages and clauses must be checked against the source every time.
A practical example
An extractor invented GSTINs on blurry scans. Allowing blank fields, checking every GSTIN in code and sending failures to a person kept guesses out of the books.
Read more about A practical example
An invoice extraction system sometimes filled in a GSTIN when the scan was blurry, producing a valid-looking but wrong number. The fixes: the prompt allows null for unreadable fields; code checks every GSTIN against its format and checksum; any invoice with a null or failed check goes to a person for review before it reaches the accounts. The model still guesses occasionally, but the guess can no longer reach the books.
How to reduce it
Ground answers in sources
Use RAG and require citations to the passages behind each answer.
Read more about Ground answers in sources
Use RAG and require citations to the passages used.
Validate in code
Check totals, dates, IDs and formats with rules, not with the model.
Read more about Validate in code
Totals, dates, IDs and formats can be checked by rules. Do not ask a model to be its own calculator.
Keep a person in the loop
Have people review high-stakes outputs and measure error rates with evaluation.
Read more about Keep a person in the loop
For high-stakes outputs, use human-in-the-loop review and measure error rates with evaluation.
Questions
Can hallucination be eliminated completely?
No current method removes it entirely. You reduce how often it happens and design the system so a wrong output is caught before it causes harm.
Does a lower temperature stop hallucination?
A low temperature makes output more consistent, which helps with extraction tasks, but a model can be consistently wrong. Grounding and validation matter more.
Last updated