An AI hallucination is a convincing-sounding output from a language model that is false or unsupported, such as an invented quotation, a wrong figure or a source that does not exist. They cannot be ruled out, but they can be limited: through answers from checked documents, source citations, review steps and approvals. This article shows how your business protects itself.
Why does an AI hallucinate?
A language model predicts text piece by piece and produces what is linguistically likely to follow. It has no fact database and does not check statements against reality. If it lacks knowledge, it often fills the gap with something that fits. Other triggers are unclear instructions, outdated knowledge, very long texts and the impression that it must answer at all costs. How language models work is covered in our explainer on large language models. An overview of the research is offered by Ji et al. and the article on hallucination in AI.
What can hallucinations do in a business?
Two well-known cases show the consequences. In a Canadian proceeding (Moffatt v. Air Canada, 2024), a tribunal ruled that the airline had to answer for wrong information its website chatbot gave about special fares. In the United States, a court in 2023 sanctioned lawyers who had filed a brief citing court decisions that a language model had invented (Mata v. Avianca).
| Use | Risk | Protective measure |
|---|---|---|
| Internal text draft | Low if a person checks | Author checks facts, figures and names |
| Customer communication | Medium: wrong promises, figures | Approval by a person before sending |
| Customer chatbot giving information | High: prices, terms, deadlines | Answer only from checked documents, cite sources, hand over to people |
| Legal, tax, health information | Very high | Do not automate; involve specialists |
How do you protect your business?
The most effective lever is to have the model answer from checked sources. In the retrieval-augmented generation (RAG) approach, the system first looks up matching passages in your documents and then formulates the answer from them (Lewis et al.). It asks for source citations so that you can verify them. How this is built is explained in What is RAG?.
Add rules to the instruction: answer only from the documents supplied, name the passage, say "I don't know" if details are missing. Limit a chatbot's subject area and set up a hand-over to people. Check figures, names, quotations and references to laws against original sources before you publish anything. For customer chats, regularly reviewing conversation logs is worthwhile. Read more on the website chatbot in AI chatbot for your website.
How do people stay in control?
Decide by risk who checks what. The author checks internal drafts, a second person checks customer communication and specialists check sensitive information. Document who approved what and when. How humans and AI work together is described in What is human in the loop?.
- Uses classified by risk: internal, customer communication, information
- Answers from checked documents with source citation set up
- Instruction contains the rule "if details are missing, ask or flag it"
- Figures, names, quotations and references to laws checked before publication
- Approval and hand-over to people set and documented
- Conversation logs of customer chats reviewed regularly
- Staff informed about the limits of the tools
Conclusion: benefit with control
Hallucinations are part of the technology and cannot be switched off, but they can be managed. Have models answer from checked sources, ask for evidence and check what goes outside. If you want to build an assistant with sources and approvals, see our AI automation service or describe your project.
Sources
- Ji et al.: Survey of Hallucination in Natural Language Generation (arXiv)
- Lewis et al.: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (arXiv)
- Wikipedia: Hallucination (artificial intelligence)
- Civil Resolution Tribunal of British Columbia: Moffatt v. Air Canada, 2024 BCCRT 149
