In this guide · 7 sections
Short answer
AI models predict likely-sounding text, so they can invent facts, figures, quotes and sources. Treat AI output as a draft: verify prices, legal, medical and policy statements against your own source, restrict customer chatbots to your own approved content, and give every chatbot a hand-over to a human.
What is an AI hallucination?
The OWASP project's LLM09:2025 Misinformation entry defines it well: hallucination is when a model 'generates content that seems accurate but is fabricated', because it fills gaps using statistical patterns without truly understanding the content. OWASP pairs it with a second risk, overreliance, where people trust the output without checking.
Language models are trained to produce the most likely next words, not to look facts up. When the answer is not well represented in what the model learned, or the question is ambiguous, it still produces fluent text. Newer models and web-search modes reduce the problem but do not remove it, and an answer that includes a source link can still misstate what that source says.
- Gaps in knowledge: your prices, opening hours and policies are not in the model unless you give them to it.
- Outdated training data: rules and prices that changed after training.
- Leading questions: asking 'which section of the Act says X' invites it to invent one.
- Long chains: maths, multi-step reasoning and summaries of long documents drift.
When AI wrong answers became a business problem
Two published decisions show who carries the risk when AI output is wrong.
- Moffatt v. Air Canada (2024). A customer asked the chatbot on Air Canada's website about bereavement fares and was told it could claim the reduced rate after travelling, within 90 days of the ticket being issued. The airline's own policy page said the policy did not apply once travel was completed. The British Columbia Civil Resolution Tribunal decided that Air Canada is responsible for all the information on its website, 'whether the information comes from a static page or a chatbot', and ordered it to pay C$812.02, including C$650.88 in damages.
- Fictitious cases in Singapore courts. In 2026] SGHC 49, the High Court ordered personal costs against two lawyers over citations it found likely to have been generated by an AI tool. The [judgment notes that the first reported Singapore case on fictitious authorities was Tajudin bin Gulam Rasul v Suriaya bte Haja Mohideen [2025] 5 SLR 518.
The lesson for any business is the same: saying 'the AI said it' does not move responsibility. If it is on your website, in your quote, or in a document you signed, it is yours.
Which business uses of AI are risky?
Hallucinations matter in proportion to what happens if nobody catches them.
| Use | Risk | Safer approach |
|---|---|---|
| Prices, fees, promotions | High: customers rely on quoted prices | Pull from your price list; never let AI calculate or invent them |
| Refund, warranty, delivery policies | High: Air Canada-type liability | Answer only from your published policy text |
| Legal, tax, regulatory answers | High: invented sections, wrong dates | Use AI to find sources, then read the source yourself |
| Health, medical, safety advice | High: harm to people | No AI answers to customers without professional review |
| Citations, statistics, quotes | High: often fabricated | Click every link and check the number is on the page |
| Marketing drafts, emails | Medium: wrong claims slip in | Human edit; check every factual claim |
| Summaries of your own documents | Medium: omissions and drift | Spot-check against the original |
| Brainstorming, rewording | Low | Fine to use freely |
Research tools that cite sources help, but they do not remove the need to open the source. Our Perplexity vs ChatGPT search guide compares how each handles citations.
How staff can check AI answers in two minutes
Most errors are caught by a short routine. Make it part of your ChatGPT at work policy so everyone uses the same habits.
Ask where it came from
Ask for sources, then open them. If a link is dead or does not say what the AI claims, treat the answer as unverified.
Check every number against a primary source
Prices, dates, penalties, percentages and section numbers: find each on the official page, not in another AI answer.
Give it your document instead of asking it to remember
Paste or upload the policy, price list or contract and ask it to answer only from that text.
Ask it to disagree with itself
A follow-up such as 'what in your answer might be wrong or outdated?' often surfaces weak points.
Decide who signs off
Anything going to a customer, regulator or court is reviewed by a named person who checked the facts.
Guardrails for a customer-facing AI chatbot
A website chatbot is where hallucination turns into liability fastest, because it speaks for you without a person in between. OWASP's mitigations include retrieving answers from verified sources, human oversight for critical information, and clearly telling users about the limits. In practice, for a small business, that means:
Chatbot guardrail checklist
Our AI chatbot for your website guide covers platform options and costs. Do not connect a chatbot to tools that can issue refunds, change bookings or send emails without approval; that adds a different risk, prompt injection, on top of wrong answers.
Do it yourself or get help?
For internal use, the verification steps above and a one-page policy go a long way, and they cost nothing. A short team session on checking AI answers is worth more than any tool.
If you want a customer chatbot with these guardrails built in, IT Sorted's AI Website Assistant (S$790) answers only from your website, FAQ and documents, captures leads, hands over to WhatsApp or email, and includes a usage report after 30 days so you can see what it got wrong. Security Awareness Training (S$390) gives staff short lessons, which can include safe AI use alongside scam awareness.
Frequently asked questions
It is when an AI tool gives an answer that sounds right but is made up, such as a fake statistic, a policy that does not exist, or a court case that was never decided. It happens because language models generate likely-sounding text rather than looking facts up.
You should assume so. In Moffatt v. Air Canada, a Canadian tribunal held the airline responsible for wrong refund advice from its website chatbot, saying it made no difference whether information came from a static page or a chatbot. Singapore law may differ in detail, but the safe position is that your chatbot speaks for you.
Give the AI your own documents and ask it to answer only from them, ask for sources and open them, verify every number on an official page, and keep a person in the loop for anything sent to customers. For chatbots, restrict answers to approved content and add a hand-over to a human.
Yes, less often for common questions, but no current model is free of it. Errors remain more likely for niche topics, recent changes, exact numbers and citations. Web search features help but can still misstate what a source says, so checking the source remains necessary.
Low-stakes tasks where errors are obvious and harmless: brainstorming ideas, rewording your own text, drafting outlines, or changing tone. Anything with facts, figures, legal or health content, prices or commitments to customers needs checking by a person.