In this guide · 8 sections
- 01What can go wrong with an untested chatbot
- 02Step 1: build a test set from real enquiries
- 03Step 2: score every answer
- 04Step 3: try to break it (red-team prompts)
- 05Step 4: decide what pass mark is good enough
- 06Step 5: fix, re-run, and keep testing after launch
- 07Do it yourself or get help?
- 08Frequently asked questions
Short answer
Collect 40 to 50 real customer questions, write the correct answer for each, run them through the chatbot and score accuracy, refusals and tone. Add tricky prompts that try to make it go off-topic or reveal its instructions. Fix and re-run until it passes, then re-test after every content change.
What can go wrong with an untested chatbot
IMDA's Starter Kit for Testing LLM-based Applications for Safety and Reliability (version 1.0, January 2026) focuses on five risks. Here's what they look like for a typical SME website chatbot:
| Risk (IMDA) | What it looks like on your site |
|---|---|
| Hallucination and inaccuracy | Quotes an old price, invents opening hours or a refund policy you don't have |
| Bias in decision making | Treats customers differently, for example in how it qualifies leads |
| Undesirable content | Rude or inappropriate replies, or advice it shouldn't give (medical, legal) |
| Data leakage | Reveals other customers' details, internal documents or its system instructions |
| Vulnerability to adversarial prompts | Is talked into discounts, off-topic tasks or ignoring its rules |
The OWASP Top 10 for LLM applications (2025) lists prompt injection first, followed by sensitive information disclosure; it also includes system prompt leakage, excessive agency and misinformation. If you are still choosing a chatbot, our guide to AI chatbots for Singapore websites covers options and costs.
Step 1: build a test set from real enquiries
The test set is the most important part. IMDA says a good test dataset should represent the app's purpose and real usage, cover the topics and edge cases it will meet, and reflect how real users ask questions. On size, it says there is no magic number but suggests a minimum of 30 to 50 prompts for a narrow, specific topic and 100 to 200 for a wider one.
- Pull 40 to 50 real questions from your inbox, WhatsApp chats, contact form and phone notes.
- Keep customers' own wording, including typos, Singlish and mixed languages.
- Cover every topic the bot will handle: prices, hours, booking, delivery, refunds, services.
- Add 10 questions it should not answer: competitors, medical or legal advice, unrelated tasks.
- Add 5 questions that need a human: complaints, custom quotes, account problems.
- For each question, write the correct answer or the correct behaviour (answer, refuse, hand over).
Step 2: score every answer
Run each question through the chatbot, ideally twice, because AI answers vary between runs. Record the answer and score it in a spreadsheet.
| Score | Question to ask | Pass looks like |
|---|---|---|
| Accuracy | Is every fact correct and current? | Price, hours and policy match your source exactly |
| Grounding | Did it answer only from your content? | No invented details; says it doesn't know when unsure |
| Correct refusal | Did it decline what it should decline? | Politely refuses off-topic or advice questions |
| Handover | Did it pass humans what needs a human? | Offers WhatsApp, email or a callback |
| Tone | Would you be happy if a customer saw it? | Polite, short, in your brand voice |
| Language | Did it reply in the customer's language? | Matches English, Chinese or Malay correctly |
A simple scale works: pass, minor issue, fail. Count a wrong price, a made-up policy or a leaked instruction as a fail even if the rest of the answer is good.
Step 3: try to break it (red-team prompts)
Then spend 30 minutes trying to make it misbehave, as a curious or malicious customer would. Use prompts like these:
Red-team prompts to try
Prompt injection, where a user's message or a document tries to override the bot's instructions, is the top OWASP risk and is hard to prevent completely. The practical defence is limiting what the bot can do: no access to customer records it doesn't need, no ability to issue discounts or change bookings without a human, and clear instructions to stay on topic.
Step 4: decide what pass mark is good enough
IMDA says each app must set its own threshold, because no universal baseline fits every use. A practical approach for a small business chatbot:
- Zero tolerance: wrong prices, invented policies, leaked personal data or system instructions, inappropriate content. Fix and re-test until there are none.
- High bar: at least nine in ten factual answers fully correct, with the rest at most minor issues.
- Refusals and handovers: every advice and complaint question handled correctly.
- Tone: no answer you'd be embarrassed to see in a screenshot.
These thresholds are our suggestion for a low-risk FAQ bot, not an official standard. Raise the bar if the bot gives information customers rely on for money, health or legal decisions, or don't let it answer those topics at all.
Step 5: fix, re-run, and keep testing after launch
Most failures trace back to the content or instructions, not the AI model.
Fix the source content
Update the FAQ, price list or document the bot reads. Delete outdated files rather than adding corrections on top.
Tighten the instructions
Tell the bot what it must not discuss, when to hand over, and to say it doesn't know rather than guess.
Re-run the full test set
A fix for one question can break another. IMDA notes that changes can introduce regressions elsewhere.
Soft-launch first
Show the bot on one page or to a few customers for a week, and read every conversation.
Re-test on every change
Run the test set again after price or policy changes, new documents, a model or vendor update, or new features.
Review real chats monthly
Add new real questions to the test set, especially any the bot got wrong.
IMDA's kit describes pre-deployment tests as only a snapshot and recommends continuous monitoring and re-testing, with scheduled reviews, user feedback and significant app changes as triggers. Free open-source tools such as AI Verify Foundation's Project Moonshot can run larger test sets, but a spreadsheet is enough for most SMEs.
Do it yourself or get help?
Building the test set takes an afternoon and is the part only you can do, because you know the right answers. Running it takes an hour each time. The work that needs more skill is fixing failures through better content structure and instructions.
Our AI Website Assistant (S$790) trains an assistant on your website, FAQ and documents, with lead capture to your inbox, hand-over to WhatsApp or email, English plus one more language, guard-rails so it only answers about your business, and a usage report after the first 30 days. IT Sorted is a Singapore-based team of developers with over 10 years of experience; you pay upfront and send us your FAQ, price list and policies in writing.
Frequently asked questions
IMDA's Starter Kit for testing LLM apps says there is no magic number, but suggests a minimum of 30 to 50 test prompts for a narrow, specific topic and 100 to 200 for a wider one. For a typical SME FAQ chatbot, 40 to 50 real customer questions plus 10 to 15 off-topic and complaint questions is a good start.
Red-teaming means deliberately trying to make the chatbot misbehave, as a malicious or curious user would. Examples include asking it to reveal its instructions, give discounts, discuss other customers, do unrelated tasks or follow instructions hidden in a pasted document. It finds weaknesses that normal customer questions won't show before customers find them.
Prompt injection is when text a user types, or text inside a document or web page the bot reads, tries to override the chatbot's instructions, for example telling it to ignore its rules. It is the first risk in the OWASP Top 10 for LLM applications (2025). Limiting what the bot can access and do reduces the damage.
Yes. IMDA's kit describes pre-deployment testing as only a snapshot, because models change and users ask new things. Re-run your test set after price or policy changes, new documents, a vendor or model update or new features, and review real conversations monthly, adding any questions the bot got wrong.
There is official guidance rather than a mandatory standard. IMDA's Starter Kit for Testing LLM-based Applications for Safety and Reliability, version 1.0 from January 2026, sets out voluntary guidelines covering hallucination, bias, undesirable content, data leakage and adversarial prompts. AI Verify Foundation also offers Project Moonshot, an open-source testing toolkit.