In this guide · 8 sections
- 01What IMDA announced on 14 October 2026
- 02Why testing an AI agent is different from testing a chatbot
- 03The four steps of agentic testing in plain English
- 04New metrics: tool use accuracy, handoffs and attack success
- 05A cut-down version for a small business agent
- 06AI Tester Accreditation: when to use an outside tester
- 07Do it yourself or get help?
- 08Frequently asked questions
Short answer
IMDA's Agentic Testing Starter Kit (v1.0, October 2026) is voluntary guidance with a four-step method: identify risks, build realistic tests, run tests and analyse logs, then monitor and re-test. Its main message is to test what the agent did, not just its final answer.
What IMDA announced on 14 October 2026
IMDA's press release of 14 October 2026 announced three initiatives at the AI Assurance Exchange, an event run by the AI Verify Foundation and SGTech:
| Initiative | What it is | Who it is for |
|---|---|---|
| Agentic Testing Starter Kit | Guidance on how to test AI agents for safety and reliability, with case studies | Any organisation deploying or testing agents |
| AI Tester Accreditation Programme (AI TAP) | Accreditation for third-party AI testing firms; first five firms named | Businesses that want an outside tester |
| FAIRS taskforce | Frontier AI Resilience and Security Taskforce for the infocomm and media sector; report due by first half of 2027 | Telcos, media and their suppliers |
The kit itself is a 50-page PDF, Version 1.0, October 2026. IMDA calls it a living document and is collecting feedback, so expect updates.
Why testing an AI agent is different from testing a chatbot
A chatbot produces text. An agent takes actions. The kit's own example: a customer service agent can issue refunds, not just answer questions. So a mistake is no longer a wrong sentence on screen, it can be money leaving your account or a record being changed.
The kit lists three things that change when you test an agent:
- Test actions, not just output. If the task was to send an email and book a meeting, check both actually happened correctly, even if the final reply says "done".
- Test boundaries. Agents try to finish the task and may find workarounds around your controls. Check that the limits you set really hold.
- Test approvals and handoffs. If a human must approve certain steps, check the agent asks, and that the human gets enough information to decide well.
This builds on IMDA's earlier Model AI Governance Framework for Agentic AI and the starter kit for LLM apps. Our IMDA AI governance guide for SMEs covers those frameworks.
The four steps of agentic testing in plain English
The kit's method has four steps. Here is what each means for an ordinary business deployment.
1. Identify the relevant risks
List what could go wrong: erroneous actions, unauthorised actions, biased or unfair actions, data breaches and disruption to connected systems. Turn each into a concrete scenario for your use case, such as "refunds a customer twice", and rank them by likelihood and impact.
2. Create realistic tests
Start in a contained environment (test accounts, sandbox data) and move closer to production as confidence grows. Turn on logging that captures every tool call and state change. Write realistic tasks: the kit notes agents can behave differently when test data looks fake, such as names like "John Doe".
3. Run tests and analyse the logs
Run each task several times, because agents do not behave the same way every time. Read the logs, not only the results. An agent can reach the right outcome while breaking a rule on the way.
4. Monitor and re-test
Keep monitoring after launch. Re-test on a schedule, and always after a change to the prompt, the model, or the tools the agent can use. Add every real incident to a regression set so it cannot quietly come back.
New metrics: tool use accuracy, handoffs and attack success
The kit suggests metrics that do not exist for plain chatbots. You do not need all of them, but these are worth knowing:
| Metric | What it checks |
|---|---|
| Tool use accuracy | Right tool chosen, right inputs given, output read correctly, tools called in the right order |
| Handoff accuracy | When one agent passes work to another, the details are complete and correct |
| Human-in-the-loop rate | How often the agent asks a person, and whether it asks at the right moments |
| Attack success rate | How often a malicious prompt or tampered tool makes the agent act unsafely |
| Goal hijacking rate | How often an attack changes what the agent is trying to do |
Attack metrics matter because agents read untrusted input: emails, web pages and uploaded files. The kit asks you to map where attacker-controlled text can enter and what the maximum damage would be if one component were compromised.
A cut-down version for a small business agent
The kit is written for enterprises and testers, but its logic scales down. If you are deploying one agent with a handful of tools, this is a reasonable minimum before go-live. It is our condensed reading of the kit, not an IMDA checklist.
Before you switch an AI agent on
For costs and realistic use cases, see our guide to what AI agents cost a small business.
AI Tester Accreditation: when to use an outside tester
The AI Verify Foundation's AI TAP page says accredited firms go through technical, operational and financial evaluations. IMDA named the first five: AIDX Tech, Knovel Engineering, PwC Singapore, Resaro and Vulcan. Accreditation lasts one year, so check the current list before you rely on it.
For a small internal agent, a structured self-test along the lines above is usually proportionate. An accredited tester makes more sense when an agent handles customer money or sensitive personal data, when a client or regulator asks for independent evidence, or when you are selling an AI product to larger companies.
Do it yourself or get help?
If the agent is a vendor product with limited tools, you can run the cut-down test yourself in an afternoon: write the tasks, run them, read the logs, and keep the results. If it is a custom agent with access to your systems, the harder parts are setting up a safe test environment and useful logging.
We build and test website assistants as an AI Website Assistant package (S$790), and scope custom agents and their test harness as a fixed-price Custom Tool or Dashboard project (from S$1,990).
Frequently asked questions
No. It is voluntary guidance published by IMDA in October 2026. It builds on the Model AI Governance Framework for Agentic AI, which is also voluntary. Sector regulators may set their own rules, but the kit itself creates no legal obligation. It is still useful evidence that you took reasonable care.
The trajectory is the full sequence of what an agent did on a task: its reasoning, every tool call, inputs and outputs, approvals requested and changes made. IMDA's kit says to test the trajectory, because an agent can produce a correct final answer while taking a wrong or unauthorised action along the way.
IMDA's 14 October 2026 press release named AIDX Tech, Knovel Engineering, PwC Singapore, Resaro and Vulcan as the first five firms accredited under the AI Verify Foundation's AI Tester Accreditation Programme. Accreditation lasts one year, so check the foundation's website for the current list.
The kit recommends scheduled re-tests or spot checks, plus a re-test after significant changes such as a new system prompt, a model upgrade, or new tools and features. Keep a regression set of past failures and incidents and run it each time, so known problems do not return.
For chatbots that only generate text, IMDA's earlier Starter Kit for Testing LLM-based Applications is the better fit. It covers hallucination, bias, undesirable content, data leakage and adversarial prompts. The agentic kit adds the extra tests needed once the system can take actions.