How to Test an AI Chatbot Before and After Launch

Why AI Chatbots Need a Different Kind of Testing

Classic software testing compares an output with an exact expected value. A button click must open a page; a function must return 42. An AI chatbot does not work like that. It writes a new reply each time, so the same question can get two answers that use different words and are both correct. A test that checks for an exact sentence fails for no reason, and a test that checks nothing misses real errors.

Three more things make chatbot testing different:

• Visitors use any wording. "Can I send this back?", "return policy", and "wrong size, what now" are the same question. One test phrase does not prove the others work.
• Small changes have wide effects. A new line in the instructions or a new FAQ page can change answers to questions that seem unrelated. This is why you need regression tests, not only a launch check.
• Wrong answers sound confident. The NIST Generative AI Profile (NIST AI 600-1, 2024) calls this risk "confabulation": content that is wrong but stated with confidence. A reader cannot tell it from a correct answer by its tone.

Wrong answers also have consequences. In Moffatt v. Air Canada (BC Civil Resolution Tribunal, 2024), the tribunal held the airline responsible for incorrect bereavement-fare information that its website chatbot gave a customer. Your chatbot speaks for your business. If you want to know why bots make things up in the first place, read our guide to preventing AI chatbot hallucinations. This article is about how to catch the errors that still get through.

What to Test: Seven Areas

A useful test set covers more than product facts. Use this table to make sure each area has at least a few tests.

The last row matters more than many teams expect. Prompt injection, where a user writes text that tries to replace the chatbot's instructions, is the first risk in the OWASP Top 10 for LLM Applications (2025). You cannot list every attack, but a few tests catch the obvious ones: invented discount codes, promises your policy does not allow, and requests to reveal the system prompt.

How to Write Chatbot Test Cases That Work

A chatbot test case has two parts: what the visitor says, and what a correct reply must do. The second part is where most test sets go wrong.

Describe behavior, not exact words

Write the expected result as a rule a person could check. Compare these two:

The first version fails whenever the bot uses different words. The second version passes any correct reply and fails the replies that matter: wrong facts, missing links, and invented offers.

Include "must not" rules

Many costly errors are things the bot added, not things it left out. Add a "does not" clause for the risks in your business: no medical or legal advice, no price promises, no delivery dates the carrier did not give, no confirmation that an order exists before the customer is verified.

Start from real questions

The best test questions come from your inbox, your support email, and your chat history, in the words customers actually use. Questions your team writes from memory tend to be too clean. Include typos, short fragments, and questions in other languages if you serve those markets.

Test conversations, not only single questions

Real chats have follow-ups. Write some tests with two to five messages, for example a visitor who names a product in the first message and asks about its warranty in the third. These tests catch bots that lose context or answer about the wrong product.

A Step-by-Step Chatbot Testing Process

This process works with any chatbot platform.

1. 1

   Collect your must-answer questions

   Start with 20 to 30: your most common questions, your most expensive mistakes, and a few from each area in the table above.

2. 2

   Write the expected behavior for each

   Use the rules above. Ask the person who owns each policy to approve the expectation, so the test checks the real policy and not a guess.

3. 3

   Run a baseline

   Run every test and record what passes and fails. A failure is not always the bot's fault; often the answer is missing from your content, or two pages disagree.

4. 4

   Fix the source, not the symptom

   Add or correct the content, clarify the instructions, or fix the tool. Then run the test again. Our guide on building a knowledge base your chatbot can use covers the content side.

5. 5

   Re-run the full set after every change

   This is regression testing. A fix for one answer can break another, so run all tests, not only the one you fixed.

6. 6

   Launch, then turn real mistakes into new tests

   Each wrong answer you find in live chats becomes a test. Over time, your test set reflects your real customers instead of your first guesses.

Testing Tools and Actions Without Side Effects

Modern chatbots do more than answer. They look up orders, book meetings, and create tickets. These tools need tests too, but a careless test can cancel a real order or send an email to a real customer. Split tools into two groups:

• Read tools (order status, product search, open time slots) are usually safe to run live. The risk is that live data changes, so a test that passed yesterday can fail today for no real reason. Use a fixed, fake reply (a mock) when you need a stable result.
• Write tools (cancel, refund, book, send) must never run in an automatic test. Block them or point them at a sandbox account, and check only that the bot chose the right tool with the right details.

For tool tests, check three things: the bot called the tool when it should, it did not call it when it should not (for example before the customer is verified), and it explained the result correctly.

Testing After Launch: Review Real Conversations

A test set only checks the questions you thought of. Real visitors ask the others. Plan a regular review of live conversations, and look for these signals:

• Unresolved conversations. The visitor left without an answer, or came back later with the same question.
• Handoffs for missing knowledge. A handoff because the visitor asked for a person is fine. A handoff because the bot had no answer is a content gap.
• Negative feedback. Thumbs-down ratings and angry messages point to the replies worth reading first.
• Unsupported facts. Replies that state a price, date, or policy that is not in your content.

Reading every transcript does not scale, so sort by these signals and read the worst first. For the numbers to track over time, such as resolution rate and handoff rate, see our guide to measuring AI chatbot success with analytics.

How to Set This Up with BubblaV

BubblaV is an AI chatbot that learns from your website and files. It has a test page for manual checks, Answer Tests for automatic regression testing, and Conversation QA for live conversations. Answer tests are on every plan, including Free. Plans differ only in the number of test runs per month, and test runs do not use AI messages. Conversation QA needs the Pro plan or higher (see plans and pricing).

On Pro, questions the bot could not answer also feed Self-Learning, which drafts Q&A from your agents' replies for you to review. For a support team view of the whole setup, see BubblaV for customer support.

The testing documentation lists every option and the monthly test-run limits. The Conversation QA documentation explains each outcome, flag, and handoff reason.

AI Chatbot Testing Checklist

Common Mistakes to Avoid

Testing only the happy path. If every test asks a question the bot can answer, you never learn what it does when it cannot. The unknown-question tests often find the most serious problems.

Testing once at launch. Content, policies, and instructions change every month. Without regression tests, a wrong answer can stay live for weeks before a customer reports it.

Matching exact wording. Exact-match tests fail on correct replies, so the team learns to ignore failures. Then the real failures get ignored too.

Running write actions in tests. An automatic test that can cancel orders or send emails is a risk, not a safety net. Block or sandbox every action that changes data.

The Bottom Line

Testing an AI chatbot is not a one-time launch task. Write down what correct looks like for the questions that matter, check behavior instead of exact words, keep actions out of your tests, and re-run everything after each change. Then let real conversations show you the next tests to write. A chatbot that is tested this way stays accurate as your business changes.