How to test an AI agent before launch: evals and guardrails

Every AI system looks good in a demo. You pick a clean example, it gives a sensible answer, and everyone in the room nods. The trouble is that real work is not made of clean examples. It is made of half-written emails, missing details, odd requests and customers who ask three things at once.

So before anything we build speaks to a customer or touches a live system, we test it against the work it will actually see.

01What should you test an AI system on?

We gather a set of past examples from your own history: conversations, requests, documents, tickets. Not the tidy ones. A fair sample, including the awkward cases your team remembers. This becomes the test set, and it stays with the project for as long as the system runs.

• The common cases, so we know it handles the everyday work.
• The edge cases your team mentions when asked what goes wrong.
• Cases where the right answer is to hand over to a person.
• Inputs that are incomplete, contradictory or in the wrong format.

02How should AI answers be scored?

We run the system against every case in the set and score each result against what a good member of your team would have done. Is it correct? Is it complete? Does it sound like you? Did it know when to stop and ask? A single good answer means little. The pattern across the whole set is what tells you whether it is ready.

03What are guardrails, and when do you set them?

Before building, we write down what the system must never do. Quote a price it was not given. Promise a date. Send anything to a customer without approval during the first weeks. These rules are part of the spec, and the tests check them as carefully as they check the good answers.

1. Step 01Draft, then approveIn the first phase, the system drafts and a person approves. Nothing leaves without a human seeing it.
2. Step 02Widen what it can do aloneAs the scores hold up on real work, the cases it handles on its own grow, one category at a time.
3. Step 03Keep testing after launchEvery time the system or the model behind it changes, the whole test set runs again before the change goes live.

04What does a good test case look like?

A test case is simple. It is a real input from your history, what a good answer would have been, and anything the answer must never contain. Writing these down forces a useful conversation early, because the team has to agree on what good looks like before the system is built, not after it disappoints.

The people who score the answers matter too. We do the first pass, but the final judgement on whether an answer is good enough belongs to the people who do the work today. They know what a customer would find odd, and they will be the ones living with the system.

05Does testing stop at launch?

Models change, your products change and customers find new ways to ask things. A system that was right in its first month can drift. So the test set grows: when the system gets something wrong in real use, that case is added, and the fix is checked against everything that came before.

None of this is glamorous, and it does not make for a good demo. But it is the difference between a system that impresses in a meeting and one your team trusts on a busy Monday.