How to test AI before your customers see it

Most AI features go live after the sentence “we tried a few examples and it looked good”. That is exactly the point where a prototype differs from production. In the Expiro app we checked the AI on 425 situations before launch, and along the way we found that one and the same system can score 99% or 72% depending on who wrote the test examples. Here is the whole process, so you can use it in your own project.

What kind of AI this is

Expiro keeps track of companies’ deadlines for vehicle technical and emissions inspections, insurance and motorway vignettes. It has a “Niečo nesedí?” (“Something wrong?”) button. When a customer presses it, they see the four most likely causes of their problem, each with a one-click fix. The suggestions are picked by rules together with an AI model that also understands what the customer types in their own words. We explain why we chose this combination in the article Rules or AI?.

The task is easy to measure: the customer’s problem is either among the first three suggestions or it is not. But the process below applies to any AI that sorts, selects or extracts something, from emails to invoices to orders.

1. A list of situations where you know the correct answer

Without one, you cannot make decisions about AI. You can only debate it. We put together 425 situations. Each contains the state of the company (plan, vehicles, deadlines), what the customer did in the app, what they typed if anything, and the correct answer.

We also randomly added other problem vehicles to the situations. It is not enough to guess that the inspection date is wrong. You also have to guess which vehicle it is. The randomness uses a fixed setting, so every new test run gets exactly the same situations and the results can be compared.

2. Test texts must not be written by the system’s author

This was the most important finding of the whole test. We wrote the first 98 texts ourselves, meaning the same people who also wrote the rules. The rules scored 99% on them. It looked as if the AI was not needed at all.

Then we had another AI model write 288 more texts. It got only a short story, such as “we sold the car, but we still get reminders”, and had to write it the way an angry, polite or terse customer would, with typos and without diacritics. It did not see our rules or the descriptions the AI in the app receives.

The same system, a different test author, and a gap of 27 percentage points. If you test on your own examples, you are measuring how well you know yourself. Test examples should be written by someone who does not know how the system works. Ideally real customers. Until you have them, a colleague from another department will do, or another model with a brief that gives no hints.

3. Compare it with a dumb version

A number on its own says nothing. 81% sounds good until you find out that something completely simple would do the same. That is why we measured four versions on the same situations:

The same list for everyone shows that guessing makes sense at all. The rules show how much an ordinary program without AI can do. Only the gap between them and the AI is what you are paying for when you pay for AI. In our case it was large only for texts. When the customer only clicked, the rules scored 97% and the AI 71%. That is why we deployed both.

4. Look at the mistakes, not just the percentages

In the first round, the AI did poorly on situations without text, only 65%. The percentage alone would not have told us why. What helped was reading the individual mistakes.

The app was sending events to the AI as codes, such as open:plan for an opened price list. The model read it as the “Schedule” button. It interpreted the code archive (vehicle removed from the fleet) as the vehicle being greyed out. The model’s judgement was not wrong. It just did not understand our shorthand.

The fix was simple: we now send events as plain sentences, such as “The customer opened the price list”. Accuracy without text rose to 71%. Combined with the rules, to 97%.

One caveat: we made this change after seeing the mistakes on the same test set. But it was a general change (sentences instead of codes), not tuning for specific cases. If you change the input just to make one particular test pass, you are only inflating your numbers.

5. Check whether the model’s confidence means anything

With every answer, the model returns how confident it is. That is only worth something if you can trust the number. So we compared what the model claimed with how often it was actually right. This is called calibration:

When the model is confident, it is mostly right. When it hesitates, it is wrong in roughly half the cases. You can work with that: at high confidence the system can act faster, at low confidence it should offer more options or ask. A model that claims 95% on everything would not give you that.

6. Cost and speed belong in the test

The test also measured what is usually least clear in quotes for AI projects:

• Cost: $0.11 per thousand openings of the window, so roughly a hundredth of a cent per opening.
• Speed: half of the answers arrived within 0.29 seconds.
• Fallback: if the AI does not answer within 2.5 seconds, the suggestions are put together by the rules alone.

For comparison: depending on the credit package, a single vehicle lookup in the registry costs Expiro as much as 40 to 120 openings of the window with AI. The AI was not the most expensive item in the product. The data was. This is often the case with AI features, and it is worth doing the maths before the AI gets cut for being “too expensive”.

What to include in the cost of an AI feature:

1. Model calls at a realistic number of uses, not one test click.
2. The test set: one-off work without which later changes cannot be assessed.
3. A fallback in case the model does not answer or the provider goes down.
4. Monitoring in live operation: a record of every decision, cost and response time, a weekly summary.
5. Data protection: what the model may see and what the provider may store.
6. People for the exceptions: someone has to take over the cases the AI should not handle.

We write about AI project budgets in general in the article how much does an AI solution cost.

7. Live operation is a continuation of the test

The test set tells you whether it is worth launching. Only real customers tell you whether it works. That is why Expiro runs as an A/B test: each company permanently gets suggestions either from the rules only, or from the rules with AI. Every week we compare:

• how often the customer clicked one of the first three suggestions,
• how many times they answered “Did this help? Yes” without us having to step in,
• how quickly they found their problem,
• how often they reached for “Something else, I’ll type it”, which reveals that something is missing from our list,
• cost and response time per thousand openings.

The record of each decision also includes which suggestions were shown and in what order. When something gets worse, we can trace back exactly when and why.

Honest caveats

• The test texts were written by a model, not by real customers. People write in even more varied ways.
• The situations were designed by the same people who wrote the rules. That is another reason why results from live operation count, not results from the test.
• The numbers in this article come from the test before launch. Once we have numbers from the A/B test, we will publish them.

What to prepare before launching an AI feature

1. A list of situations with the correct answer. Starting with a hundred is fine.
2. At least half of the examples from someone who does not know the system.
3. A dumb version for comparison, and a version without AI.
4. Time to read the mistakes, not just the percentages.
5. A check that the model’s confidence matches reality.
6. Cost and response time per thousand uses, and a fallback for outages.
7. A plan for how you will measure in live operation.

This is the difference between a demo and a feature you can trust. Do you have an AI prototype that looks good, but you do not know how it would hold up in front of customers? See how we approach taking an AI prototype to production, or get in touch. We will help you build the test set as the very first step, before any changes are made.