Testing AI features: a test set, not a lucky demo

A good demo is not yet a good product

An AI feature can look impressive in a presentation and still disappoint in everyday use. A clear input is selected for the demo, the result happens to land well, and somebody rewrites the prompt until the answer works. Real users bring incomplete information, unusual spelling, conflicting requests and situations that never appeared on the roadmap.

The problem is not that a language model sometimes responds differently. The problem is that the team has no shared definition of which differences are acceptable and which ones make the product unusable.

A small, maintained test set creates that definition. It makes quality visible in a repeatable way before a new instruction, model or data source reaches production.

Describe the job before the model

“The answers should be better” cannot be tested. A useful product task explains who provides which input and what must become possible afterwards.

For AI-assisted sorting of customer enquiries, the task might read like this: derive topic, urgency and responsible team from a free-form message. Do not invent missing information; flag uncertain cases for review. The result must be unambiguous enough for the next automation step to process.

That description already reveals the main quality dimensions:

• Is the extracted information factually correct?
• Is an important part missing?
• Does the result follow the expected format?
• Is uncertainty visible?
• Would an error trigger an incorrect or risky action?

The final question matters especially. A misplaced comma and a misclassified safety report are not equivalent failures. Evaluation needs the context of the decision that follows.

Build a starter set from real patterns

The first version does not need a giant benchmark. It matters more that the cases represent the actual product. Useful sources include anonymised support situations, sample content from workshops, known edge cases and deliberately constructed boundaries.

A compact set should contain several groups:

1. Normal cases: common, clearly expressed tasks that must work reliably.
2. Incomplete cases: essential information is absent or ambiguous.
3. Awkward cases: typos, informal language, long text, mixed languages or contradictory statements.
4. High-risk cases: content that should be refused, escalated or handed to a person.
5. Non-tasks: inputs that look similar but should deliberately remain outside the feature's scope.

Every case needs a name, input, relevant context and a description of expected behaviour. Personal data should not enter this collection unchecked. The structure and difficulty of a real situation can usually be preserved while names, companies and specific details are replaced.

Do not demand one reference answer every time

Traditional software can often be checked against one exact return value. Natural language allows several good answers. A single golden phrasing would incorrectly mark useful alternatives as failures.

A combination of hard requirements and a short scoring rubric works better. For a generated summary, it might say:

• All facts required for a decision are included.
• Nothing has been added that is absent from the source.
• Open questions are identified as open.
• The answer stays within the agreed length.
• Tone and language fit the audience.

Each criterion can be rated as “met”, “partly met” or “not met”. Critical failures should be recorded separately. Elegant writing must not receive a good overall score when it invents an incorrect payment deadline.

Measure the unambiguous parts first

Some quality can be evaluated without another AI system. Structured output can be validated against a schema. Required fields, allowed categories, lengths, links and number formats can be checked deterministically. Known terms and prohibited content can also be covered by explicit rules.

These checks are fast, inexpensive and explainable. They should run before every softer assessment. Criteria such as relevance, clarity and suitable tone come next.

For open criteria, a human sample remains the most dependable starting point. Model-based grading can later reduce routine effort, but it should first be calibrated against cases already assessed by people. If human reviewers regularly disagree, the rubric is usually unclear — the automated evaluator is not the first problem to solve.

Run every change through the same harbour

Prompts, models, data sources and post-processing form one product capability. Changing only one part can shift quality somewhere unexpected. A more precise answer may become too long. A stricter output structure may fail more often on unusual input. A new document may improve subject knowledge while reintroducing outdated information.

Every testable variant should therefore have an explicit version. Run the test set before and after the change under the same conditions. Compare not only average scores, but individual regressions and critical failures.

Useful comparison questions include:

• Which previously good cases became worse?
• Which error class was actually reduced?
• Do latency and cost still fit the product journey?
• Does the improvement exist only in one language or user group?
• Which new situation now belongs in the test set?

This turns “it feels better” into a decision with visible consequences.

Example: routing enquiries for a service team

Imagine a website receiving free-form messages and routing them to sales, support or billing. The feature should also identify urgency and prepare a short summary.

The first test set includes common product questions, existing customers with technical problems, invoice enquiries, spam and several mixed cases. One message asks about a new contract while also mentioning an outage in an existing account. The right response is not simply “sales” or “support”, but a defined priority rule and a visible explanation for the employee.

Deterministic checks confirm that category, urgency and summary are present. Reviewers assess whether the routing makes sense, whether essential information is missing and whether uncertainty is handled honestly. Any result that files an urgent outage as a routine sales enquiry counts as critical.

After a change, exactly the same cases run again. New production failures join the suite as anonymised regression cases. The test set grows through real product lessons rather than random accumulation.

Keep observing quality after launch

A test set covers only known situations. Language, customers, offers and underlying data change in production. The product therefore needs a safe feedback route.

Explicit user corrections, sampled reviews and technical signals such as malformed output or repeated retries can all help. The system should collect only what is genuinely required for improvement and covered by its privacy design. Sensitive content must not automatically become a permanent training archive.

A recurring failure is first described, then added as a cleaned test case, and only then fixed. This keeps the cause traceable and prevents the improvement from disappearing unnoticed later.

A pragmatic routine for small teams

For many AI features, this sequence is enough to begin:

1. Describe the core job and the consequence of failure.
2. Collect and anonymise realistic, difficult and high-risk cases.
3. Record expected behaviour and critical failures for each case.
4. Check unambiguous requirements automatically.
5. Score open quality criteria with a short rubric.
6. Run every meaningful variant against the same set.
7. Fix or consciously accept regressions before launch.
8. Add production lessons as regression tests.

Good AI comes from good decisions

A test set does not make a probabilistic system perfectly predictable. It does reveal whether a change makes the product more dependable for its intended work or merely more impressive in one demonstration.

That is the real gain. The team stops debating individual favourite answers and starts deciding together which quality users need, which failures are tolerable and when an AI feature is ready to take on real work.