How Do You Test an AI That Doesn't Give the Same Answer Twice?

Traditional software testing rests on an assumption so basic nobody states it: the same input produces the same output. Write a test, assert the result, done.

Language models break that assumption. Ask the same question twice and you may get two differently-worded answers — both potentially correct, or one subtly wrong. So how do you test a system that doesn't repeat itself, when it's answering questions about someone's insurance coverage?

This is the question that separates AI demos from AI in production, and most teams hit it after they've already promised a launch date.

Why your existing QA process doesn't transfer

Your test suite asserts equality. `assert result == expected`. With an LLM, the correct answer might be phrased fifty ways, so exact-match assertions fail on correct output and pass on nothing useful.

Worse, the failure modes are different in kind:

  • Fluent wrongness. Traditional bugs look like errors. LLM failures look like confident, well-written, incorrect answers. Nothing throws.
  • Silent regression. A model or prompt update degrades quality on a category of question with no error, no stack trace, nothing in the logs.
  • Distribution, not determinism. A system that's right 97% of the time isn't "passing" or "failing" — it has a quality distribution, and you need to measure it.

The evaluation harness: your new test suite

The answer is to replace assertions with evaluation — a repeatable harness that scores model behaviour across a curated set of cases. Treat it exactly as seriously as unit tests, because it plays the same role.

1. Build a golden dataset. A few hundred real questions with expert-verified correct answers. This is the single highest-value artifact you'll build, and it's tedious human work nobody wants to fund. Fund it anyway — without it you're guessing forever.

Cover: common questions, known edge cases, previously-observed failures, and — crucially — questions that should be refused ("what should I do about my neighbour's dispute?").

2. Score on multiple dimensions. "Correct" is too coarse. For a coverage-answering system:

  • Factual accuracy — does it match the source document?
  • Groundedness — is every claim traceable to retrieved text, or did it invent?
  • Completeness — did it mention the exclusion that changes the answer?
  • Appropriate refusal — did it decline when it should have?
  • Retrieval quality — measured separately, because most "model" failures are retrieval failures.

That last point is the one teams learn late: when the answer is wrong, the model usually did fine with what it was given. Evaluate retrieval independently or you'll spend months tuning the wrong component.

3. Combine automated and human scoring. Automated metrics and LLM-as-judge give you speed and regression detection on every change. Periodic human expert review calibrates whether the automated scores mean anything. You need both; either alone drifts.

4. Run it on every change. Prompt edit, model version, retrieval tweak, document update — re-run the harness and compare. This is your CI. Without it, you're shipping changes to a system whose behaviour you can't characterise.

Production monitoring is not optional

Offline evaluation tells you about your golden set. Production tells you about reality, and reality drifts:

  • Log everything — question, retrieved context, answer, confidence, latency.
  • Track refusal rate. A sudden drop often means the system started answering things it shouldn't.
  • Capture user corrections and escalations. Every time a human overrides the AI, that's a free labelled failure case. Feed it into the golden set.
  • Watch for question drift. Users will ask things your golden set never anticipated.

The governance angle

In insurance this stops being engineering hygiene and becomes a compliance requirement. With the NAIC bulletin adopted across 23 US jurisdictions and the EU AI Act treating insurance AI as high-risk, you must demonstrate your AI system is tested, monitored, and performing as intended.

"We tried it and it seemed good" is not a defensible position. An evaluation harness with versioned results is exactly the evidence those frameworks expect — which means the engineering discipline and the regulatory requirement are the same work. Do it once, satisfy both.

Where teams go wrong

The common pattern: build the demo, impress everyone, get a launch date, then discover there's no way to know whether it's actually good — and no time to build one. The evaluation harness gets deferred and never arrives, so the system runs in production with quality nobody can characterise.

Build the harness before the launch date exists. It's less exciting than the demo and it's the thing that determines whether the demo survives contact with real users.

You can't assert your way to confidence with a non-deterministic system. You can measure your way there — but only if you build the thing that measures.

We build evaluated, monitored, governed AI systems and the data foundations under them. More at IntelliBooks.

Comments

Post a Comment

Popular posts from this blog

Why Your Insurance Data Warehouse Didn't Fix Anything

You built the data warehouse. It took eighteen months and cost more than planned. Policy data, claims data, billing — it's all in there. The project was declared a success and everyone moved on. And yet: underwriting still can't see the whole customer. The AI initiative is still stalled. Someone still asks for a report and waits three weeks. Nobody says it out loud, but the question hangs there — what exactly did we buy? You're not alone, and it isn't because you built it badly. It's because a warehouse solves a problem you didn't have. The problem a warehouse solves Warehouses solve location . Before: your data was in eight systems. After: it's in one place. That's genuinely valuable — you can run a query across policy and claims without begging two teams for extracts. But notice what that fixes: it fixes where the data is . It doesn't touch what the data means , whether it's trustworthy, or whether the records refer to who you think t...
Read more

Straight-Through Processing: From 10% to 90%

Straight-through processing is the number every insurance executive quotes and almost nobody achieves. The industry benchmark has moved from 10–15% to 70–90% at the carriers who've cracked it. That's not a marginal efficiency gain — it's a different cost structure. So why is your STP rate still stuck in the teens? Because STP gets treated as a workflow project. It isn't. It's a data problem wearing a workflow costume. What STP actually requires Straight-through means a quote, policy, or claim goes from submission to completion with zero human touches . For that to happen, every decision point in the chain has to be answerable automatically. Every single one. A chain of nine automated steps and one manual check isn't 90% straight-through — it's 0%. One human touch and the whole thing is a manual process with extra steps. That's the bar. And it means the constraint isn't the workflow engine — it's whether the data needed at each decisio...
Read more

Embedded Insurance: Why the API Is the Easy Part

Embedded insurance is the idea that coverage should be offered at the moment of need, inside someone else's product — travel insurance at checkout, device protection when you buy the phone, cover for the rental car in the booking flow. It's one of the fastest-growing distribution models in the industry, and every insurer has a strategy slide about it. Most of those strategies underestimate the same thing, and it isn't the partnerships or the product. It's the data plumbing. Everyone assumes the API is the hard part. The API is the easy part. What embedded insurance really demands The promise of embedded is speed and invisibility: a customer buys a flight and, in the same breath, is offered exactly the right travel cover at exactly the right price, and accepts with one tap. No forms, no friction. Deliver that and you've won. But "exactly the right cover at exactly the right price in one tap" makes brutal demands on your back end: You must price i...
Read more