Data Lineage, Explained: What Regulators Ask For That You Can't Fake

Data lineage is one of those terms that sounds like governance bureaucracy until a regulator asks you a question you can't answer. Then it becomes the most important thing in your data platform.

Here's what it actually is, why insurers increasingly can't operate without it, and — the part that catches everyone — why you can't add it after the fact.

What lineage actually means

Data lineage is the ability to trace any piece of data back through every step it took to get where it is. For a number on a screen — a premium, a risk score, a declined application — lineage answers: which source records produced this? Through which transformations? Under which version of which logic? At what time?

Think of it as an unbroken chain of custody for data. In the same way a courtroom needs to know exactly who handled a piece of evidence and how, a regulator increasingly needs to know exactly what data produced a decision and how it was processed.

It sounds abstract until you need it. Then it's the difference between answering a regulator in an afternoon and not being able to answer at all.

Why insurers now genuinely need it

Three forces have moved lineage from nice-to-have to prerequisite:

1. AI regulation. The NAIC Model Bulletin, adopted across 23 US jurisdictions, and the EU AI Act, treating most insurance AI as high-risk, both require insurers to explain and document how AI-influenced decisions are made. You cannot explain a decision whose inputs you can't trace. Lineage is the substrate under every explainability and audit requirement.

2. Adverse action. When you decline or surcharge someone, you must be able to state why — which specific data drove it. That's a lineage question: what fed this decision?

3. Trust in your own numbers. When two reports disagree about the same figure — and in fragmented insurers they always do — lineage is how you find out which is right and why. Without it, you're arbitrating between numbers whose origins nobody can reconstruct.

The three levels of lineage

Not all lineage is equal. There are roughly three levels, and most insurers overestimate which one they have:

Table-level: "This report draws from these tables." Better than nothing, nearly useless for a regulator. It tells you the neighbourhood, not the house.

Column-level: "This field was derived from these specific fields via this transformation." Now you can trace a number. This is the practical target.

Row/decision-level: "This specific decision used these exact records, this model version, and produced this output at this timestamp." This is what adverse-action and AI-audit requirements actually demand, and it's the level almost nobody has.

When someone says "we have lineage," they usually mean table-level, and they usually discover the gap at the worst possible moment.

The catch: you cannot retrofit it

This is the part that costs people, so it's worth stating bluntly: lineage is captured as data flows, or it doesn't exist.

Lineage is a record of what happened to data as it moved. If you didn't record it while the data was moving, the information is gone — you can't reconstruct, months later, exactly which records and which logic version produced a specific number in a pipeline that wasn't instrumented to remember. You can guess. A guess is not a chain of custody, and a regulator knows the difference.

So the teams that treat lineage as a phase-2 concern — "we'll add governance once the platform's built" — discover that phase 2 requires having captured, in phase 1, information they threw away. It's not a feature you bolt on. It's a property the pipeline has from the first day it runs, or never.

What building it actually involves

1. Instrument at pipeline-build time. Every transformation records its inputs, logic version, and outputs as it runs. This is a design decision made before launch.

2. Version your logic. To trace a decision you must know which version of the transformation and which version of the model was in effect. Un-versioned logic makes historical lineage impossible.

3. Log decisions immutably. For the regulated decision points, an append-only record: inputs, model version, features, output, timestamp.

4. Make it queryable. Lineage you can't query in response to a specific question is just exhaust. The value is answering "trace this number" in minutes, not weeks.

The upside

Lineage reads like pure compliance cost, but teams that build it well get something back: they can debug data issues in a fraction of the time, they can trust their numbers, and they can move faster on AI precisely because they can prove what it's doing. The discipline that satisfies the regulator also makes the platform genuinely better to operate.

But the window to get it for free is now, while you're building. Instrument lineage as the data flows, or spend the next audit explaining why you can't answer a question you should have been able to answer all along.

We build lineage-aware, audit-ready data platforms — instrumented from day one, not retrofitted. More at IntelliBooks.

Comments

Post a Comment

Popular posts from this blog

Why Insurance Fraud Rings Are a Graph Problem (And Your SQL Database Can't See Them)

Your fraud team is good at catching individuals. A claimant whose story shifts, a receipt that looks altered, someone who files a bit too often. That's what the rules were built for, and they work. They're nearly blind to organised fraud — and organised fraud is where the money goes. The reason isn't that your team lacks skill or your rules lack sophistication. It's that you're asking a question your database structurally cannot answer. What a fraud ring actually looks like A ring isn't one bad claimant. It's a network: a handful of claimants, a couple of "witnesses", a repair shop, a medical clinic, a few bank accounts, some phone numbers, and a device or two — recombined across dozens of claims over months. Take any single claim from that ring and it looks fine. The claimant has no history. The damage is plausible. The repair estimate is in range. Nothing trips a rule, because the rule is examining one claim in isolation, and in isolati...
Read more

Claims Leakage: The Quiet Loss AI Can Close

Claims leakage is the money your company pays out that it didn't have to. Not fraud — leakage. The extra amount on legitimate claims that got overpaid because someone missed something, applied the wrong rule, or lacked the information to know better. Industry estimates put it at 3–10% of claims spend. For a mid-size insurer that's a rounding error you could retire. And almost nobody manages it directly, because it's nearly invisible — every individual leaked rupee looks like a normal payment. Why leakage is so hard to see Fraud has a villain. Leakage doesn't. It's the accumulation of small, defensible, individually-reasonable decisions: An adjuster settled at the reserve figure because they were busy, and the reserve was set high. A repair was authorised at shop rates when a network rate applied. A subrogation opportunity was never flagged, so recovery never happened. A policy limit or deductible was applied wrongly because the coverage detail was buried...
Read more

The Subrogation Money Insurers Leave on the Table

Subrogation is the least glamorous word in insurance and one of the most profitable. It's the process of recovering money from the party actually responsible for a loss, after you've paid your customer. Done well, it flows straight to the bottom line — recovered money is nearly pure margin. Most insurers leave a lot of it on the table. Not because they don't try, but because the opportunities are identified too late, by the wrong people, from incomplete information. It's a data problem, and it's an unusually clear one. Why subrogation is a timing game you're losing Recovery potential decays fast. The evidence that establishes another party's liability — the scene, the witnesses, the vehicle, the maintenance records, the CCTV — degrades or disappears within days. The other party's insurer gets less cooperative once they've closed their own file. So the single biggest lever in subrogation is identifying the opportunity early , ideally at first ...
Read more