Data Contracts: Ending the "Someone Changed a Field" Outage

Here's a failure everyone in data engineering recognises. A dashboard breaks. Or worse, it doesn't break — it just starts showing numbers that are subtly wrong. Days later someone traces it to an upstream team who renamed a field, changed a code, or dropped a column in a release nobody outside their team read.

Nobody did anything wrong, exactly. The upstream team owned that system and changed it. They had no idea a pricing model depended on that field, because nobody ever told them.

That's the problem data contracts exist to solve, and it's the reason mature data platforms stay reliable while immature ones lurch from breakage to breakage.

What a data contract is

A data contract is an explicit, versioned agreement between a data producer and its consumers about what will be provided and what guarantees come with it. Concretely:

  • Schema — fields, types, nullability
  • Semantics — what each field actually means ("policy_status = ACTIVE means bound, in-force, not lapsed")
  • Quality guarantees — completeness thresholds, valid ranges, allowed code values
  • Delivery guarantees — frequency, freshness, late-arrival behaviour
  • Change policy — how breaking changes are announced, and the notice period
  • Owner — a named person accountable, not a team mailbox

The critical property: it's machine-checkable. A contract that lives in Confluence is documentation. A contract enforced in the pipeline is a contract.

Why insurance needs this more than most

Insurance data has three characteristics that make undeclared coupling especially dangerous:

Systems are old and shared. The policy admin core feeds everything. A change there ripples into pricing, reserving, reporting, and every model — and nobody has a complete list of consumers.

The numbers are regulated. A silently-wrong figure in a solvency return or a rating decision isn't an inconvenience. And when the regulator asks why, "an upstream team changed a code value and we didn't notice for six weeks" is a governance finding, not an explanation.

The blast radius is invisible. Because ownership is usually undefined (the "who owns the data?" problem), producers genuinely don't know who depends on them. Contracts make the dependency explicit — which is the actual fix.

How to introduce them without a two-year programme

1. Start with your most critical dataset, not all of them. Pick the feed that most decisions depend on — usually policy or claims core extract. One contract, done properly, teaches you more than a framework rollout.

2. Write down what's already true. The first contract shouldn't demand change; it should document the current schema, meanings, and de-facto guarantees. This alone surfaces disagreements — you'll find two teams believe different things about the same field, which is a discovery worth the exercise on its own.

3. Enforce at the boundary. Validate incoming data against the contract at ingestion. Violations should fail loudly and route to the owner. The key discipline: reject bad data rather than silently transforming it. Silent coercion is how a misread column becomes a misstated reserve.

4. Version, and treat breaking changes like an API. Additive changes are fine. Removing or redefining a field is a breaking change requiring a version bump and a notice period. Data producers should think like API owners, because that's what they are.

5. Publish the consumer list. The most valuable side-effect: producers can finally see who depends on them. Most upstream breakage is caused by people who genuinely didn't know.

6. Wire it to quality scores and lineage. Contracts define the promise, quality scores measure whether it's kept, lineage shows where the data went. Together they're an actual data platform; separately they're three projects.

The honest constraint

Data contracts are a social mechanism with a technical implementation. The schema validation is straightforward; getting an upstream team to accept accountability for a contract is the hard part — especially when they're measured on delivering their own system's features, not on downstream stability.

This is where it connects to data ownership. A contract needs a named owner with a mandate. If nobody owns the data, nobody can sign the contract, and you're back to hoping people read release notes.

So the sequencing is: establish ownership, then contracts become possible. Try it in the other order and you'll produce beautiful YAML that nobody feels bound by.

The point

Every organisation with a data platform has a recurring category of incident: something upstream changed and something downstream broke, quietly. Teams treat each instance as bad luck and add another monitoring rule.

It isn't bad luck. It's an undeclared dependency, and the fix is to declare it — explicitly, versioned, machine-checked, with a name attached.

Boring, unglamorous, and the difference between a data platform people trust and one they work around.

We build governed pipelines with contracts, quality gates, and lineage wired in from the start. More at IntelliBooks.

Comments

Post a Comment

Popular posts from this blog

Why Your Insurance Data Warehouse Didn't Fix Anything

You built the data warehouse. It took eighteen months and cost more than planned. Policy data, claims data, billing — it's all in there. The project was declared a success and everyone moved on. And yet: underwriting still can't see the whole customer. The AI initiative is still stalled. Someone still asks for a report and waits three weeks. Nobody says it out loud, but the question hangs there — what exactly did we buy? You're not alone, and it isn't because you built it badly. It's because a warehouse solves a problem you didn't have. The problem a warehouse solves Warehouses solve location . Before: your data was in eight systems. After: it's in one place. That's genuinely valuable — you can run a query across policy and claims without begging two teams for extracts. But notice what that fixes: it fixes where the data is . It doesn't touch what the data means , whether it's trustworthy, or whether the records refer to who you think t...
Read more

Straight-Through Processing: From 10% to 90%

Straight-through processing is the number every insurance executive quotes and almost nobody achieves. The industry benchmark has moved from 10–15% to 70–90% at the carriers who've cracked it. That's not a marginal efficiency gain — it's a different cost structure. So why is your STP rate still stuck in the teens? Because STP gets treated as a workflow project. It isn't. It's a data problem wearing a workflow costume. What STP actually requires Straight-through means a quote, policy, or claim goes from submission to completion with zero human touches . For that to happen, every decision point in the chain has to be answerable automatically. Every single one. A chain of nine automated steps and one manual check isn't 90% straight-through — it's 0%. One human touch and the whole thing is a manual process with extra steps. That's the bar. And it means the constraint isn't the workflow engine — it's whether the data needed at each decisio...
Read more

Embedded Insurance: Why the API Is the Easy Part

Embedded insurance is the idea that coverage should be offered at the moment of need, inside someone else's product — travel insurance at checkout, device protection when you buy the phone, cover for the rental car in the booking flow. It's one of the fastest-growing distribution models in the industry, and every insurer has a strategy slide about it. Most of those strategies underestimate the same thing, and it isn't the partnerships or the product. It's the data plumbing. Everyone assumes the API is the hard part. The API is the easy part. What embedded insurance really demands The promise of embedded is speed and invisibility: a customer buys a flight and, in the same breath, is offered exactly the right travel cover at exactly the right price, and accepts with one tap. No forms, no friction. Deliver that and you've won. But "exactly the right cover at exactly the right price in one tap" makes brutal demands on your back end: You must price i...
Read more