The Hidden Cost of Dirty Data in Insurance Underwriting

Every insurer knows their data isn't perfect. Ask an underwriting leader and you'll get a shrug and something like "yeah, it's messy, we work around it." That shrug is one of the most expensive gestures in the industry.

Dirty data in underwriting doesn't announce itself. It doesn't crash a system or trigger an alert. It quietly mis-prices risk, one policy at a time, for years — and the bill arrives as a loss ratio nobody can fully explain.

What "dirty" actually means here

It's rarely dramatic corruption. It's mundane:

  • Missing fields. Roof age is blank on 30% of property records, so the model treats "unknown" as "average." Average is wrong in both directions.
  • Inconsistent codes. Construction type is "frame" in one system, "FR" in another, and "Wood Frame" in a third. Your model sees three risk classes where there's one.
  • Free text where structure belongs. Prior claims history sits in an adjuster's notes field. No model can read it, so it's simply absent from pricing.
  • Stale values. The property was renovated in 2019. Your record says 1974. You're pricing a house that no longer exists.
  • Duplicates. The same insured appears three times, so their claims history looks a third as bad as it is.

None of this looks like a crisis on a dashboard. All of it distorts price.

Why this is worse than a model that obviously fails

Here's the part that should worry you. A model that plainly doesn't work gets caught — someone notices the predictions are nonsense and pulls it. A model fed dirty data works. It produces confident, plausible, precisely-wrong numbers.

And the errors aren't random. They're systematic, because the dirt is systematic. If roof age is missing more often for older properties (it usually is), you're consistently under-pricing your riskiest book. You don't discover that from the model. You discover it from claims, three years later, when the losses land.

That's the asymmetry: bad data doesn't produce noisy pricing, it produces biased pricing. And biased pricing in one direction is adverse selection — the customers you've under-priced are exactly the ones who'll buy.

The costs nobody puts in the business case

Mis-priced risk. The big one. Every point of loss-ratio drift traceable to bad inputs is money that never had to be lost.

Underwriter time. Your most expensive people spend their days chasing missing data, reconciling systems that disagree, and manually correcting records. That's not underwriting; that's data entry with a licence.

Stalled automation. Straight-through processing stops dead at data quality. You can't auto-issue a policy on fields you don't trust, so everything routes to a human — and your STP rate stays at 15% while competitors hit 90%.

Unprovable fairness. With the NAIC bulletin adopted across 23 US jurisdictions and the EU AI Act treating insurance AI as high-risk, you increasingly have to demonstrate your pricing isn't discriminatory. You cannot prove fairness on data you can't slice cleanly. Dirty data turns a compliance question into an unanswerable one.

What actually fixes it

Not a data-cleansing project. Those end the moment the consultants leave, because the systems that produced the dirt keep producing it.

What works is treating quality as a property of the pipeline, not a one-off cleanup:

  1. Measure it. You cannot manage what you don't score. Completeness, consistency, and freshness — per field, tracked over time. Most insurers have never seen this number.
  2. Fix it at the source where you can, in the pipeline where you can't. Standardise codes, resolve entities, and enforce validation as data lands — not in a downstream spreadsheet.
  3. Make gaps explicit. "Unknown" should be a first-class value the model can reason about, not silently imputed to the mean.
  4. Enrich deliberately. Third-party property, geospatial, and permit data can fill the fields your systems never captured — but only once you can join it reliably to the right record.
  5. Block on quality. If a field falls below threshold, that model doesn't ship. Make it a gate, not a warning.

The uncomfortable summary

Most carriers are trying to buy their way to better pricing with better models. But a sharper model on the same dirty inputs mostly gets you more confident wrong answers, faster.

The unglamorous truth is that data quality is underwriting performance. It's not the plumbing beneath the strategy — it's most of the strategy. Carriers that measure and fix it find that their existing models suddenly get better, their STP rate climbs, their underwriters go back to underwriting, and their loss ratio stops drifting for reasons nobody can name.

You don't need a shrug. You need a score.

We build the pipelines, quality frameworks, and governed data platforms that insurance AI depends on — 200+ cloud migrations across Snowflake, Databricks, and the modern lakehouse. More at IntelliBooks.

Comments

Post a Comment

Popular posts from this blog

Snowflake or Databricks for Insurers? An Honest Comparison

It's the question in every insurance data platform decision, and it generates more heat than insight: Snowflake or Databricks? Here's an honest answer from having migrated a lot of both. Short version: they're both excellent, the gap is narrower than either vendor implies, and the decision matters far less than what you do next. But there are real differences, and your workload should pick. Where each genuinely leads Snowflake is stronger when your centre of gravity is SQL analytics. Reporting, actuarial analysis, regulatory extracts, BI over structured policy and claims data. It's operationally calmer — less to tune, fewer knobs, and your existing SQL people are productive on day one. Separation of storage and compute means the finance team's month-end doesn't fight the actuaries' model run. For a carrier whose data is mostly relational and whose consumers are mostly analysts, it's the shorter path. Databricks is stronger when your centre of g...
Read more

What 200+ Cloud Migrations Taught Us About Insurance Data

Over 200 cloud migrations, patterns emerge. Not the ones in the vendor case studies — the ones you only see from inside, at 2am, when the reconciliation doesn't balance. Here's what we've actually learned about insurance data. Some of it is uncomfortable. 1. The estimate is always wrong in the same direction Every migration takes longer than planned, and almost never because of the technology. It's because of what you find: a rating rule nobody documented, a batch job that silently corrects bad data at 2am, a product variant that exists only in one state. The work isn't moving data. It's discovering what the old system actually does — which nobody knows in full, because the people who knew have retired. Budget for archaeology, not just engineering. 2. Every reconciliation mismatch is a gift When your new premium calculation disagrees with the legacy system by ₹40 on 300 policies, the instinct is to treat it as a bug to squash quickly. Don't. Tha...
Read more

From Mainframe to Lakehouse: How to Modernize a 30-Year-Old Policy System Without Downtime

Ask any insurance CTO what keeps them from modernizing, and you'll rarely hear "budget" or "talent." You'll hear something closer to fear: the policy administration system has been running since the 1990s, it processes every quote, endorsement, and renewal the company makes, nobody who wrote it still works there — and the business cannot go dark for even an afternoon. So the migration gets deferred. Another year passes. Meanwhile every AI initiative, every customer-experience project, and every analytics roadmap quietly starves, because they all need data that's trapped inside that system. Here's the thing: the fear is rational, but the conclusion isn't. Migrating a legacy policy core without downtime is well-trodden ground. It just requires abandoning the approach everyone instinctively reaches for. Why the big-bang cutover fails The instinct is to build the replacement, pick a weekend, flip the switch, and pray. This fails for a predict...
Read more