Real-Time vs Batch: Which Insurance Data Needs to Be Live

Every insurance data conversation eventually reaches a fork: should this be real-time or batch? And it's usually answered badly — either "stream everything, it's 2026" or "batch is fine, it always has been."

Both answers are expensive. Here's the rule that actually works.

The rule: follow the decay rate

Data should be as fresh as the decision that consumes it — no fresher.

Ask one question of any data flow: how fast does this signal lose its value?

A fraud indicator at first notice of loss decays in minutes — once the claim is paid, the signal is worthless no matter how accurate. A policyholder's date of birth doesn't decay at all. Streaming the second is pure cost; batching the first is a fraud loss.

That's the whole framework. Not "is it important" — importance is not the same as urgency. Loss-ratio reporting is extremely important and perfectly fine at T+1.

What genuinely needs real-time in insurance

FNOL and claims triage. The decision — fast-track, escalate, or flag — happens at intake. Data arriving tomorrow means the decision was made without it.

Fraud signals. Graph relationships, device fingerprints, behavioural patterns. Detecting a fraud ring after the payout is expensive analytics, not fraud prevention.

Quote and bind. If a customer is on your site expecting a price in seconds, every input to that price must be reachable in seconds.

Telematics-based intervention. Crash detection or driving alerts — the entire value is immediacy.

Agentic workflows. An agent orchestrating a claim calls tools mid-conversation. It cannot wait for tonight's batch, and it won't say "unknown" — it'll answer from stale data, confidently.

What is perfectly fine on batch

Regulatory and financial reporting. Monthly and quarterly cycles. Nightly is generous.

Actuarial and reserving analysis. The models look at years of history. An extra day is noise.

Model training data. You retrain periodically, not continuously. Batch.

Most BI dashboards. Be honest: nobody makes a different decision because the dashboard is four hours fresher. They just said they wanted real-time because it was free to ask for.

Policy history and static attributes. Date of birth, construction type, VIN. These barely change.

The costs of getting it wrong in each direction

Streaming too much is the fashionable mistake. Every stream is infrastructure to run, monitor, and debug at 3am. Streaming pipelines fail in ways batch doesn't — out-of-order events, late arrivals, exactly-once semantics, replay after an outage. You've taken on a permanent operational burden so a dashboard nobody watches can be four hours fresher. Multiply across 40 feeds and you have a team whose full-time job is keeping unnecessary streams alive.

Batching too little is the traditional mistake, and it's quieter. Your "real-time fraud detection" runs overnight, so it catches fraud you already paid. Your STP rate stays low because the eligibility signal isn't there at quote time, so it routes to a human. Nobody logs this as a data problem — it shows up as loss ratio and manual volume, which get blamed on other things.

The middle path most carriers should be on

Not a choice — a split, deliberately:

  • Streaming (Kafka or similar) for FNOL events, fraud signals, telematics, quote interactions — the decay-sensitive minority.
  • CDC / micro-batch for policy and claims core data. Change data capture from the legacy system every few minutes covers most "we need it fresh" requirements without a true streaming architecture.
  • Nightly batch for history, reporting, training sets, and everything else.
  • One lakehouse underneath so it all lands in the same place and joins cleanly regardless of how it arrived.

That last point is what makes the split workable. If your streaming data lives in a different world from your batch data, nobody can join a live FNOL event to five years of claims history — and that join is the whole point.

The question to ask in the next meeting

When someone requests real-time, don't argue. Ask: "what decision changes if this is seconds old instead of hours old?"

If there's a crisp answer — a claim routes differently, a quote is priced differently, a fraud case is stopped — stream it. If the answer is "it'd be nice to see it sooner," that's batch, and you just saved yourself a pipeline and the person who'd have spent their career maintaining it.

Real-time isn't a maturity level. It's a tool with a real cost, and the mature move is knowing exactly where it earns that cost — and where it doesn't.

We design and build the streaming, CDC, and batch pipelines behind insurance AI — on Snowflake, Databricks, and the modern lakehouse. More at IntelliBooks.

Comments

Post a Comment

Popular posts from this blog

What 200+ Cloud Migrations Taught Us About Insurance Data

Over 200 cloud migrations, patterns emerge. Not the ones in the vendor case studies — the ones you only see from inside, at 2am, when the reconciliation doesn't balance. Here's what we've actually learned about insurance data. Some of it is uncomfortable. 1. The estimate is always wrong in the same direction Every migration takes longer than planned, and almost never because of the technology. It's because of what you find: a rating rule nobody documented, a batch job that silently corrects bad data at 2am, a product variant that exists only in one state. The work isn't moving data. It's discovering what the old system actually does — which nobody knows in full, because the people who knew have retired. Budget for archaeology, not just engineering. 2. Every reconciliation mismatch is a gift When your new premium calculation disagrees with the legacy system by ₹40 on 300 policies, the instinct is to treat it as a bug to squash quickly. Don't. Tha...
Read more

From Mainframe to Lakehouse: How to Modernize a 30-Year-Old Policy System Without Downtime

Ask any insurance CTO what keeps them from modernizing, and you'll rarely hear "budget" or "talent." You'll hear something closer to fear: the policy administration system has been running since the 1990s, it processes every quote, endorsement, and renewal the company makes, nobody who wrote it still works there — and the business cannot go dark for even an afternoon. So the migration gets deferred. Another year passes. Meanwhile every AI initiative, every customer-experience project, and every analytics roadmap quietly starves, because they all need data that's trapped inside that system. Here's the thing: the fear is rational, but the conclusion isn't. Migrating a legacy policy core without downtime is well-trodden ground. It just requires abandoning the approach everyone instinctively reaches for. Why the big-bang cutover fails The instinct is to build the replacement, pick a weekend, flip the switch, and pray. This fails for a predict...
Read more

The Insurance Data Readiness Checklist (Before You Buy Any AI)

Before you sign an AI contract, spend twenty minutes on this instead. It'll tell you more about whether the project will succeed than any vendor demo. Roughly 80% of the effort in a successful insurance AI project goes into data, not models. So the useful question isn't "which AI should we buy?" It's " is our data ready for any AI at all? " That question is measurable. Here's how to measure it. Score each item 0, 1, or 2. Be honest — the point is to find gaps, not to pass. Section 1: Foundation 1. Single source of truth. Is there one place where policy, claims, and billing data live together and can be queried in one join? 0 = data lives only in source systems · 1 = a warehouse exists but is partial/stale · 2 = one governed platform, all core domains 2. Stable customer key. Can you identify the same person across every product line, reliably, today? 0 = no · 1 = partial matching, no durable ID · 2 = resolved identity with a persistent k...
Read more