Document Intelligence: Getting Insurance Data Out of 40 Years of PDFs

Here's an uncomfortable estimate: somewhere between 60% and 80% of what your insurance company knows isn't in a database at all. It's in documents.

Policy wordings. Medical records. Police reports. Repair estimates. Surveyor reports. Correspondence. Endorsement letters. Forty years of accumulated PDFs, scans, faxes-turned-images, and email attachments — full of exactly the facts every AI model on your roadmap wants, and structurally unable to give them up.

This is the quietest bottleneck in insurance data, and it blocks more than people realise.

Why this is worse in insurance than elsewhere

Plenty of industries have documents. Insurance has a specific compounding problem:

The documents are the contract. A policy wording isn't a description of coverage — it is the coverage. What's in that PDF is legally what you owe. You can't approximate it.

They arrive from everywhere. Hospitals, garages, police, surveyors, brokers, customers — none of whom use your template, and many of whom send a photo of a printout.

They span decades. A claim on a 1998 policy needs the 1998 wording, which exists as a scan of a typewritten document with a coffee stain on it.

They're mostly unstructured on purpose. An adjuster's narrative captures nuance no dropdown ever could. That nuance is genuinely valuable — and genuinely unreadable by a model expecting a column.

What it costs you

Every use case stalls one layer down. Underwriting automation needs roof age — which is in a surveyor's PDF. Fraud detection wants prior claim narratives — free text in the claims system. Customer 360 wants correspondence history — attachments. Each project starts confident, hits the document wall, and quietly scopes down to whatever's already structured, which is the least interesting subset of what you know.

You're paying people to be OCR. A meaningful share of your operations headcount spends its day reading documents and typing what they say into fields. That's not underwriting or adjusting; that's transcription with a licence.

Your straight-through rate is capped by it. If the field a rule needs lives in a PDF, that case routes to a human. Every unextracted document type is a permanent manual queue.

Why the old approach failed

Most insurers tried this before, around 2015, with template-based OCR — and it disappointed. Worth knowing why, so you don't repeat it:

Template OCR works by knowing where on the page a value sits. It's brittle to the point of uselessness the moment a garage changes its invoice layout, a hospital uses a different form, or someone photographs a document at an angle. Teams built templates for the top five formats, covered maybe 40% of volume, and gave up on the long tail — which in insurance is most of the tail.

Modern document AI doesn't locate values by coordinates; it reads the document and understands what things are. "Total repair cost" is found because it means that, not because it's at x=420, y=310. That's the difference that makes the long tail tractable — and it's genuinely new since the last time you tried.

What a working document pipeline looks like

1. Ingest anything. PDFs, scans, photos, emails, faxes. If it needs a human to convert it before processing, you've reintroduced the bottleneck.

2. Classify first. Before extraction, know what it is: medical report, repair estimate, police report, policy wording. Extraction schemas differ per type; misclassify and everything downstream is garbage.

3. Extract to a schema, with confidence scores. Not "here's the text" — that's just a different unstructured blob. Extract to defined fields, each with a confidence value.

4. Route by confidence. This is the design decision that makes it work. High confidence flows straight through; low confidence goes to a human, who corrects it. Don't aim for perfect automation; aim for a tunable threshold.

5. Feed corrections back. The human corrections are training data. A pipeline that doesn't learn from them wastes the most valuable signal you have.

6. Keep provenance. Every extracted field must point back to the source document and page. When a regulator or a court asks where "roof age: 18 years" came from, "the model said so" is not an answer.

The trap: don't extract everything

The instinct is to digitise the archive — all forty years, all types, all fields. That programme takes three years and gets cancelled at eighteen months.

Invert it. Ask: which specific fields are blocking a decision right now? Usually it's a handful — roof age, construction type, prior claim cause, repair line items. Extract those, from the document types that carry them, for the cases you're actually processing. Ship value in six weeks instead of never.

The archive can wait. Nobody's asking for a searchable 1998 policy wording. They're asking why underwriting still takes three days.

The point

Everyone talks about the model. Almost nobody talks about the fact that the data the model needs is sitting in a scanned fax from a garage in Nashik.

Document intelligence isn't a glamorous use case. It's an enabling one — the thing that has to work before half your roadmap becomes possible. Carriers that treat it as infrastructure rather than a project find that several stalled initiatives quietly start moving at once.

We build the document pipelines and structured data foundations insurance AI depends on. More at IntelliBooks.