RAG Over Policy Documents: What Works and What Doesn't

"Just point an AI at our policy documents and let people ask questions." It's the most requested AI use case in insurance and one of the easiest to build badly.

Retrieval-augmented generation — RAG — genuinely works for this. But policy documents have properties that break the naive implementation, and the failure mode is dangerous: a confident, fluent, wrong answer about what someone's coverage includes.

Why the standard tutorial fails on policy wordings

The default RAG recipe is: chop documents into fixed-size chunks, embed them, retrieve the top few by similarity, stuff them into a prompt. On blog posts and FAQs, fine. On insurance contracts, it produces confidently wrong answers for four specific reasons.

1. Meaning depends on structure. A policy is a legal instrument where an exclusion three pages after a coverage grant completely changes what that grant means. Naive chunking retrieves the coverage paragraph, misses the exclusion, and the model cheerfully tells the customer they're covered. The chunk was retrieved correctly and the answer is still wrong.

2. Definitions live elsewhere. Wordings define terms — "Dwelling", "Occurrence", "Actual Cash Value" — in a definitions section, then use them throughout with specific legal meaning. A chunk using a defined term is meaningless without its definition, and similarity search won't pull the definitions section because it doesn't look similar to the question.

3. The right document matters more than the right paragraph. A claim on a 2019 policy must be answered from the 2019 wording, not the current one. If your index blends versions, you'll retrieve a passage that's perfectly relevant and legally inapplicable.

4. Absence is an answer. "Is flood covered?" is often answered by nothing being there. Retrieval finds what exists; it's structurally bad at proving a negative. A model that can't distinguish "not covered" from "I didn't retrieve it" will guess.

What actually works

Structure-aware chunking. Chunk on the document's own boundaries — sections, clauses, coverage parts — not on character counts. A chunk should be a semantically complete unit, and it should carry its ancestry (which section, which coverage part) in its metadata.

Attach definitions and exclusions to their coverage. At index time, link each coverage grant to the definitions it uses and the exclusions that qualify it. Then retrieving the grant retrieves its qualifiers automatically. This is index-time work that no amount of clever prompting substitutes for.

Filter hard before you search. Policy version, product, jurisdiction, and effective date should be metadata filters, not things you hope similarity handles. Retrieve only from the exact wording that applies to this customer on this date. This single change eliminates a large class of wrong answers.

Hybrid retrieval. Pure vector search misses exact terms — a specific clause number, a defined term, a peril name. Combine semantic search with keyword/BM25 and fuse the results. Insurance language is precise; you need both the meaning and the literal match.

Ground every answer in citations. The answer must quote and link the specific clause it came from. This isn't a nice-to-have UI feature — it's the mechanism that lets a human verify, and it's what makes the system defensible if the answer is challenged.

Teach it to refuse. The model must be able to say "I can't find that in this policy — here's who to ask." Every answer should trace to retrieved text, and unretrieved must mean unanswered. A language model's willingness to fill gaps with plausible text is a feature in creative writing and a defect in coverage questions.

Scope it by who's asking

The same technical system carries very different risk depending on the audience:

• Internal, assistive (helping an adjuster or underwriter find a clause faster) — low risk, immediate value, and a human validates every answer. Start here.
• Agent/broker-facing — medium risk; they're expert enough to catch nonsense, but they'll act on it.
• Customer-facing coverage answers — high risk. An answer about someone's coverage is a statement your company made. This needs citations, refusal behaviour, tight scoping, and logging of everything you told people.

Most teams jump straight to the third because it's the impressive demo. That's backwards — the internal version delivers most of the value at a fraction of the exposure, and it's how you learn where your retrieval is weak before a customer finds out.

The unglamorous prerequisite

None of this works if your documents aren't accessible and identified. If wordings are scanned PDFs, you need a document pipeline first. If you can't reliably determine which wording version applies to a given policy, your metadata filters have nothing to filter on. If policy-to-wording links are missing, you can't scope retrieval to the right contract at all.

So the sequence is the familiar one: get the documents structured and correctly linked to policies, then build retrieval on top. Teams that skip to the model produce a demo that impresses in the room and can't be trusted in production.

RAG over policy documents is genuinely one of the best AI applications in insurance. It just has to be built like the legal instruments it's reading — structurally, precisely, with citations, and with the humility to say "I don't know."

We build document pipelines, hybrid retrieval, and the governed data foundations behind them. More at IntelliBooks.