Most RAG Hallucinations Are Extraction Errors: Seven Patterns for a Typed Technology Contract

The naive pipeline asks the LLM for a solution and trusts what comes again. There isn’t any separation between discovered and invented, no place to file confidence, and no method to flag not discovered as a first-class consequence. We fill the schema as a substitute: each area is a query the pipeline asks, and each reply is checkable.

Under are the seven patterns that maintain technology on the typed-contract aspect. The primary six are the contract itself. The seventh names a constraint the contract should respect when the mannequin is small.

The reply schema is a contract: structured retrieval in, a typed Pydantic object out, with citations, constancy flags, and pipeline-feedback fields, by no means “the reply as a string”. The whole lot the technology brick does is in service of filling that contract, area by area.

Take “what’s the premium quantity?”. The naive baseline returns a sentence: "The premium is $124 monthly, per the contract.". To make use of that downstream you must parse the quantity again out of the prose, and nothing tells you the place it got here from or how certain the mannequin was. The sequence returns a row as a substitute: Quantity(worth=124.0, foreign money="USD", unit="month", proof=[Span(line_start=42, line_end=42, quote="premium of $124 / month")], answer_found=True, confidence=0.95). The worth is already typed, the proof factors on the actual line, and the self-assessment fields journey with it. The caller reads structured values, not textual content it has to re-read.

Ask for Quantity(worth, foreign money, unit) and let Python examine with a visual trade charge; computation contained in the mannequin erases the audit path and silently does conversions no human can replay. The LLM extracts, Python computes.

Take “is the premium increased than 100 euros monthly?”. Ask the LLM instantly and it solutions "sure, the premium is round 130 EUR / month", having transformed $124 to euros with a charge it by no means exhibits you. You can’t verify the arithmetic, and subsequent month the identical query would possibly convert at a unique invisible charge. The sequence extracts the uncooked worth as a substitute, Quantity(worth=124, foreign money="USD", unit="month"), and lets Python do the comparability with a charge that sits within the audit log, 1 USD = 0.92 EUR. That offers 114 euros, so the reply is True, and each step, the extraction, the speed, and the comparability, could be replayed and checked by hand.

The LLM contained in the retrieval scope can’t see whether or not the subsequent web page continues an inventory; retrieval pulls one further web page and the pipeline checks for a piece boundary, so completeness is deterministic and grounded in construction. A mannequin that may solely see what was despatched to it can’t decide what was lacking.

Take “record all exclusions”. The exclusions part runs from web page 7, and the subsequent part, “Limitations”, begins on web page 10, so the reply lives on pages 7 to 9. Naive RAG fingers these pages to the LLM and asks “is the record full?”. The mannequin can solely see what it was given, so it reads pages 7 to 9 and says sure, with no method to know whether or not a tenth exclusion was ever retrieved. The sequence checks the construction at retrieval time as a substitute: does the subsequent web page nonetheless belong to the exclusions part? If it does, fetch it; as soon as it crosses into “Limitations”, cease. Completeness turns into a truth about the place the part ends, not a guess the mannequin makes about textual content it can’t see.

Off-corpus, partial, and full every path to a unique subsequent motion, and two flags power clearer selections than a single scale. The float-confidence body collapses three completely different outcomes into one quantity.

Take “record all exclusions” once more, however this time retrieval discovered web page 7 and missed web page 8. A single confidence = 0.6 tells the orchestrator nearly nothing: it can’t inform whether or not two of three exclusions got here again or whether or not the entire reply was invented. The sequence splits that into two booleans. answer_found = True says the query is answerable from this doc, and complete_answer_found = False says the record that got here again is partial. These two flags map to 3 clear subsequent strikes: maintain retrieving when the reply is discovered however incomplete, ship when it’s discovered and full, and refuse when it’s not discovered in any respect. A single float collapses these three outcomes into one quantity and loses the choice.

A dispatcher composes BASE + one form fragment + elective constraints and data which fragments utilized, so a unsuitable format six months later is traceable, in contrast to a mega-prompt that grows uncommented clauses. A single immediate that everybody retains appending to is a typical RAG codebase odor; composing it from named fragments is what retains every reply’s immediate reconstructable.

Take “efficient date?”, whose reply form is Date. A naive baseline runs one generic immediate that at all times asks for “a solution”, regardless of the query. The sequence composes the immediate from elements as a substitute: the BASE immediate, plus a Date form fragment that claims "return YYYY-MM-DD", plus the DateAnswer schema. The audit log data precisely which elements had been used, prompt_id=BASE@v3 + form=date_v2. So when the mannequin returns “July 1st 2026” six months later and somebody has to elucidate the unsuitable format, the staff can rebuild that actual immediate from the log and reproduce the bug, as a substitute of guessing which clause of a thousand-line mega-prompt fired.

The schema already constrains the work, so the additional “considering” provides latency and wanders off the schema with out including precision. Reasoning fashions earn their place on open-ended duties; on schema-constrained extraction, they’re over-engineered.

Take “what’s the deductible quantity?”. A staff hoping for cleaner extraction switches to a reasoning mannequin. It thinks for eight seconds and returns the identical quantity a plain mannequin returned in a single, as a result of the schema had already pinned the output to a single typed area. The reasoning tokens price latency and cash and acquired nothing, since there was no open-ended judgment to make. The stance that follows is straightforward: use the smallest mannequin that fills the schema reliably, and save reasoning for the steps that genuinely want it, such because the LLM arbiter on the finish of retrieval.

A frontier mannequin can fill a compound schema (extract + convert + format) in a single name; a small mannequin can’t, and asking for it invents the derived fields silently. Mannequin dimension units the granularity of the contract, not the form of it.

Take “is the premium increased than 100 euros monthly?” with a compound schema, PremiumComparison(raw_amount: Quantity, converted_eur: float, exchange_rate: float, over_100_eur: bool). GPT-4.1 fills each area accurately in a single name: it extracts $124/month into raw_amount, converts to 114.08 EUR with a visual charge, and returns over_100_eur=True. Hand the identical schema to llama-3.2-3B and the mannequin returns raw_amount populated, converted_eur invented at 117.6 (from an imaginary 0.95 charge the log by no means carries), and over_100_eur=True derived from that invented charge. The small mannequin doesn’t fail JSON validation. It fills each area. Three of the 4 are fabricated.

The repair is to decompose the contract into levels the small mannequin can deal with in isolation. First name: extract Quantity(worth=124, foreign money="USD", unit="month"). Python takes over: charge lookup (logged, 1 USD = 0.92 EUR), conversion (124 * 0.92 = 114.08 EUR), threshold (114.08 > 100 = True). Non-compulsory second name if a natural-language abstract is required: hand the finished compound row again with a plain format immediate. Identical last row, identical accuracy, no fabricated intermediate.

The rule: the scale of the mannequin determines the variety of extraction calls, not the form of the schema. Massive mannequin = one dense name. Small mannequin = a sequence of typed extracts with Python computing within the center and one elective format name on the finish. The typed contract survives both manner; solely the granularity adjustments.

Article B05 (choosing a mannequin) benchmarks the granularity trade-off throughout a fleet of 13 fashions (hyperlink to come back).

The seven patterns share one transfer: refuse the LLM-as-oracle body, and deal with technology as filling a typed contract. The schema is the contract; citations and self-assessment fields are first-class outputs; the validator runs earlier than the consumer sees something. Sample 7 provides the operational nuance: the identical contract holds throughout mannequin sizes, however a small mannequin wants it damaged into levels the mannequin can fill with out inventing derived values. The deep-dives (8A, 8B, 8C) ship runnable code on actual paperwork; this piece is {the catalogue} that names every sample and factors to the code that implements it.

The typed-contract self-discipline (Pydantic schema + citations + self-assessment fields) holds in each area. What adjustments per query is the schema form. The contract sample stays the identical. 5 sectors under, 5 reply shapes, one validator operating the identical checks throughout all of them.

The validator runs the identical checks throughout all 5 rows: line spans inside doc bounds, verbatim quotes match cited strains, format constraints revered (a Period really parses as a length, an Quantity carries foreign money and unit). The authorized row demonstrates the two-boolean break up (Sample 4): the reply was discovered on this doc however will not be full with no cross-reference, so the orchestrator continues retrieval as a substitute of transport a partial reply. The medical row exhibits the audit path (Sample 1): the caveat captures a known-unknown the LLM couldn’t resolve.

The numbered articles develop every sample in code, with runnable notebooks:

Most writing on LLM technology comes out of chatbots and open-ended textual content. The sequence assumes one thing completely different: the reply should be auditable, and the LLM should refuse to invent.