(Half A and Half B) we constructed the upgraded pipeline and watched it work: pointed at a analysis paper, a NIST commonplace, and a report with a damaged desk of contents, it returned a typed, cited reply every time. So why does it work? And would a naive RAG, the sort virtually everybody builds first, embed the pages, hold the closest few, ask, have executed the identical?
This text runs the 2 aspect by aspect. The quick reply isn’t any, and the helpful half is the place the naive one breaks: not in a single place, however at every of the 4 bricks in flip, doc parsing, query parsing, retrieval, and technology, every time for a motive you’ll be able to title and repair.
Additionally it is why the standard reflexes — rewrite the immediate, shrink the chunks, swap the embedding mannequin — don’t assist. The improper reply doesn’t come from the immediate. It comes from a brick upstream handing the mannequin the improper context, which the mannequin then solutions faithfully. Getting all 4 bricks proper, not simply the immediate, is what this sequence calls context engineering. Each failure under is an actual run, by no means an assumed one; the companion pocket book reproduces each. Article 7quinquies (most RAG hallucinations are retrieval failures) makes the identical case on the retrieval degree on one doc; this one exhibits it throughout all 4 bricks.
📓 The runnable pocket book for this text is on GitHub: doc-intel/notebooks-vol1. It runs the naive baseline and the upgraded pdf_qa aspect by aspect on every doc under, prints each solutions with their confidence, and reproduces each cross within the figures by yourself machine.
1. The naive baseline, brick by brick
The baseline is the 100-line pipeline from Article 1 (minimal RAG): parse the PDF, flip the query into key phrases, hold the highest few pages by match, ask the mannequin to reply. On a brief, clear, prose paper it really works, and Article 9’s personal checks confirmed it: on Consideration Is All You Want and the RAG paper, naive and upgraded each reply accurately.
The upgraded pipeline is the identical 4 bricks, every with a tighter contract: parsing returns a relational line_df as an alternative of flat textual content, query parsing expands the question into the doc’s vocabulary, retrieval routes on the desk of contents, technology returns a typed reply. A naive RAG runs the free model of each brick. The hole opens precisely the place every free brick fingers the mannequin the improper factor. Every part under isolates one brick: it swaps that brick for its naive model and leaves the others affordable, so the failure, and the repair, belong to the brick named within the heading.
Learn it prime to backside: the identical symptom, a assured improper reply, arrives 4 other ways, and a distinct brick closes every. The remainder of the article walks the 4, every on an actual doc.
2. Parsing: a desk flattened into noise
Doc: World Financial institution Commodity Markets Outlook (a report constructed round worth tables). Query: What’s the 2025 annual common worth forecast for U.S. pure fuel (Henry Hub)?
A naive pipeline parses a PDF the frequent means: dump flat textual content with get_text() and minimize it into fixed-size chunks. That’s wonderful for prose and deadly for tables. A worth desk is a grid; flat extraction linearises it, and a fixed-size chunker then cuts the stream wherever the character funds runs out. The row label Henry Hub lands in a single chunk and its 3.5 cell in one other. Retrieval fingers the mannequin chunks that by no means carry each, and it solutions actually: “not acknowledged in these traces,” confidence 0.00.
Nothing was hallucinated. The quantity was within the doc; the parser destroyed the one factor that made it a solution, the alignment between the label and the cell.
The repair is relational parsing. As a substitute of flat textual content, the upgraded brick returns a line_df: one row per textual content line, every with its bounding field. The desk’s row stays intact, the mannequin reads Henry Hub and 3.5 on the identical line, and returns “$3.5 per mmbtu” at confidence 0.99. That is the entire argument of the parsing brick (Article 5): cease returning flat textual content, hold the relational form the reply lives in.
3. Query: a phrase the doc by no means makes use of
Doc: NIST SP 800-207, Zero Belief Structure. Query: What are the pillars of zero belief structure?
The doc by no means makes use of the phrase pillars. It calls them tenets. A naive pipeline searches the pages for the question’s personal phrases, pillars scores in opposition to nothing, and the mannequin reviews “the precise pillars should not listed,” confidence 0.20. The reply is true there, below a distinct title; the appropriate pages have been by no means even searched.
This isn’t a rating downside, and no greater top-k fixes it: the question phrase merely isn’t within the doc. It’s a vocabulary downside, and it lives one brick earlier than retrieval, in query parsing. The consumer’s phrase and the doc’s phrase are synonyms the uncooked string can’t bridge.
The repair is query parsing. Earlier than retrieval runs, the brick normalizes and expands the question, mapping area synonyms (pillars → tenets, ideas) so retrieval searches for the phrases the doc truly makes use of. It then anchors the tenets part, and technology returns all seven at confidence 0.95, noting the doc’s personal time period. The identical enlargement lets a consumer ask a couple of canine and match a coverage that solely ever says animal, the case Article 7quinquies (most RAG hallucinations are retrieval failures) walks intimately.
4. Retrieval: the reply under the cutoff
Doc: NIST Cybersecurity Framework 2.0 (32 pages, native desk of contents). Query: How is a Profile outlined in CSF 2.0?
The phrase Profile is on many pages of CSF 2.0: it names a bit, recurs in prose, heads examples. Just one part defines it. Key phrase and cosine retrieval rank a web page by how related it seems to be to the time period, and each Profile web page seems to be equally related. The naive pipeline retains the highest few by frequency, none of which is the defining web page, and the mannequin reviews “not outlined in these traces,” confidence 0.10. The answering web page fell under the top-k cutoff.
The repair is retrieval that routes on construction. The upgraded brick reads the doc’s personal desk of contents by means of a small LLM, sees the part titled CSF Profiles, and anchors retrieval there as an alternative of on uncooked frequency. Era sees the defining paragraph and returns the total definition with a citable span, confidence 0.95.
The identical routing scales the place naive retrieval will get worse. On the 400-plus-page NIST SP 800-53 management catalog, requested for the one AU-2 Occasion Logging management, naive retrieval dilutes it previous the cutoff amongst a thousand near-identical siblings and returns “NA” at 0.00; the router locates the AU household and the AU-2 entry straight and returns the total requirement, clause by clause, at 0.98. Routing on construction doesn’t degrade because the doc grows; rating on frequency does.
5. Era: a assured reply with no self-check
Doc: World Financial institution Commodity Markets Outlook, April 2024 (its forecasts cease at 2025). Query: What’s the 2026 annual common worth forecast for crude oil (Brent)?
This time each pipelines retrieve the identical, right context: the vitality pages with the value desk. The doc merely has no 2026 row. A naive pipeline’s technology brick asks the mannequin for a free-text reply, and a free-text mannequin requested a query tends to reply. It grabs the closest quantity, the 2025 worth of $79, and returns “the 2026 Brent forecast is $79 per barrel”, assured, fluent, and improper. Nothing in a prose reply might have stated “this isn’t within the doc.” That is the hallucination readers report, and it occurred on the final brick, with good context in hand.
The repair is a typed technology contract. The upgraded brick doesn’t ask for prose; it asks for a schema with a complete_answer_found area the mannequin should set, an proof span it should cite, and a confidence it should justify. Confronted with a price that’s not within the traces, the mannequin can’t quietly fill the hole: it units complete_answer_found: false and returns “the 2026 forecast isn’t supplied; the newest is 2025.” The technology brick doesn’t make the reply smarter. It makes a lacking or partial reply seen, so a improper one can’t ship as executed. The identical complete_answer_found area guards partial solutions typically: every time retrieval surfaces solely a part of what the query requested, it’s the distinction between transport a fraction as if it have been complete and flagging the hole.
6. One root, 4 doorways
The 4 failures are one factor seen 4 methods. In none of them did the mannequin invent from nothing. Every time it answered the context it was handed, faithfully, and the context was improper: a desk the parser scrambled, pages discovered for a phrase the doc by no means makes use of, the answering web page left under the cutoff, or a price that was by no means there and acquired crammed in anyway. The label hallucination factors on the mannequin; the trigger was a brick upstream.
The frequent repair isn’t a greater immediate, a much bigger mannequin, or a bigger top-k. It’s context engineering, one contract per brick: hold the doc’s relational form (parsing), search within the doc’s personal vocabulary (query), route on the doc’s personal map (retrieval), and bind the reply to a typed, checkable contract (technology). Get the context proper and a assured improper reply has nowhere to return from, as a result of the mannequin is not being requested to reply the improper context.
Two sincere caveats. First, naive RAG isn’t at all times improper: on quick, clear, prose paperwork it retains up, and the sincere determine in Article 9 exhibits it. The hole opens precisely the place enterprise paperwork reside: tables, home vocabulary, size, construction. Second, these 4 crosses are those that held up. We ran the naive baseline and the upgraded pipeline on many extra (doc, query) pairs, and the place naive did wonderful we are saying so reasonably than manufacture a failure. Every cross above is an actual run, not a manufactured one.
7. Sources and additional studying
The failure circumstances are actual runs of the naive baseline (pdf_qa_baseline, plus a flat-chunk parser and a free-text generator for the parsing and technology bricks) in opposition to the upgraded pdf_qa (Article 9); each is an actual run, reproducible within the companion pocket book, not an assumed outcome. The responses.parse(text_format=Schema) sample behind the typed reply makes use of OpenAI’s Structured Outputs.
Earlier within the sequence:
- Doc Intelligence: sequence intro. What the sequence builds, brick by brick, and in what order.
- The minimal Enterprise RAG that by no means lies about its supply: PDF in, highlighted reply out. The naive four-brick baseline this text stress-tests.
- Retrieval is filtering, not search: a psychological mannequin for enterprise RAG. Retrieval reframed as filtering, the thought behind TOC routing.
- Most RAG hallucinations are retrieval failures (Article 7quinquies, hyperlink to return). The identical argument on the retrieval degree, on one doc, together with the vocabulary-mismatch case.
- A manufacturing RAG pipeline for PDFs: relational parsing, TOC retrieval, typed solutions (Article 9, hyperlink to return). The upgraded pipeline these circumstances run on.
Paperwork used (all brazenly licensed):
- World Financial institution, Commodity Markets Outlook, April 2024. The worth-table parsing case (part 2) and the absent-value technology case (part 5). CC BY 3.0 IGO.
- NIST, Zero Belief Structure, SP 800-207, August 2020. The vocabulary-mismatch case (part 3). US Authorities work, public area within the US.
- NIST, The NIST Cybersecurity Framework (CSF) 2.0, NIST CSWP 29, February 2024 (DOI 10.6028/NIST.CSWP.29). The buried-definition retrieval case (part 4). US Authorities work, public area within the US.
- NIST, Safety and Privateness Controls for Data Methods and Organizations, SP 800-53 Rev. 5. The diluted-control retrieval case at scale (part 4). US Authorities work, public area within the US.
