A Manufacturing RAG Pipeline for PDFs: Relational Parsing, TOC Retrieval, Typed Solutions

III of Enterprise Doc Intelligence, a collection that builds an enterprise RAG system from 4 bricks: doc parsing, query parsing, retrieval, and technology.

It’s the first of two elements on the upgraded pipeline: this half upgrades every brick, one contract at a time, on the identical paper and the identical query as Article 1 (minimal RAG). The second half, Composing the 4 RAG bricks into one pipeline, examined on actual paperwork (hyperlink to return), wires them into one name and runs it on a number of actual paperwork.

📓 Runnable companion notebooks are on GitHub: doc-intel/notebooks-vol1.

100 traces of Python wire 4 features collectively: parse the PDF, parse the query, retrieve a couple of pages, ask a mannequin.

That pipeline returns the precise reply on a clear query in opposition to a paper with a built-in desk of contents. On an actual corpus it breaks the primary time the person varieties “positonal encodig” with two typos, the primary time the doc is a 200-page contract with no PDF define, the primary time the query asks for each exclusion as a substitute of 1, the primary time downstream code desires a typed object as a substitute of a string. 4 bricks want an improve every earlier than the pipeline ships.

• Doc parsing now returns greater than a flat line_df: a relational set together with a TOC, page-level metadata, and a typed parsing_summary carrying the doc’s sort, language, and a one-paragraph abstract of what it’s about.
• Query parsing turns the noisy person enter right into a structured transient, with key phrases corrected in opposition to the corpus’s personal vocabulary and an inferred reply form (single worth, itemizing, desk).
• Retrieval reads the TOC the best way an skilled would: hand the entire TOC to a small LLM that picks sections by semantic relevance, then merge with the key phrase pages.
• Technology returns a typed reply with one citable span per merchandise, plus 4 context-quality indicators the pipeline reads to resolve whether or not to ship the reply or run one other go.

The working paper is a public 15-page arXiv submission, Consideration Is All You Want. The query, “What are the choices for positional encoding?”, is available in with two typos so the question-parsing brick has one thing to do. The output is what an enterprise person wants: a typed reply, with verbatim quotes tied to line ranges, and a full audit path from query to quotation.

1. The place the baseline RAG breaks

Decide a clear query on a clear PDF, run it by way of the only RAG pipeline: parse the PDF, extract key phrases from the query, retrieve a couple of pages, ask an LLM. On the Consideration paper with the query “What are the choices for positional encoding?”, that pipeline returns “sinusoidal positional encoding and discovered positional embeddings” with one contiguous span of pages cited. It labored, on a clear query, on a clear paper.

Drop the identical pipeline into an enterprise setting and 4 weak factors seem rapidly, one per brick:

• Doc parsing: the doc is flattened. Article 1 (minimal RAG) parsed the PDF into one flat listing of traces, sufficient to depend key phrases however nothing extra. The pages, the sections, the tables, all of the construction a downstream brick might scope on, thrown away at step one.
• Query parsing: the query customers sort isn’t clear. “What are the optoins for posiitional encoding?” has two typos. The baseline key phrase extractor by no means noticed the phrase positional, it noticed posiitional, so retrieval misses the very pages the reply lives on.
• Retrieval: the baseline by no means appears at construction. The Consideration paper carries a clear built-in TOC, named sections with web page numbers, three ranges deep. Probably the most exact retrieval sign in the entire doc, ignored.
• Technology: the reply comes again as a uncooked string. A listing query (choices) asks for one merchandise per possibility, every with its personal proof. Free-form prose forces the caller to re-parse the reply to search out the gadgets; a typed schema with a per-item proof span removes that step.

The 4 bricks under deal with these 4 weak factors. Identical paper, similar query, actual LLM calls. The output is a typed listing with verbatim quotes, line ranges, and the total chain of selections that produced it.

2. 4 bricks, upgraded

The form that survives the improve is identical 4 bricks Article 1 (minimal RAG) launched. What adjustments is the contract per brick: what every one consumes, what it produces, and the way the subsequent one consumes it. The diagram under is the total contract: each brick, the outputs it produces, and which downstream brick consumes every one (together with the parsing_summary facet channel into each query parsing and technology). The per-brick subsections then zoom into every field.

Every of the 4 bricks is upgraded in its personal articles, value studying for the total contract:

2.1 Doc parsing: a small relational set

Parsing runs as soon as and turns the PDF into the small set of tables each later brick reuses.

In: pdf_path, the PDF on disk. Out: line_df, page_df, toc_df, parsing_summary.

What comes out, one row per unit:

• line_df: one row per seen line (page_num, line_num, textual content, bounding field). The quotation unit.
• page_df: one row per web page (page_num, textual content). The coarse scan floor.
• toc_df: one row per part (title, stage, start_page). The doc’s personal map.
• parsing_summary: document-level metadata (doc sort, language, web page depend, format). The facet channel into the LLM bricks.

Article 1 (minimal RAG) parsed the PDF into one DataFrame known as line_df, one row per seen line of textual content. Sufficient for key phrase retrieval, not sufficient for anything. Article 5 (doc parsing) reframes parsing as constructing a small relational set: line_df stays, page_df aggregates traces to pages with their textual content, and toc_df carries the doc’s native desk of contents.

The bootstrap chunk on the high of the article already produced the three DataFrames on the Consideration paper. A take a look at every:

The page_num and line_num on every row are what flip a solution right into a quotation. Drawn again onto the web page they got here from, the rows are literal: each acknowledged line is one field, and its line_num sits within the gutter.

Aggregating the traces web page by web page provides page_df, the pure web page unit that carries whole-page textual content and page-level context:

toc_df carries the native define; on the Consideration paper it has three ranges and twenty-two entries, one row per part with its title, stage, and begin web page:

Three tables, similar form contract, similar numeric major keys. Downstream bricks learn what they want with out re-parsing the PDF; Retrieval (Part 2.3) scans key phrase hits and reads toc_df to anchor on the precise part, then sizes the context round it (the entire part, or a line window) on the granularity the query implies. page_df is the page-level scan unit, toc_df the map, line_df the atomic traces the anchor and window are reduce from. parse_pdf truly returns extra in the identical dict (picture areas, inner references, named objects) plus a parsing_summary carrying document-level metadata (doc sort, language, web page depend, format, typical fields, a brief abstract); this part focuses on the three tables retrieval reads, and parsing_summary returns within the second half because the facet channel that travels into the LLM bricks.

2.2 Query parsing: from noise to transient

Query parsing turns the uncooked person string right into a typed transient the subsequent two bricks can act on.

In: the uncooked query, the skilled’s concept_keywords_df, and parsing_summary for doc context. Out: a ParsedQuestion carrying intent, key phrases, a RetrievalQuery transient, and a GenerationBrief.

What comes out:

• key phrases: typos mounted and content material phrases pulled in a single LLM name, then expanded with the skilled’s vocabulary.
• intent: the reply form the query implies (factual, itemizing, comparability).
• RetrievalQuery: the transient retrieval consumes (main_query, rewrites, anchor_keywords, section_hint, layout_hint).
• GenerationBrief: solely the fields technology can act on (the query, the reply form, disambiguation).

Article 1 (minimal RAG) known as get_keywords_from_question on the clear query and bought a corrected_question plus a brief listing of key phrases again, in a single LLM name. That single name does two jobs directly: it fixes the floor typos and pulls the content material key phrases. The corrected key phrases come straight out of the parse, with no separate spell-checking go bolted on afterward. If a key phrase genuinely doesn’t exist within the doc, the restoration is retrieval’s suggestions loop (Article 13, the workflow pipeline), which re-searches with the phrases the doc truly makes use of, not a blind snap onto the closest corpus token.

What this text provides on high of the parse is the skilled vocabulary layer. The corrected key phrases are expanded with the area phrases a practitioner would additionally seek for, pulled from the skilled’s concept_keywords_df. Requested about positional encoding, the enlargement provides the 2 concrete strategies, sinusoidal and discovered, so retrieval matches them despite the fact that the query by no means named them.

Article 6 (query parsing) bundles each layers into the total parse_question brick. The output is a typed ParsedQuestion Pydantic carrying the person’s intent, the extracted key phrases, a RetrievalQuery sub-brief that retrieval consumes (main_query, rewrites, anchor_keywords, section_hint, layout_hint), and structural_hints for web page / sheet / slide pinning when the query carries them. The query’s intent additionally fixes how a lot to maintain across the anchor: the entire part when it maps to 1, or a line window for a pinpoint reality, so retrieval is aware of the granularity, not simply the place to look. Article 7A (retrieval as filtering) develops this two-level anchor / context mannequin in full. Part 1.2 of Article 6 (query parsing) develops the two derived briefs sample: the identical ParsedQuestion yields one transient for retrieval and a GenerationBrief for technology, every carrying solely the fields its client can act on.

Every step is specific, so retrieval downstream can use all of it and an audit log might be reconstructed.

The noisy query is identical one a annoyed person would sort:

The one name already corrected the typos: optoins and posiitional got here again as clear, content material key phrases, with no second spell-checking go bolted on. Now broaden them with the skilled’s vocabulary, the concept_keywords_df desk that maps every subject to the phrases a practitioner would additionally seek for:

{
  "original_question": "What are the optoins for posiitional encoding?",
  "key phrases": ["positional encoding"],
  "expanded_keywords": ["positional encoding", "sinusoidal", "learned"]
}

Two issues occurred in a single go. The key phrases got here again corrected: posiitional and optoins have been typos, and the one parse_question name mounted them whereas pulling the content material noun phrase, dropping framing phrases like choices. If a key phrase nonetheless didn’t exist within the doc, retrieval’s suggestions loop (Article 13, the workflow pipeline) would recuperate it, not a blind snap onto the closest corpus token.

The expanded key phrases come from concept_keywords_df. Requested about positional encoding, the enlargement provides the 2 concrete strategies the paper makes use of, sinusoidal and discovered, so retrieval matches them despite the fact that the query by no means named them. The desk is small on objective: every entry narrows on the subject, not a generic phrase like place that will pollute the search. Article 6 (query parsing) develops how it’s constructed and maintained.

For the question-parsing code intimately, see the three articles that develop the brick:

• Article 6A (the thesis): parse the query earlier than you search, the lacking step in most RAG pipelines.
• Article 6B (extraction): the 5 fields the parser pulls from any query (key phrases, scope, form, decomposition, clarification).
• Article 6C (dispatch): what the parsed query decides downstream (chunk technique, mannequin tier, fragments, audit path).

2.3 Retrieval: structured tables

Retrieval narrows the doc all the way down to the traces technology will learn. It filters on the structured tables, it doesn’t search a vector index.

In: the RetrievalQuery transient (from query parsing), plus line_df and toc_df (from doc parsing). Out: a RetrievalResult: the saved pages and filtered_line_df, the part or line window technology reads.

The way it works, in two phases:

• Anchor: key phrase hits counted per TOC part, then the LLM TOC router reads the define and picks the part that solutions the query.
• Context: sized across the anchor on the granularity the query implies (the entire part for an inventory, a line window for a pinpoint reality).
• filtered_line_df: simply the traces technology will learn, every carrying its page_num and line_num for citations.

Article 1 (minimal RAG) ran one retrieval methodology, key phrase matching on page_df, and saved the highest three pages by match depend. Article 7 (retrieval) reframes retrieval as a filter on the small relational set inbuilt Part 2.1: slim the candidate scope utilizing structured tables earlier than scoring key phrases. The key phrase methodology nonetheless runs, however toc_df opens a second sign. The pure method to make use of it’s not substring matching on titles. The writer of the doc already grouped traces into sections and wrote a title for every. A small LLM name can learn the entire TOC, motive about which part solutions the query, and return its picks with a one-sentence rationale.

Retrieval works in two phases (Article 7A, retrieval as filtering). First it finds the anchor: key phrase hits, counted per TOC part, inform the router which sections carry the query’s phrases; reason_on_toc reads the TOC plus these counts and picks the part. Then it sizes the context round that anchor, following the granularity the query implies (Part 2.2): the entire part for an inventory or part query, or a line window across the match for a pinpoint reality. The part is the pure context unit; page_df is the coarse scan floor, line_df the atomic traces the window is reduce from.

Right here, to remain incremental on Article 1 (minimal RAG), this text runs the 2 detectors and merges their pages; the manufacturing hybrid routes them into sections as a substitute (Article 7 (retrieval), the section-first arbiter above). The key phrase methodology runs first (low-cost, no LLM, deterministic). The LLM TOC router runs subsequent on the identical toc_df. The union of their pages goes to technology.

Article 7B (anchor detection) develops the LLM TOC router intimately (the immediate, the Pydantic output, why it beats substring matching on actual paperwork). Article 7C (the LLM arbiter) completes the image, rating each candidate web page from each detector in a single name. We use the standalone reason_on_toc right here as a result of it carries its weight by itself: the improve from substring is the one most impactful change a crew working Article 1 (minimal RAG)’s pipeline could make to retrieval.

Article 7 (retrieval) introduces the unified retrieve_context(query, line_df, *, methodology, top_k, ...) → RetrievalResult dispatcher that routes to key phrase / embedding / TOC / hybrid behind a single signature, and returns a typed RetrievalResult as a substitute of two tuples. We use the uncooked retrieve_pages and reason_on_toc right here to maintain this text incremental on high of Article 1 (minimal RAG); manufacturing code calls retrieve_context or the shared dispatch_page_retrieval helper.

A helpful verify earlier than trusting that desk. The primary matching line column scans line by line, one line at a time. That works for single-token key phrases. A multi-word key phrase like positional encoding might be damaged throughout a line break within the PDF, with positional on the finish of 1 line and encoding initially of the subsequent. Every line by itself incorporates neither phrase. The road-by-line scan finds nothing, even when the key phrase is true there on the web page.

Rely the misses throughout the doc:

The road is a PDF rendering artifact. The textual content the parser sees as one line is no matter suits between two visible line-break selections made by the PDF generator. There is no such thing as a semantic motive the unit of detection ought to map to that arbitrary boundary. The repair is to detect on a passage as a substitute, the place a passage is a small window of adjoining traces joined with an area. Any key phrase that exists within the passage might be discovered, regardless of the place the road breaks fall.

The page-level match_count was already right, as a result of retrieve_pages joins all traces on a web page earlier than scanning. The repair targets the line-grained helpers that the snippet column, the highlighting, and any later chunking step want. From right here on, each helper that scans under the web page stage makes use of a passage window, not a single line.

The LLM TOC router fills the opposite hole uncovered above. Identical toc_df from parsing, however the LLM reads it entire and causes about which part solutions the query, returning the picked part ids plus a one-sentence rationale. The operate itself is brief (format every TOC row, drop it in a immediate with the query, parse a typed SectionSelection again); Article 7B (anchor detection) exhibits it in full.

Run on the noisy query in opposition to the paper’s 22 TOC entries:

One LLM name, the entire TOC contained in the immediate, a typed listing of section_ids again with a sentence of reasoning. The substring matcher this text used to ship would have caught 3.5 Positional Encoding on this particular query as a result of encoding seems within the title. It could have missed each query phrased in another way from the writer. “What occurs if we exit early?” in opposition to a contract whose part is titled “Termination”: substring zero, LLM one. The associated fee is a small LLM name (a couple of thousand tokens for a typical TOC, a couple of hundred milliseconds), and it’s cached ceaselessly on an identical inputs.

This view is what an skilled reads. Not a flat web page listing with cosine scores, however the doc’s personal define, with the parsed-question key phrases marked the place they land. Article 7C (the LLM arbiter) consumes precisely this form as a structured transient, one row per candidate, and ranks them with per-candidate roles + causes. That arbiter is an improve on high of the LLM TOC router walked above, saved out of scope right here to remain incremental on Article 1 (minimal RAG).

For the retrieval code intimately, see the three articles that develop the brick:

2.4 Technology: a typed reply

Technology fills a typed schema from the retrieved traces. It’s managed execution in opposition to a contract, not free-form prose.

In: the GenerationBrief (query and reply form, from query parsing), filtered_line_df (from retrieval), and the reply schema. Out: a typed AnswerWithEvidence, or a ListAnswer when the query asks for an inventory.

What comes out:

• reply: formed to the query, one string for a single reality, one entry per merchandise for an inventory.
• evidence_spans: a line vary plus a verbatim quote per merchandise, checkable in opposition to filtered_line_df.
• high quality indicators: confidence, caveats, and a context_structured flag the LLM units when the retrieved textual content not reads so as.

Article 1 (minimal RAG)’s AnswerWithEvidence returns one reply: str, one contiguous proof span, a couple of quotes. Good for a query with a single reply (“What dataset was used?”). The working query asks for choices, plural. The baseline schema can listing them contained in the string, however the caller has no clear strategy to iterate over the gadgets or to attribute one quotation per possibility.

Article 8 makes the schema match the form of the anticipated reply. When the query parser flagged expected_answer_shape = "itemizing" (Article 12 (itemizing) develops these in depth), the technology brick returns a ListAnswer with one entry per merchandise, every carrying its personal proof span and its personal verbatim quote.

The form is what issues: one AnswerItem per possibility (its textual content, a begin/finish line-range span, and a verbatim quote), wrapped in a ListAnswer that additionally carries the answer-quality flags (answer_found, complete_answer_found, context_structured, confidence, caveats). Article 8 (technology) develops the schema in full.

Stuffed in on the noisy positional-encoding query, the schema returns two gadgets and the total set of high quality indicators:

{
  "gadgets": [
    {
      "text": "Sinusoidal positional encodings (sine and cosine functions of different frequencies).",
      "start_page_num": 6, "start_line_num": 33, "end_page_num": 6, "end_line_num": 37,
      "quote": "we use sine and cosine functions of different frequencies"
    },
    {
      "text": "Learned positional embeddings.",
      "start_page_num": 6, "start_line_num": 39, "end_page_num": 6, "end_line_num": 41,
      "quote": "we also experimented with using learned positional embeddings"
    }
  ],
  "answer_found": true,
  "complete_answer_found": true,
  "context_completeness": 1.0,
  "context_structured": true,
  "confidence": 0.98,
  "caveats": []
}

The 4 indicators carry the reply’s belief profile. complete_answer_found says whether or not the reply covers each possibility the doc mentions or solely a subset. context_completeness says how properly the retrieved traces coated the query, separate from whether or not the reply itself is true. context_structured flips to false when the LLM can’t comply with the studying order, the canonical OCR-failure sign. A downstream router reads them: excessive confidence + full + structured = ship the reply; low completeness or unstructured context = retry retrieval, or fall again to a deeper parse (the trail Article 10 develops).

context_structured is the indicator the article doesn’t get to see hearth on a clear paper. To show the LLM makes use of it, the identical traces fed in random order ought to flip it to false:

{
  "clear context": {
    "answer_found": true,
    "complete_answer_found": true,
    "context_completeness": 1.0,
    "context_structured": true,
    "confidence": 1.0,
    "n_items": 2,
    "caveats": []
  },
  "shuffled context (similar traces, random order)": {
    "answer_found": true,
    "complete_answer_found": true,
    "context_completeness": 1.0,
    "context_structured": true,
    "confidence": 0.95,
    "n_items": 2,
    "caveats": []
  }
}

In manufacturing, scrambled context comes from a distinct supply than df.pattern. PDFs with two-column layouts that the parser learn column by column as a substitute of row by row, scanned paperwork OCR’d by a mannequin that misplaced the format, exports from Phrase that broke lengthy tables throughout hidden anchors. The pipeline can’t self-correct on these, however the reply schema tells the caller one thing went improper, and a separate code path takes over.

Evaluate with the baseline RAG output. Identical query, similar paper, however the caller now will get a structured object: variety of gadgets identified up entrance, every merchandise independently citable, 4 high quality indicators that route the reply to the precise subsequent step. A UI renders the gadgets as a clickable listing. A SQL pipeline writes one row per merchandise. An annotated PDF highlights every quote on its supply web page. None of these customers needed to parse prose, and none of them ships a confidently-wrong reply as a result of the schema makes the failure modes seen.

2.5 Earlier than and after, per brick

4 bricks, 4 upgrades. Aspect by facet, the identical pipeline earlier than and after, one brick per row:

The 4 upgrades are impartial. A crew working Article 1 (minimal RAG)’s pipeline can undertake them separately: a TOC at parsing, a corpus-vocab spell-check at query parsing, an LLM TOC router + key phrase fallback at retrieval, a list-shaped schema at technology. Each is a small drop-in, none of them rewrites the encircling code.

3. Conclusion

It helps to see the place this go sits on an extended climb. The identical 4 bricks keep mounted; what adjustments from one rung to the subsequent is the place the management lives. This text is rung two: pdf_qa, a single richer go that already emits the suggestions fields (retrieval confidence, lacking key phrases) however does nothing with them but. Article 13 turns that go into pdf_qa_flow, the Quantity 1 composite that dispatches on query patterns and re-runs the go in a bounded loop, performing on precisely these fields. Quantity 2’s pdf_chat places a multi-intent entry in entrance. Solely the final rung fingers the management loop to the LLM itself.

4. Sources and additional studying

This half upgrades the 4 bricks (from Articles 5-8) one contract at a time, on the Consideration Is All You Want paper. The second half composes them finish to finish and runs the assembled pipeline on a number of paperwork. The responses.parse(text_format=Schema) sample on the question-parsing and technology boundaries makes use of OpenAI’s Structured Outputs (Aug 2024). The closest revealed production-grade write-up of this sort of pipeline is Anthropic’s Contextual Retrieval (Sept 2024). The agentic improve path on high of the identical 4 bricks is follow-up work; the per-brick provenance retains the agent’s selections auditable.

Earlier within the collection:

What works, what breaks

• Baseline Enterprise RAG, from PDF to highlighted reply. The four-brick pipeline finish to finish: PDF in, highlighted reply out.
• Embeddings Aren’t Magic: The Predictable Failure Modes of RAG Retrieval. The place embedding similarity wins (synonyms, typos, paraphrase), the place it predictably breaks (unknown phrases, negation, term-vs-answer relevance), and methods to use it anyway.
• RAG just isn’t machine studying, and the ML toolkit solves the improper downside. Why chunk-size sweeps and finetuning optimize the improper factor; route by query sort as a substitute.
• From regex to imaginative and prescient fashions: which RAG method suits which downside. Two axes, doc complexity and query management, that decide the method for every case.

Doc parsing

• Past extract_text: the 2 layers of a PDF that drive RAG high quality. The primary half of the parsing brick: the doc’s nature, indicators, and abstract.
• Cease returning flat textual content from a PDF: the relational tables RAG wants. The second half of the parsing brick: the relational tables each downstream brick reads.
  • When PyMuPDF can’t see the desk: parse PDFs for RAG with Azure Format. The identical tables from Azure Format: native desk cells, OCR, paragraph roles.
  • Parse PDFs for RAG regionally with Docling: wealthy tables, no cloud add. The identical tables computed regionally with Docling: TableFormer cells, nothing leaves the machine.
  • Imaginative and prescient LLMs are PDF parsers too: studying charts and diagrams for RAG. Imaginative and prescient as a parser: the images grow to be searchable textual content.
  • Parse scanned PDFs for RAG with EasyOCR: free OCR provides you phrases, not a doc. The place conventional OCR stops: textual content recovered, construction misplaced.
  • Making a PDF’s pictures searchable for RAG, with out paying to learn all of them. The picture cascade: filter low-cost, classify, describe solely what’s value studying.
  • Reconstructing the desk of contents a PDF forgot to ship, so RAG can scope by part. Rebuilding toc_df when the PDF prints a contents web page however ships no define.

Query parsing

• RAG questions want parsing too: flip the person’s string into briefs for retrieval and technology. The thesis of query parsing: why a person string wants the identical parsing as a doc, and the way it splits right into a retrieval transient and a technology transient.
• What the query parser extracts from a person string: key phrases, scope, form, decomposition, clarification. The 5 households of columns the parser reads straight from the person’s query, with the code that fills every one.
• Dispatching the parsed RAG query: chunk technique, mannequin tier, activations, audit. The selections the parser makes on high of the person string, utilizing the doc’s profile: dispatch, activations, full schema, the audit path (pipeline_trace.json), and a broker-corpus walkthrough.

Retrieval

Technology

• Make RAG technology return a typed contract: citations, typed values, and self-checks (hyperlink to return). The reply schema because the contract: typed values, gadgets with proof spans, self-assessment fields, and the completeness sign the pipeline computes itself.
• Assemble every RAG technology immediate from a base immediate plus the foundations every query wants (hyperlink to return). The dispatcher: a set BASE immediate plus the foundations every query wants, the schema picked from the registry, and the total hint saved on each name.
• Validating the RAG reply earlier than the person sees it: spans, quotes, and the suggestions loop (hyperlink to return). The post-generation validator (spans, verbatim quotes, codecs), not-found as a first-class reply, and the suggestions loops that shut the pipeline.

Identical route because the article:

• Anthropic, Contextual Retrieval (Sept 2024 engineering put up). The closest revealed “minimal however production-grade” improve write-up; lands on hybrid retrieval + reranking, enhances the TOC-aware brick improve on this article.
• OpenAI, Structured Outputs. The responses.parse(text_format=Schema) sample used on the question-parsing and technology boundaries.
• Vaswani et al., Consideration Is All You Want, NeurIPS 2017 (arXiv:1706.03762). The paper this half runs each brick on. arXiv non-exclusive distribution license, declared on the arXiv summary web page.
• NIST, The NIST Cybersecurity Framework (CSF) 2.0, NIST CSWP 29, February 2024 (DOI 10.6028/NIST.CSWP.29). A compliance doc the assembled pipeline is examined on within the second half. US Authorities work, public area within the US, see the NIST copyright assertion.
• Lewis et al., Retrieval-Augmented Technology for Data-Intensive NLP Duties, NeurIPS 2020 (arXiv:2005.11401). The RAG paper itself, a take a look at for the assembled pipeline within the second half. arXiv non-exclusive distribution license, declared on the arXiv summary web page.
• World Financial institution, Commodity Markets Outlook, April 2024 situation. The degenerate-TOC stress take a look at (clean bookmark titles), within the second half. CC BY 3.0 IGO, as declared on the OKR publication web page for April 2024.

Runnable code paths name OpenAI companies ruled by OpenAI’s Phrases of Use.

Totally different angle, completely different context:

• Yao et al., ReAct: Synergizing Reasoning and Appearing in Language Fashions, ICLR 2023 (arXiv:2210.03629). Founding paper of agentic RAG. The context is general-purpose tool-picking at runtime. Growing this line, the place the 4 upgraded bricks grow to be the agent’s audited toolkit, is follow-up work.
• Lee et al., Can Lengthy-Context Language Fashions Subsume Retrieval, RAG, SQL, and Extra?, 2024 (arXiv:2406.13121). The long-context-replaces-RAG improve path: skip parsing, skip retrieval, dump the entire doc in. Empirical information on the place this works and the place it breaks.