Loop Engineering for Cross-References: When RAG Solutions ‘see Part 7.2’ As a substitute of the Precise Reply

The reference occurs at parse time (the doc writer writes it), at retrieval time (a bit containing the reference matches the question), and at technology time (the mannequin writes “see Part X” as an alternative of fetching Part X). The pointer might be resolved at any of those three moments, and is silently dropped at every. The repair is a suggestions subject on the technology output that flags an unresolved reference, plus an orchestrator that catches the sign, resolves the reference in opposition to the parsing brick’s relational tables, re-retrieves the linked area, and re-runs technology with the linked context now in hand.

The architectural level is that the orchestrator loops on alerts from the structured output, not on confidence scores. The identical equipment utilized in Article 10 for parsing high quality is used right here for references. Article 12 applies it to a 3rd dimension (itemizing completeness); Article 13 (the RAG workflow) names the sample explicitly and reveals how the patterns compose.

Parsing is the place references are initiated, however cheaply. Native PDF hyperlinks (the sort you click on on in a PDF viewer) come free of charge from the parser and go into the cross_ref_df desk at zero value. Something past that, regex detection of “see Part 4.2”, glossary tagging, conditional-clause flagging, is not finished upfront. The vocabulary is simply too assorted (“cf.”, “see”, “per”, “as outlined in”) and the quantity too excessive to extract all the pieces earlier than figuring out what shall be wanted.

Query parsing doesn’t change. The consumer’s query is parsed into the same old question_df plus satellites (Article 6). References don’t require a brand new query form.

Retrieval doesn’t change on the primary cross. It runs the usual retrieval from Article 9 (TOC pages plus key phrase pages, merged). The references that exist within the candidate passages aren’t but resolved.

Technology is the place the brand new contract seems. The Pydantic schema provides two fields: a listing of pending_references (references the LLM detected within the candidates that appear to be they’d change the reply if resolved) and an answer_completeness flag (full, references_unresolved, or partial). These two fields are the suggestions sign.

The orchestrator (the determine.py from Article 13) reads the structured output and decides whether or not the reply is shippable or whether or not a second cross is required.

Query parsing (Article 6) extracts the key phrases discovered positional embeddings, sinusoidal, positional encoding, marks the anticipated reply form as “comparability”, and units the scope filter empty (the query is about the entire doc).

Retrieval runs the usual stack from Article 9 (TOC pages plus key phrase pages, merged), and we intentionally take solely the top-1 web page. On this query the highest hit is web page 6 (Part 3.5 Positional Encoding). Web page 9 (Desk 3) is the second-ranked candidate however shouldn’t be fetched beneath a top-1 coverage. Part 6 defends the top-1 selection; for the remainder of this part, take it because the coverage.

The one candidate (web page 6) accommodates a reference to “Desk 3 row (E)” in its prose. On the primary cross we don’t resolve it. We cross web page 6 to technology and let the LLM flag what it finds.

The technology schema on this article extends the one from Article 8 with two fields: pending_references and answer_completeness. The LLM is prompted to populate each: if any candidate passage factors to a area of the doc that was not retrieved, the LLM lists that pointer in pending_references and units answer_completeness to "references_unresolved".

class ReferenceAwareAnswer(BaseModel):
    reply: str                                     # the prose reply
    citations: checklist[Citation]                       # line-level provenance

    # Reference-loop suggestions fields (the contribution of this text).
    pending_references: checklist[PendingReference]      # what to resolve in cross 2
    answer_completeness: Literal["complete",
                                 "references_unresolved",
                                 "partial"]         # the routing sign


class PendingReference(BaseModel):
    raw_text: str             # "Part 5.2", "Desk 3 row (E)", "SP 800-161r1"
    ref_type: str             # "part" | "desk" | "determine" | "exterior" | ...
    origin_page: int          # web page the place the reference was talked about
    origin_line: int          # line the place the reference was talked about

On this query, the LLM produces this:

{
  "reply": "In accordance with the Transformer paper, the authors experimented with discovered positional embeddings as an alternative choice to the sinusoidal encodings used of their base mannequin. They report that the 2 variations produced practically equivalent outcomes. The detailed comparability is in Desk 3 row (E) of the paper.",
  "citations": [
    {"page": 6, "line_start": 22, "line_end": 30, "retrieved_via": "primary"}
  ],
  "pending_references": [
    {"raw_text": "Table 3 row (E)", "ref_type": "table", "origin_page": 6, "origin_line": 28}
  ],
  "answer_completeness": "references_unresolved"
}

The prose half is what the naive pipeline would have returned and stopped. It’s a half-answer: qualitatively right however lacking the numbers, that are pointed to however not retrieved. The 2 structured fields on the backside are what makes the second cross doable.

The orchestrator runs after technology. It reads answer_completeness first. If full, the reply is returned to the consumer. If references_unresolved and the loop depend is beneath the price range (1 by default, like Article 10’s adaptive re-parse), it triggers the second cross.

def decide_next_pass(reply: ReferenceAwareAnswer,
                     loop_count: int,
                     max_loops: int = 1) -> str:
    """Return one in all: 'return_to_user', 'resolve_references', 'give_up_partial'."""
    if reply.answer_completeness == "full":
        return "return_to_user"
    if reply.answer_completeness == "references_unresolved":
        if loop_count < max_loops and reply.pending_references:
            return "resolve_references"
        return "give_up_partial"
    return "return_to_user"

For our instance, the choice is resolve_references. Loop depend is 0, the price range permits 1, the pending_references checklist has one entry pointing at “Desk 3 row (E)”.

Learn decide_next_pass as a loop and its three management surfaces are all seen within the code. The set off is a typed subject (answer_completeness == "references_unresolved" with a non-empty pending_references), by no means a confidence rating. The termination is the loop price range (max_loops = 1) plus the 2 exit verdicts (return_to_user, give_up_partial). The restoration modifications one factor earlier than the retry: the context now consists of the resolved goal area. Article 10’s adaptive re-parse has the identical three surfaces with completely different subject names: each are huge loops of the identical household, crossing bricks on a typed flag.

The second cross is the place the precise reference work occurs. For every merchandise in pending_references, the resolver tries a budget path first (deterministic lookup in opposition to toc_df and the item registry), and falls again to the LLM solely when a budget path fails.

The resolver normalizes the raw_text and appears it up. “Desk 3 row (E)” normalizes to the important thing (desk, 3) plus the row tag (E). The object_registry constructed at parsing time accommodates (desk, 3) mapped to its location on web page 9. The row tag is carried alongside as a sub-selector that the second-pass retrieval makes use of to give attention to row (E) when the complete desk is fetched. That could be a one-line question in opposition to the registry, deterministic, free.

Part references work the identical means: “Part 3.2.2” turns into the lookup key 3.2.2, which toc_df matches to the “Multi-Head Consideration” part. Determine and annex references observe the identical sample.

The deterministic path covers numbered references. It doesn’t cowl ambiguous phrasings: “see the earlier part”, “as mentioned earlier”, “per the foregoing”, “consistent with the framework launched above”. These want a small LLM name that takes the origin context (the road the place the reference appeared, plus a couple of traces of surrounding context) and the toc_df of the doc, and returns one of the best matching part.

def resolve_pending(ref: PendingReference,
                    toc_df, object_registry, line_df,
                    llm_client=None) -> ResolvedReference:
    """Resolve one pending reference. Attempt low cost first, fall again to LLM."""
    # 1. Deterministic : numbered part lookup.
    if ref.ref_type == "part":
        section_id = _section_id_from_text(ref.raw_text)  # "Part 5.2" -> "5.2"
        match = toc_df[toc_df["section_id"] == section_id]
        if not match.empty:
            return ResolvedReference.from_toc(ref, match.iloc[0])

    # 2. Deterministic : object registry for tables, figures, annexes.
    if ref.ref_type in ("desk", "determine", "annex"):
        obj_id = _object_id_from_text(ref.raw_text)  # "Desk 3 row (E)" -> "3"
        obj = object_registry.get((ref.ref_type, obj_id))
        if obj shouldn't be None:
            return ResolvedReference.from_object(ref, obj)

    # 3. Exterior : flag as out-of-corpus, don't re-retrieve.
    if ref.ref_type == "exterior":
        return ResolvedReference.exterior(ref)

    # 4. Ambiguous phrasings : ask the LLM with origin context and toc_df.
    if llm_client shouldn't be None:
        origin_context = _surrounding_lines(line_df, ref.origin_page,
                                            ref.origin_line, window=3)
        guessed_section = _llm_resolve(llm_client, ref.raw_text,
                                       origin_context, toc_df)
        if guessed_section shouldn't be None:
            return ResolvedReference.from_llm(ref, guessed_section)

    return ResolvedReference.unresolved(ref)

For the one pending reference in our instance:

• “Desk 3 row (E)” resolves deterministically in opposition to the object_registry to web page 9, with the row tag (E) saved as a sub-selector so retrieval can give attention to the precise row of the desk.

The pipeline re-retrieves the traces from web page 9 (the Desk 3 space, with the row (E) sub-selector utilized), joins them with the unique major candidate (web page 6), and re-runs technology. The schema is similar ReferenceAwareAnswer, however the immediate now consists of the newly-retrieved content material with metadata that tells the LLM “this passage was pulled in due to the reference ‘Desk 3 row (E)’ on web page 6, line 28”.

The second-pass reply:

{
  "reply": "The Transformer paper compares discovered positional embeddings in opposition to the sinusoidal positional encodings used within the base mannequin. Each produce primarily the identical efficiency on the English-to-German growth set newstest2013 : 4.92 perplexity and 25.7 BLEU for discovered embeddings (Desk 3 row (E)), versus 4.92 PPL and 25.8 BLEU for the bottom mannequin with sinusoids. The 0.1 BLEU distinction is nicely inside noise on this benchmark, confirming the authors' declare that the 2 variations produced practically equivalent outcomes.",
  "citations": [
    {"page": 6, "line_start": 22, "line_end": 30, "retrieved_via": "primary"},
    {"page": 9, "line_start": 38, "line_end": 39, "retrieved_via": "reference", "referenced_from": [6, 28], "reference_text": "Desk 3 row (E)"}
  ],
  "pending_references": [],
  "answer_completeness": "full"
}

The orchestrator reads answer_completeness = "full" and returns the reply to the consumer. The loop depend is 1, the price range was 1, no additional iteration is allowed even when technology had requested for one.

Two cheap-default selections made the instance above work: the parser doesn’t pre-extract each reference, and the retriever returns solely the top-1 web page. Each observe the identical self-discipline that Article 10 utilized to parsing: begin low cost, let technology sign what’s lacking, broaden solely on the sign. This part makes each decisions express and defends them.

The intuition when implementing that is to do a regex cross at parse time and populate cross_ref_df with each detected reference upfront. Identical intuition Article 10 had to withstand for parsing: run the costly parser in all places simply in case. Three arguments in opposition to.

Quantity: A 200-page contract could include a whole bunch of in-prose references. Most won’t ever be wanted as a result of retrieval won’t ever return the passages they sit in. Pre-extracting all of them is waste.

Vocabulary: “See”, “per”, “cf.”, “as mentioned in”, “consistent with”, “topic to”, “in accordance with the foregoing”. Every doc household invents its personal variants. A regex that covers all of them is both large and brittle, or slim and incomplete. Each are unhealthy. The LLM, requested at decision time with the origin context in hand, handles the paradox accurately.

Upkeep: A pre-extracted cross_ref_df turns into another artifact the workforce has to maintain present as new doc households are available in. A resolver that runs on demand reads the parsed constructions (toc_df, object_registry) that exist already and asks the LLM for the remaining. No new artifact to keep up.

The exception, talked about in part 3.1, is a budget circumstances: native PDF hyperlinks and structural anchors derived from toc_df. These value nothing at parse time and are value populating upfront as a result of they assist deterministic decision and not using a regex or an LLM name.

The second low cost default is the retrieval coverage. We ship the LLM solely the top-1 web page, not the top-3 or top-5. The intuition in most RAG tutorials is to default to top-3 or top-5 “to be protected”; the argument right here goes the opposite means.

Enterprise solutions are often distinctive. A selected query on an enterprise doc has one canonical passage that solutions it. The premium for water injury lives in a single row of 1 schedule. The indemnification cap is in a single clause. The Transformer’s positional-embedding comparability is in a single row of 1 desk. Pulling top-3 by default often means pulling one good passage plus two weak ones that eat context price range and confuse technology.

The loop catches what top-1 misses, when the miss issues. If the top-1 passage is a stub that factors elsewhere (the “see Desk 3 row (E)” form), the cross-reference loop fetches the goal deterministically. If top-1 is incomplete in a means that doesn’t flag a reference, the orchestrator’s different alerts (parsing high quality, completeness) catch it on a distinct axis.

Price provides up throughout queries: The Transformer instance above value two LLM calls (first cross on web page 6, second cross on web page 6 plus web page 9 area). A top-3 first cross that already included web page 9 would have value one LLM name however with three pages of context, two of which contributed nothing to the reply. Throughout 1000’s of queries the second sample wastes extra tokens than the primary.

The exception is questions the place the reply is in a number of locations: “each clause that mentions termination” (a list query, Article 12) or “all cross-references to Annex A” (a sweep, additionally Article 12). For these, top-k is the precise coverage. For factual questions of the shape “what’s X”, top-1 plus loop is the default.

The identical form recurs in each article of Half III. Article 10 retains low cost parsing because the default and escalates on a context_structured = False sign. Article 11 retains low cost reference initiation and top-1 retrieval because the default, and expands on a pending_references sign. Article 12 retains a small candidate set because the default and expands on a completeness sign. Article 13 (the RAG workflow) names the sample explicitly and composes the three: the orchestrator reads typed suggestions fields from the structured output and routes the pipeline accordingly. Low cost default, signal-driven enlargement, bounded iteration. These generation-triggered loops are the huge loops of the pipeline, those that cross bricks; every brick additionally runs its personal small bounded loops inside (the picture cascade in parsing, the TOC descent in retrieval, the schema retry in technology), and the 2 scales observe the identical three-surface self-discipline.

The instance walked via a desk reference (“Desk 3 row (E)”) resolved deterministically in opposition to the item registry. Three different kinds seem in enterprise paperwork and observe the identical loop construction with small tweaks to the resolver.

Part references: “See Part 4.2”, “As described in Part 3.2.2”. The resolver makes use of toc_df as an alternative of the item registry. Re-retrieval pulls the traces from the resolved part’s web page vary. That is the canonical case in contracts and requirements, the place a clause delegates to a different clause by part quantity.

Conditional clauses: “This is applicable provided that X”, “within the case the place the coverage is renewed”. The technology schema flags these too, however the loop conduct is completely different. The pipeline doesn’t attempt to resolve the situation (no automated logical reasoning). It re-runs technology with an instruction to state the situation explicitly within the reply, so the consumer sees “the rule applies if the coverage is renewed inside 30 days” as an alternative of a solution that silently assumes the situation is met.

Definitional references: “Insured Social gathering (as outlined in Part 1.3)”, “Multi-Head Consideration (outlined in Part 3.2.2)”. The resolver pulls the definition into the candidate set on the second cross. This prevents the LLM from importing a generic notion when the doc defines the time period particularly. Definition passages are quick, so the second-pass context value is low.

Exterior references: “see SP 800-161r1”, “per the necessities of ISO 27001 Annex A”. These level to paperwork exterior the corpus. The resolver flags them as exterior and the technology step names them explicitly, so the consumer is aware of there’s a supplementary useful resource with out the pipeline pretending to have fetched it.

The cross-reference loop is similar architectural sample as Article 10’s adaptive parsing: begin low cost, let technology sign what’s lacking via typed fields, and react to the sign in a deterministic orchestrator somewhat than an autonomous agent. The pending references resolve in opposition to the relational tables parsing produced (deterministic when doable, LLM-assisted when ambiguous); the second cross re-retrieves and re-generates with the resolved areas; quotation provenance data the loop so an audit can replay it finish to finish.

The subsequent article seems to be at a distinct class of query: ones whose reply shouldn’t be in any single passage however distributed throughout the doc, the place top-k retrieval is structurally flawed and the pipeline wants a distinct form.