Loop Engineering for RAG Era: Iterate top-k One at a Time

Naive RAG ships batch by default. Retrieval returns top-5, the LLM will get all 5, the reply comes again. It really works, and on laborious questions (comparability, itemizing) it’s the proper selection. However the silent value is paid on each different query: the straightforward factual ones the place the top-1 already had the reply. This text walks the second regime and the dispatch that picks per query.

Retrieval fingers era an ordered top-Ok. There are two methods to feed it in, and so they value very in a different way.

The default sample throughout most RAG tutorials appears like this:

top_k = retrieval(query, okay=5)
reply = era(query, top_k)

It runs in two steps for each query. Token value is roughly generation_cost(query + 5 chunks of context). Latency is roughly retrieval + one LLM name. And the LLM fortunately processes the 5 chunks even when the top-1 was already enough and chunks 2..5 added zero data.

Concretely: the consumer asks “what’s the efficient date of this coverage?”. Retrieval returns 5 line-windows the place the key phrase "efficient" seems. The primary one is the reply (“efficient from January 1, 2026”). Chunks 2..5 are signatures, footnotes, and a paragraph about historic efficient dates of previous insurance policies. The LLM reads all 5 to extract the identical date the primary one already had. On a corpus of 1 doc, the fee is rounding error. On a corpus of 50k insurance policies, that is actual cash per thirty days.

The sequential regime treats the Ok candidates as an ordered checklist and asks the era brick to validate sufficiency at every step:

for i, candidate in enumerate(top_k):
    reply = era(query, [candidate])
    if reply.answer_found and reply.complete_answer_found:
        break

The sufficiency sign lives contained in the typed contract Article 8A launched. The AnswerWithEvidence schema exposes answer_found (was the query’s data current on this candidate?) and complete_answer_found (was the complete reply current, not a fraction?). The loop reads these fields, not a customized heuristic.

Within the efficient date instance above: era runs as soon as on the top-1 candidate, the LLM returns answer_found = True, complete_answer_found = True, the loop exits. Tokens spent: generation_cost(query + 1 chunk of context). That’s 1/5 of the batch value for this query, on a pipeline that handles 1000’s of comparable lookups per day.

Batch mode retains the default behaviour: one LLM name sees all Ok candidates and produces one typed reply. Its case is constructed on three query varieties the place sequential breaks down:

• Itemizing questions: “checklist all exclusions on this contract”. The reply is each matching candidate, not the primary. Sequential would cease at top-1 (one exclusion discovered) and miss the opposite 4. Batch is the one right mode.
• Comparability questions: “is the premium greater than the earlier 12 months’s?”. The reply requires each candidates (this 12 months + final 12 months) in the identical name so the LLM can examine them. Sequential would extract every one independently and lose the be a part of.
• Tight-score retrieval: when the top-Ok candidates’ relevance scores are inside 5% of one another, retrieval can not reliably promote top-1 to the highest. Batch lets the LLM arbitrate with the total proof seen.

Price evaluation: batch all the time pays the generation_cost(query + Ok chunks of context) as soon as. Sequential pays generation_cost(query + 1 chunk of context) within the straightforward case (top-1 enough) and generation_cost(query + 1 chunk) × Ok within the worst case (each candidate inadequate). On a typical enterprise corpus with Ok = 5, sequential is cheaper on common for factual lookups (~80% of typical site visitors) and dearer for itemizing / comparability (~20%).

The clear structure doesn’t choose batch or sequential globally. It picks per query, utilizing the parsed question_df row from brick 2. Query form, decomposition sample, and intent drive the selection:

The dispatcher reads question_df.answer_shape and question_df.decomposition and routes. Naive RAG has no approach to make this distinction as a result of it has no parsed query to learn from.

The identical routing desk holds throughout sectors and professions. Completely different domains carry the identical form patterns and the identical sequential / batch choice flows out of them:

In each row, the sequential column is a single typed worth (Quantity, Date, Boolean) and advantages from the top-1 cease. The batch column is a listing or a comparability and desires all Ok candidates seen directly. The dispatcher’s desk covers all 5 sectors with the identical logic.

Sequential mode leans on one factor: the era brick reporting whether or not the candidate it simply learn was sufficient. That sign, and the foundations that cease the loop, reside right here.

Sequential mode solely works if the era brick can self-report whether or not the candidate it simply noticed contained the reply. The typed contract from Article 8A is what makes this doable:

class AnswerWithEvidence(BaseModel):
    worth: Any
    proof: checklist[Span]
    answer_found: bool
    complete_answer_found: bool
    confidence: float = Area(ge=0, le=1)
    caveats: checklist[str] = []

The sequential loop reads answer_found and complete_answer_found, not a confidence float or a customized heuristic. The clear separation between the 2 booleans (from Sample 4 of Article 8ter) is what makes the loop deterministic. Discovered + incomplete says “proceed to the subsequent candidate”; discovered + full says “cease”; not discovered says “proceed or quit at Ok”.

A confidence float would power a threshold (e.g. “cease at 0.8”), and that threshold drifts mannequin to mannequin. Two booleans don’t drift.

The sequential loop has three exits, not one:

• Sufficiency: answer_found and complete_answer_found on a candidate. Cease, ship the reply.
• Exhaustion: each one of many Ok candidates seen, none enough. Cease, return answer_found = False (a first-class reply the validator passes by means of).
• Finances: a token or time price range set by the dispatcher. Helpful when the corpus is massive and a runaway sequential loop would blow the cap. Similar form because the bounded iteration M4 (loop engineering) names.

Naive sequential implementations skip the third exit and burn tokens endlessly on edge instances. The sequence model units a price range upfront and the dispatcher logs it on each name.

Two regimes, two value profiles, and one boundary the sequence doesn’t cross.

To make the trade-off actual, here’s a back-of-envelope for an enterprise insurance coverage Q&A workload:

• 100 questions / day, Ok = 5, common chunk measurement = 600 tokens.
• Batch baseline: 100 × generation_cost(query + 5×600 tokens) = ~330k enter tokens per day for era.
• Sequential (80% top-1 enough): 80 × generation_cost(query + 600) + 20 × generation_cost(query + 5×600) (worst case for the 20% advanced questions) = ~115k enter tokens per day.

The ratio is 65% saving on enter tokens for era on this workload. The precise ratio will depend on the easy-question proportion and the chunk measurement; the precept is that sequential is a budget default as soon as the typed contract is in place. Batch is reserved for the query varieties that want it.

A pure subsequent step is to let the LLM resolve between batch and sequential per query (agentic dispatch). The sequence stops in need of this. The dispatcher in Part 2 is deterministic-dispatcher (one of many three approaches catalogued in Article 6C), not LLM-decides. The reason being the identical one which holds throughout the sequence: audit. The identical query on the identical day should route the identical means; an LLM that re-plans the dispatch per name can not give that assure.

When you want a stronger agentic loop (the LLM picks which candidates to take a look at, in what order, with what scope), the larger article on adaptive RAG loops is the proper place. This text retains the scope to the deterministic sequential vs batch choice pushed by the parsed query.

The fastened top-Ok + batch pipeline is the proper default for the query varieties the place each candidate issues. It’s the incorrect default for the factual lookups that make up most enterprise site visitors. The sequence provides two items: the typed sufficiency sign (answer_found, complete_answer_found) from Article 8A, and the dispatch desk from Part 2. Collectively they let the era brick cease after the primary candidate when that’s sufficient, and course of all Ok when the query kind requires it. The choice is made by the parser, not by the LLM, which retains the audit path intact.

The sequential / batch choice sits on the boundary between retrieval and era. The articles that body either side: