Loop Engineering for Itemizing Questions: When the Reply Is Each Passage, Not the Prime One

Take a easy itemizing query on the NIST CSF:

“What are all of the classes underneath the GOVERN perform?”

The correct reply is a listing of six classes: Organizational Context (GV.OC), Danger Administration Technique (GV.RM), Roles, Obligations, and Authorities (GV.RR), Coverage (GV.PO), Oversight (GV.OV), and Cybersecurity Provide Chain Danger Administration (GV.SC).

These six gadgets seem:

• In Desk 1 on web page 20 (compact record of Capabilities and Classes).
• In Appendix A on pages 21 to 23 (every class will get its personal subsection with subcategories under).

A naive RAG pipeline does this:

1. Embeds the query.
2. Retrieves top-5 chunks by similarity.
3. Sends them to the LLM with a era immediate.

What comes again from retrieval is usually the introduction to GOVERN (web page 17, the bulleted record of class descriptions) plus a number of paragraphs round it. That’s sufficient for the LLM to record 4 or 5 classes, however it may miss GV.SC if the availability chain mentions are clustered in a distinct chunk that didn’t make the top-k.

The LLM then produces:

“The classes underneath GOVERN are: Organizational Context, Danger Administration Technique, Roles and Obligations, Coverage, and Oversight.”

5 out of six. Seems to be like an entire reply. The consumer has no option to know that GV.SC is lacking except they cross-check towards the doc.

The retrieval activity has a distinct form for itemizing questions than for factual ones.

For a factual query, top-k retrieval works as a result of the reply is in one passage. You should discover that one passage, and similarity rating does an inexpensive job.

For a list query, the reply is in N passages, the place N is unknown forward of time. Prime-k with a set ok both retrieves too few (lacking gadgets) or too many (diluting the LLM’s consideration with irrelevant chunks).

Worse: if the gadgets within the record are comparable to one another (which they often are, since they’re variations on the identical theme), embedding similarity offers them comparable scores. Setting ok excessive sufficient to seize all gadgets additionally captures their near-duplicates, their summaries, their cross-references, and their introductory paragraphs. The LLM finally ends up with twenty mentions of “GOVERN” and has to determine that are precise class definitions and that are passing references.

The structural repair: don’t use top-k for itemizing questions. Use a retrieval technique designed to search out all gadgets moderately than a set variety of top-ranked ones.

The very first thing the pipeline wants is to acknowledge that the query is a list query, not a factual one. From Article 6, that is an intent: itemizing worth on the parsed query, set both by a small regex move on the query wording or by a devoted LLM classifier when the wording is ambiguous.

Detection occurs on the query understanding stage (Article 6). A number of alerts:

LISTING_MARKERS = [
    r"b(?:what|which)s+(?:ares+)?alls+(?:thes+)?",
    r"blists+(?:alls+)?(?:thes+)?",
    r"benumerates+(?:alls+)?",
    r"bgives+mes+(?:alls+)?(?:thes+)?",
    r"beverys+",
    r"bhows+manys+",  # counting questions are also listing
]

def is_listing_question(query: str) -> bool:
    """Heuristic: does this query desire a set of things moderately than one truth?"""
    return any(re.search(p, query, re.IGNORECASE) for p in LISTING_MARKERS)

This can be a place to begin. It misses some circumstances (“What does GOVERN cowl?” is implicitly a list query if GOVERN has a number of classes) and false-positives others (“What’s the record worth?” incorporates “record” however isn’t a list). The correct model makes use of an LLM classifier within the query understanding stage, returning an intent and an anticipated cardinality. Article 6 develops that classifier; right here we assume the query has already been tagged as itemizing.

When the query understanding stage tags an intent as itemizing, the orchestrator (Article 13) prompts the itemizing sample.

Each perform this text walks via ships in docintel.pipeline.itemizing. Import them as soon as and the remainder of the article is the methods they implement and when every one fires.

from docintel.pipeline.itemizing import (
    is_listing_question,        # detection (§1.3)
    list_via_structure,         # technique 1 (§2.1)
    list_via_pattern,           # technique 2 (§2.2)
    list_via_semantic,          # technique 3 (§2.3)
    deduplicate_items,          # merge (§3.1)
    detect_cardinality_cue,     # completeness sign (§3.2)
    assess_completeness,        # completeness verdict (§3.2)
)

The strongest sign for itemizing on enterprise paperwork is construction. Part headings, bullet lists, numbered enumerations, desk rows, subcategory codes: these are the writer’s personal enumeration.

For the NIST CSF query “What are all of the classes underneath GOVERN?”, the construction does the work:

def list_via_structure(
    *,
    section_hint: str | None,
    line_df: pd.DataFrame,
    toc_df: pd.DataFrame | None,
    page_range: tuple[int, int] | None = None,
) -> record[dict]:
    """Itemizing by way of the doc's personal structural markers."""
    # Technique A: TOC youngsters.
    if section_hint isn't None and toc_df isn't None and never toc_df.empty:
        if "parent_id" in toc_df.columns:
            youngsters = toc_df[toc_df["parent_id"] == section_hint]
            if not youngsters.empty:
                return youngsters.to_dict(orient="information")

    # Technique B: enumeration regex on a area.
    if page_range isn't None:
        lo, hello = page_range
        area = line_df[(line_df["page_num"] >= lo) & (line_df["page_num"] <= hello)]
    else:
        area = line_df
    gadgets = area[region["text"].astype(str).str.match(_ENUM_LINE_PATTERN)]
    return gadgets.to_dict(orient="information")

When the gadgets don’t comply with a strict structural sample however do comply with a recognizable form, you may combination them by sample matching throughout the doc.

Take the query “What are all of the subcategory codes underneath GOVERN?” on the NIST CSF. The codes comply with a strict sample: GV.XX-NN the place XX is a class code (OC, RM, RR, PO, OV, SC) and NN is a sequence quantity.

def list_via_pattern(line_df: pd.DataFrame, sample: str) -> record[dict]:
    """Discover each match of `sample` throughout `line_df`, deduplicated by match."""
    rx = re.compile(sample)
    matches: record[dict] = []
    seen: set[str] = set()
    for _, row in line_df.iterrows():
        textual content = str(row.get("textual content", ""))
        if not textual content:
            proceed
        for m in rx.finditer(textual content):
            code = m.group(0)
            if code in seen:
                proceed
            seen.add(code)
            matches.append({
                "code": code,
                "page_num": int(row.get("page_num", 0)),
                "line_num": int(row.get("line_num", 0)),
                "context": textual content,
            })
    return matches

Run it with the GV sample:

31 subcategories underneath GOVERN, all retrieved deterministically with one regex move over the doc. No embedding, no top-k, no LLM name till the very finish. The retrieval is exhaustive by building.

Sample-based aggregation works at any time when the gadgets have an everyday form: codes, identifiers, numbered clauses, normalized references, method identifiers. Many enterprise paperwork have these.

When the gadgets don’t comply with a structural marker or a set sample (free-form prose lists), the pipeline wants an LLM to establish them. However the usual top-k era form is mistaken. It returns what’s within the top-k chunks, not what’s within the doc.

The repair: a two-pass method.

Go 1: discovery: The pipeline does a broad retrieval (extra permissive than top-k, typically section-level by way of TOC). It sends the broad context to the LLM and asks for the gadgets discovered, plus a completeness self-assessment.

class ListingResult(BaseModel):
    gadgets: record[ListingItem]
    is_likely_complete: bool
    reason_if_incomplete: str | None = None
    suggested_additional_keywords: record[str] = Subject(default_factory=record)

Go 2: refinement: If is_likely_complete=False, the pipeline makes use of the LLM-suggested key phrases to broaden retrieval, fetches further areas, and re-runs the itemizing. A number of passes till completeness or till a max-iterations sure.

def list_via_semantic(
    *,
    query: str,
    line_df: pd.DataFrame,
    toc_df: pd.DataFrame | None,
    initial_keywords: record[str],
    llm_extract: LLMExtractFn,
    broad_retrieve: Callable | None = None,
    max_iterations: int = 3,
) -> record[ListingItem]:
    """Broad retrieval + LLM extraction in a bounded loop."""
    if broad_retrieve is None:
        broad_retrieve = _default_broad_retrieve
    gathered: record[ListingItem] = []
    seen_ids: set[str] = set()
    key phrases = record(initial_keywords)
    for _ in vary(max_iterations):
        candidates = broad_retrieve(line_df, toc_df, key phrases)
        new = [c for c in candidates if c.get("id") not in seen_ids]
        if not new:
            break
        seen_ids.replace(c.get("id") for c in candidates if c.get("id") isn't None)
        consequence = llm_extract(query, new, gathered)
        gathered = _merge_items(gathered, consequence.gadgets)
        if consequence.is_likely_complete:
            break
        key phrases = record(set(key phrases + consequence.suggested_additional_keywords))
    return gathered

This can be a suggestions loop particular to itemizing, and its three management surfaces are all seen within the code. The set off is is_likely_complete=False (or a cardinality mismatch, part 3.2). The termination is threefold: no new passages, a completeness verdict, or max_iterations. The restoration adjustments one factor earlier than the retry: the key phrase set, expanded with the LLM’s strategies. It’s a small loop within the sense of Article 10’s cascade: it runs totally contained in the itemizing department, iterating by itself materials, and by no means sends the pipeline again to a different brick. Article 13 develops the final iteration mechanics that sure this loop.

For the Transformer paper query “What are all of the regularization strategies used?”, the structural technique already finds them in Part 5.4. However for a query like “What are all of the strategies used to enhance translation high quality?”, the gadgets are scattered: dropout (Part 5.4), label smoothing (Part 5.4), beam search (Part 6.1), checkpoint averaging (Part 6.1), warmup schedule (Part 5.3). Semantic aggregation with a number of passes is the proper technique.

Discovering candidate gadgets is half the job. The opposite half is popping them into a solution the consumer can belief: merged duplicates, an express completeness verdict, and a presentation that exhibits each.

Whichever technique you employ, you’ll typically retrieve the identical merchandise a number of occasions. The identical subcategory talked about within the introduction and within the appendix. The identical regularization method talked about within the summary and in Part 5.4. Itemizing pipelines have to deduplicate.

Deduplication has two ranges:

Floor deduplication: Similar string, completely different occurrences. Straightforward: hash the normalized textual content.

Semantic deduplication: Completely different floor types of the identical merchandise. More durable: “Residual Dropout” and “dropout utilized to residual connections” are the identical method. “GV.SC” and “Cybersecurity Provide Chain Danger Administration” are the identical class.

For semantic deduplication, the LLM is the proper software. Ship the candidate record of things, ask the LLM to merge synonyms:

def deduplicate_items(
    gadgets: record[ListingItem], llm_dedupe: DedupeFn | None = None,
) -> record[ListingItem]:
    """Floor-form deduplication, with elective LLM semantic merge."""
    # Floor dedup first (exact-string, case-insensitive on canonical_name).
    by_lower: dict[str, ListingItem] = {}
    for it in gadgets:
        ok = it.canonical_name.strip().decrease()
        if ok in by_lower:
            cur = by_lower[k]
            cur.surface_forms = sorted(set(cur.surface_forms + it.surface_forms))
            cur.citations.lengthen(it.citations)
        else:
            by_lower[k] = it
    surface_unique = record(by_lower.values())
    if llm_dedupe is None or len(surface_unique) <= 1:
        return surface_unique
    return llm_dedupe(surface_unique)

The output is a listing the place every entry has:

• A canonical identify.
• All of the floor types encountered.
• All of the web page/line citations.

That is what the consumer desires: a clear enumeration of distinct gadgets, every one totally cited.

The pipeline additionally wants an express sign that the record is full; nothing within the candidate set says “you may have all of them” by itself.

Three sources of completeness:

Supply 1: structural completeness: When the gadgets come from a recognized structural sample (TOC youngsters, regex matches), completeness is assured by building. If the doc has 31 subcategory codes matching GV.[A-Z]{2}-d{2}, the regex finds all 31. There’s nothing to overlook.

Supply 2: express cardinality cues: Typically the doc tells you what number of gadgets there are. “The framework has six Capabilities”. “The encoder consists of N=6 layers”. “Three varieties of regularization are used”. The pipeline detects these cues and validates: if the doc says six and we discovered 5, one thing is lacking.

If the doc explicitly says “six classes underneath GOVERN” and the itemizing returns 5, the pipeline is aware of to iterate.

Supply 3: LLM self-assessment: When neither structural nor cardinality cues apply, the LLM’s is_likely_complete area is the one sign. It’s a weak sign (LLMs will be mistaken about completeness), however it’s higher than no sign. Mixed with bounded iteration (max 3 passes), it offers cheap conduct.

The ultimate completeness verdict is the strongest of the three:

def assess_completeness(
    gadgets: record,
    items_seen_last_iteration: record | None,
    document_text: str,
    llm_signal: bool | None,
) -> tuple[bool, Literal["structural", "cardinality", "llm_assessment"]]:
    """Mix the three sources of completeness with a strict priority."""
    cardinality = detect_cardinality_cue(document_text or "")
    if cardinality is None:
        cardinality = count_explicit_enumeration(document_text or "")
    if cardinality isn't None:
        return (len(gadgets) >= cardinality, "cardinality")
    if items_seen_last_iteration isn't None and len(gadgets) == len(
        items_seen_last_iteration
    ):
        return (True, "structural")
    if llm_signal isn't None:
        return (bool(llm_signal), "llm_assessment")
    return (False, "llm_assessment")

A list reply must be offered in another way from a factual reply. The consumer must see:

• The complete enumerated record, with every merchandise cited.
• A completeness assertion (“that is the whole record” or “this record could also be incomplete as a result of…”).
• Optionally, the floor types encountered for every merchandise.

A schema for itemizing output:

class ListingItem(BaseModel):
    canonical_name: str
    surface_forms: record[str] = Subject(default_factory=record)
    citations: record[Citation] = Subject(default_factory=record)
    extraction_confidence: float = 1.0

class ListingAnswer(BaseModel):
    gadgets: record[ListingItem]
    is_complete: bool
    completeness_source: Literal["structural", "cardinality", "llm_assessment"]
    not_found_items: record[str] = Subject(default_factory=record)
    notes: str | None = None

For our NIST query, the reply may appear like:

The consumer can instantly see the complete record, the citations, and the completeness verification. If a class had been lacking, the cardinality verify would have flagged it throughout retrieval.

Two runs, one per doc, chosen to indicate the 2 completeness outcomes: a run the place the cardinality verify confirms the record on the primary move, and a run the place it catches a lacking merchandise and the loop fires.

Let’s run a barely completely different query to indicate the cardinality verification:

“What are all of the Capabilities of the NIST Cybersecurity Framework?”

Query understanding. Intent categorised as itemizing. Anchor key phrases: ["Function", "GOVERN", "IDENTIFY", "PROTECT", "DETECT", "RESPOND", "RECOVER"].

Technique choice. The orchestrator (Article 13) selects the structural technique first. Capabilities are top-level entries within the TOC.

The CSF Capabilities don’t seem as top-level TOC entries; they seem as subsections inside Appendix A. The orchestrator falls again to pattern-based aggregation.

Sample-based retrieval. The pipeline scans for the sample b(GOVERN|IDENTIFY|PROTECT|DETECT|RESPOND|RECOVER)s*([A-Z]{2}):

Six matches, deduplicated.

Cardinality verify. The pipeline scans the doc for cardinality cues:

page8_text = "n".be part of(line_df.loc[line_df["page_num"] == 8, "textual content"])
cue = detect_cardinality_cue(page8_text)
# -> 6   (matched: "The Framework Core consists of six Capabilities:")

The doc explicitly lists six on web page 8 (Part 2, “Introduction to the CSF Core”). The retrieval returned six. Cardinality verified.

Technology with completeness verdict.

The consumer will get a clear, full, cited record with verification that nothing is lacking.

A free-form itemizing query:

“What are all of the regularization strategies used to coach the Transformer?”

Query understanding. Intent: itemizing. Anchor key phrases: ["regularization", "dropout", "label smoothing", "training"].

Technique choice. No clear sample (regularization strategies don’t have a set code). Fall again to structural technique with TOC: discover the part titled “Regularization”.

Part retrieval. toc_df has Part 5.4 “Regularization” on web page 7. The pipeline retrieves it.

Merchandise extraction. Inside Part 5.4, the doc makes use of daring headers as enumeration markers (web page 8):

• Residual Dropout, utilized to sub-layer outputs and to embeddings + positional encodings sums. Pdrop = 0.1.
• Label Smoothing, worth εls = 0.1.

Two gadgets discovered.

Cardinality verify. The primary sentence of Part 5.4 says “We make use of three varieties of regularization throughout coaching”. The retrieval discovered two. Cardinality says three. Mismatch, the pipeline iterates.

Iteration. The pipeline reads the LLM’s suggested_additional_keywords. The LLM, seeing two gadgets however understanding three are talked about, suggests ["attention dropout", "checkpoint averaging"].

Re-running retrieval with expanded key phrases finds a 3rd merchandise earlier within the paper: “consideration dropout”. Wanting again at Part 5.4 fastidiously, the third regularization is talked about however as a follow-up to Residual Dropout; the identical paragraph says dropout can be utilized to consideration weights. On nearer studying the third kind is implicit within the dropout dialogue. The LLM extracts it on the second move.

This iteration mechanic is listing-specific: cardinality mismatch because the set off, key phrase enlargement because the restoration (the one factor that adjustments earlier than the retry), completeness verify because the termination. The overall iteration mechanism (bounding, drift detection, anti-patterns, audit path) is what Article 13 develops.

Closing reply.

The mismatch between anticipated rely (three) and preliminary extraction (two) triggered iteration. With out the cardinality verify, the pipeline would have returned a assured two-item reply that was lacking a 3rd.

Itemizing questions want three issues naive top-k can not give: detection up entrance so the pipeline picks the itemizing department, a technique that matches the doc’s construction (TOC for sections, regex for markers, semantic iteration for scattered prose), and a completeness sign pushed by a cardinality cue when the doc declares one. Not one of the three methods is common; the dispatcher (Article 13) picks the proper one per query.

The sample ships because the library’s itemizing module and composes with TOC retrieval (Article 9), cross-references (Article 11), and the final iteration equipment Article 13 names.