Impressed by the ICLR 2026 blogpost/article, The 99% Success Paradox: When Close to-Excellent Retrieval Equals Random Choice
an Edinburgh-trained PhD in Info Retrieval from Victor Lavrenko’s Multimedia Info Retrieval Lab at Edinburgh, the place I educated within the late 2000s, I’ve lengthy seen retrieval via the framework of conventional IR pondering:
- Did we retrieve not less than one related chunk?
- Did recall go up?
- Did the ranker enhance?
- Did downstream reply high quality look acceptable on a benchmark?
These are nonetheless helpful questions. However after studying the current work on Bits over Random (BoR), I feel they’re incomplete for the Agentic methods many people at the moment are truly constructing.
The ICLR blogpost sharpened one thing I had felt for some time in manufacturing LLM methods: retrieval high quality ought to take into consideration each how a lot good content material we discover and in addition how a lot irrelevant materials we carry together with it. In different phrases, as we crank up our recall we additionally enhance the danger of context air pollution.
What makes BoR helpful is that it provides us a language for this. BoR tells us whether or not retrieval is genuinely selective, or whether or not we’re attaining success principally by stuffing the context window with extra materials. When BoR falls, it’s a signal that the retrieved bundle is changing into much less discriminative relative to likelihood. In observe, that always correlates with the mannequin being compelled to learn extra junk, extra overlap, or extra weakly related materials.
The vital nuance is that BoR doesn’t straight measure what the mannequin “feels” when studying a immediate. It measures retrieval selectivity relative to random likelihood. However decrease selectivity typically goes hand in hand with extra irrelevant context, extra immediate air pollution, extra consideration dilution, and worse downstream efficiency. Put merely, BoR helps inform us when retrieval continues to be selective and when it has began to degenerate into context stuffing.
That concept issues rather more for RAG and brokers than it did for traditional search.
Why retrieval dashboards can mislead agent groups
One of many best traps in RAG is to take a look at your retrieval dashboard, see wholesome metrics, and conclude that the system is doing nicely. You would possibly see:
- excessive Success@Okay,
- robust recall,
- a very good rating metric,
- and a bigger Okay seeming to enhance protection.
On paper issues might look higher however, in actuality, the agent would possibly truly behave worse. Your agent might have any variety of maladies resembling diffuse solutions to queries, unreliable instrument use or just an increase in latency and token value with none actual consumer profit.
This disconnect occurs as a result of most retrieval dashboards nonetheless mirror a human search worldview. They assume the buyer of the retrieved set can skim, filter, and ignore junk. People are surprisingly good at this. LLMs will not be constantly good at it.
An LLM doesn’t “discover” ten retrieved objects and casually deal with the very best two in the way in which a robust analyst would. It processes the total bundle as immediate context. Which means the retrieval layer is surfacing proof that’s actively shaping the mannequin’s working reminiscence.
This is the reason I feel agent groups ought to cease treating retrieval as a back-office rating drawback and begin treating it as a reasoning-budget allocation drawback. When constructing performant agentic methods, the important thing query is each:
- Did we retrieve one thing related?
and:
- How a lot noise did we power the mannequin to course of to be able to get that relevance?
That’s the lens BoR pushes you towards, and I’ve discovered it to be a really helpful one.
Context engineering is changing into a first-class self-discipline
One motive this paper has resonated with me is that it matches a broader shift already taking place in observe. Software program engineers and ML practitioners engaged on LLM methods are progressively changing into one thing nearer to context engineers.
Which means designing methods that resolve:
- what ought to enter the immediate,
- when it ought to enter,
- in what type,
- with what granularity,
- and what must be excluded completely.
In conventional software program, we fear about reminiscence, compute, and API boundaries. In LLM methods, we additionally want to fret about context purity. The context window is contested cognitive actual property.
Each irrelevant passage, duplicated chunk, weakly associated instance, verbose instrument definition, and poorly timed retrieval consequence competes with the factor the mannequin most must deal with. That’s the reason I just like the air pollution metaphor. Irrelevant context contaminates the mannequin’s workspace.
The BoR poster provides this instinct a extra rigorous form by telling us that we should always cease evaluating retrieval solely by whether or not it succeeds. We must also ask how a lot better the retrieval is in comparison with likelihood, on the depth (high Okay retrieved objects) that we are literally utilizing. That may be a very practitioner-friendly query.
Why instrument overload breaks brokers
That is the place I feel the BoR work turns into particularly vital for real-world agent methods.
In traditional RAG, the corpus is usually giant. You might be retrieving from tens of hundreds or thousands and thousands of chunks. In that regime, random likelihood stays weak for longer. Software choice could be very totally different.
In an agent, the mannequin could also be selecting amongst 20, 50, or 100 instruments. That sounds manageable till you notice that a number of instruments are sometimes vaguely believable for a similar process. As soon as that occurs, dumping all instruments into context is just not thoroughness. It’s confusion disguised as completeness.
I’ve seen this sample repeatedly in agent design:
- the workforce provides extra instruments,
- descriptions change into longer,
- overlap between instruments will increase,
- the agent begins making brittle or inconsistent decisions,
- and the primary intuition is to tune the immediate more durable.
However typically the actual problem is architectural, not prompt-level. The mannequin is being requested to select from an overloaded context the place distinctions are too weak and too quite a few.
What BoR provides here’s a helpful solution to formalize one thing individuals typically really feel solely intuitively: there’s a level the place the choice process turns into so crowded that the mannequin is now not demonstrating significant selectivity.
That’s the reason I strongly want agent designs with:
- Staged instrument retrieval: narrowing the search in steps, first discovering a small set of believable instruments, then making the ultimate alternative from that shortlist moderately than from the total library directly.
- Area routing: earlier than closing instrument alternative means first deciding which broad space the duty belongs to, resembling search, CRM, finance, or coding, and solely then choosing a selected instrument inside that area.
- Compressed functionality summaries: presenting every instrument with a brief, high-signal description of what it’s for, when it must be used, and the way it differs from close by instruments, as a substitute of dumping lengthy verbose specs into the immediate.
- Specific exclusion of irrelevant instruments: intentionally eradicating instruments that aren’t acceptable for the present process so the mannequin is just not distracted by believable however pointless choices.
In my expertise instrument alternative must be handled extra like retrieval than like static immediate ornament.
Understanding BoR via instrument choice
One of the vital helpful issues about BoR is that it sharpens what top-Okay actually means in tool-using brokers.
In doc retrieval, growing top-Okay typically means transferring from top-5 passages to top-20 or top-50 from a really giant corpus. In instrument choice, the identical transfer has a really totally different character. When an agent solely has a modest instrument library, growing top-Okay might imply transferring from a shortlist of three candidate instruments, to five, to eight, and finally to the acquainted however harmful fallback: simply give all of it 15 instruments to be protected.
That usually improves recall or Success@Okay, as a result of the right instrument is extra more likely to be someplace within the seen set. However that enchancment may be deceptive. As Okay grows, you aren’t solely serving to the router. You might be additionally making it simpler for a random selector to incorporate a related instrument.
So the actual query is just not merely: Did top-8 comprise a useful gizmo extra typically than top-3? The extra vital query is: Did top-8 enhance significant selectivity, or did it principally make the duty simpler via brute-force inclusion?That’s precisely the place BoR turns into helpful.
A easy instance makes the instinct clearer. Suppose you might have 10 instruments, and for a given class of process 2 of them are genuinely related. Should you present the mannequin just one instrument, random likelihood of surfacing a related one is 20 %. At 3 instruments, the random baseline rises sharply. At 5 instruments, random inclusion is already pretty robust. At 10 instruments, it’s one hundred pc, as a result of you might have proven all the things. So sure, Success@Okay rises as Okay rises. However the which means of that success modifications. At low Okay, success signifies actual discrimination. At excessive Okay, success might merely imply you included sufficient of the menu that failure turned troublesome.
That’s what I imply by serving to random likelihood moderately than significant selectivity.
This issues as a result of, with instruments, the issue is worse than a deceptive metric. If you present too many instruments, the immediate will get longer, descriptions start to overlap, the mannequin sees extra near-matches, distinctions change into fuzzier, parameter confusion rises, and the possibility of selecting a plausible-but-wrong instrument will increase. So though top-Okay recall improves, the standard of the ultimate choice might worsen. That is the small-tool paradox: including extra candidate instruments can enhance obvious protection whereas lowering the agent’s capability to decide on cleanly.
A sensible approach to consider that is that instrument choice typically falls into three regimes. Within the wholesome regime, Okay is small relative to the variety of instruments, and the looks of a related instrument within the shortlist tells you the router truly did one thing helpful. For instance, 30 whole instruments, 2 or 3 related, and a shortlist of three or 4 nonetheless appears like real choice. Within the gray zone, Okay is giant sufficient that recall improves, however random inclusion can be rising shortly. For instance, 20 instruments, 3 related, shortlist of 8. Right here you should still achieve one thing, however it’s best to already be asking whether or not you’re really routing or merely widening the funnel. Lastly, there may be the collapse regime, the place Okay is so giant that success principally comes from exposing sufficient of the instrument menu that random choice would additionally succeed typically. When you’ve got 15 instruments, 3 related ones, and a shortlist of 12 or all 15, then “excessive recall” is now not saying a lot. You might be getting near brute-force publicity.
Operationally, this pushes me towards a greater query. In a small-tool system, I like to recommend avoiding the overexposure mindset that asks:
- How giant should Okay be earlier than recall appears good?
The higher query is:
- How small can my shortlist be whereas nonetheless preserving robust process efficiency?
That mindset encourages disciplined routing.
In observe, that often means routing first and selecting second, conserving the shortlist very small, compressing instrument descriptions so distinctions are apparent, splitting instruments into domains earlier than closing choice, and testing whether or not growing Okay improves end-to-end process accuracy, not simply instrument recall. A helpful sanity test is that this: if giving the mannequin all instruments performs about the identical as your routed shortlist, then your routing layer will not be including a lot worth. And if giving the mannequin extra instruments improves recall however worsens total process efficiency, you’re probably in precisely the regime the place Okay helps random likelihood greater than actual selectivity.
When the failure mode modifications: giant instrument libraries
The big-tool case is totally different, and that is the place an vital nuance issues. A bigger instrument universe does not imply we should always dump a whole bunch of instruments into context and count on the system to work higher. It simply means the failure mode modifications.
If an agent has 1,000 instruments out there and solely a handful are related, then growing top-Okay from 10 to 50 and even 100 should still symbolize significant selectivity. Random likelihood stays weaker for longer than it does within the small-tool case. In that sense, BoR continues to be helpful: it helps cease us from mistaking broader publicity for higher routing. It asks whether or not a bigger shortlist displays real selectivity, or whether or not it’s merely serving to by exposing a bigger slice of the search area.
However BoR doesn’t seize the entire drawback right here. With very giant instrument libraries, the difficulty might now not be that random likelihood has change into too robust. The problem could also be that the mannequin is just drowning in choices. A shortlist of 200 instruments can nonetheless be higher than random in BoR phrases and but nonetheless be a horrible immediate. Software descriptions overlap, near-matches proliferate, distinctions change into more durable to keep up, and the mannequin is compelled to motive over a crowded semantic menu.
So BoR is effective, however it’s not enough by itself. It’s higher at telling us whether or not a shortlist is genuinely discriminative relative to likelihood than whether or not that shortlist continues to be cognitively manageable for the mannequin. In giant instrument libraries, we due to this fact want each views: BoR to measure selectivity, and downstream measures resembling tool-choice high quality, latency, parameter correctness, and end-to-end process success to measure usability.
BoR tells us whether or not retrieval is genuinely selective, or whether or not we’re attaining success principally by stuffing the context window with extra materials. When BoR falls, it’s a signal that the retrieved bundle is changing into much less discriminative relative to likelihood. In observe, that always correlates with the mannequin being compelled to learn extra junk, extra overlap, or extra weakly related materials. The nuance is that BoR doesn’t straight measure what the mannequin “feels” when studying a immediate. It measures selectivity relative to random likelihood. However low BoR is usually a warning signal that the mannequin is being requested to course of an more and more noisy context window.
The design implication is similar though the rationale differs. With small instrument units, broad publicity shortly turns into unhealthy as a result of it helps random likelihood an excessive amount of. With very giant instrument units, broad publicity turns into unhealthy as a result of it overwhelms the mannequin. In each instances, the reply is to not stuff extra into context. It’s to design higher routing.
My very own rule of thumb: the mannequin ought to see much less, however cleaner
If I needed to summarize the sensible shift in a single sentence, it will be this: for LLM methods, smaller and cleaner is usually higher than bigger and extra complete.
That sounds apparent, however many methods are nonetheless designed as if “extra context” is robotically safer. In actuality, as soon as a baseline degree of helpful proof is current, extra retrieval can change into dangerous. It will increase token value and latency, however extra importantly it widens the sector of competing cues contained in the immediate.
I’ve come to consider immediate development in three layers:
Layer 1: obligatory process context
- The core instruction, constraints, and rapid consumer goal.
Layer 2: extremely selective grounding
- Solely the minimal supporting proof or instrument definitions wanted for the following reasoning step.
Layer 3: optionally available overflow
- Materials that’s merely believable, loosely associated, or included “simply in case.”
Most failures come from letting Layer 3 invade Layer 2. That’s the reason retrieval must be judged not simply by protection, however by its capability to protect a clear Layer 2.
The place I feel BoR is particularly helpful
I don’t see BoR as a alternative for all retrieval metrics. I see it as a really helpful extra lens, particularly in these instances:
1. Selecting Okay in manufacturing
- Many groups nonetheless enhance top-Okay till recall appears ok. BoR encourages a extra disciplined query: at what level is growing Okay principally serving to random likelihood moderately than significant selectivity?
2. Evaluating agent instrument routing
- This can be essentially the most compelling use-case. Brokers typically fail not as a result of no good instrument exists, however as a result of too many practically related instruments are offered concurrently.
3. Diagnosing why downstream high quality falls regardless of “higher retrieval”
- That is the traditional paradox. Protection goes up. Last reply high quality goes down. BoR helps clarify why.
4. Evaluating methods with totally different retrieval depths
- Uncooked success charges may be misleading when one system retrieves way more materials than one other. BoR helps normalize for that.
5. Stopping overconfidence in benchmark outcomes
- Some benchmarks might merely be too straightforward on the chosen retrieval depth. A powerful-looking consequence could also be nearer to luck than we expect.
The place I feel BoR could also be inadequate by itself
I just like the paper, however I’d not deal with BoR as the ultimate reply to retrieval analysis. There are not less than just a few vital caveats.
First, not each process solely wants one good merchandise. Some duties genuinely require synthesis throughout a number of items of proof. In these instances, a success-style view can understate the necessity for broader retrieval.
Second, retrieval usefulness is just not binary. Two chunks might each depend as “related,” whereas one is much extra actionable, concise, or decision-useful for the mannequin.
Third, immediate group nonetheless issues. A loud bundle that’s rigorously structured might carry out higher than a barely cleaner bundle that’s poorly ordered or badly formatted.
Fourth, the mannequin itself issues. Completely different LLMs have totally different tolerance for litter, totally different long-context conduct, and totally different tool-use reliability. A retrieval coverage that pollutes one mannequin could also be acceptable for one more.
Fifth, and that is particularly related for giant instrument libraries, BoR tells us extra about selectivity than about usability. A shortlist can nonetheless look meaningfully higher than random and but be too crowded, too overlapping, or too semantically messy for the mannequin to make use of nicely.
So I’d not use BoR in isolation. I’d pair it with:
- downstream process accuracy,
- latency and token-cost evaluation,
- tool-call high quality,
- parameter correctness,
- and a few express measure of immediate cleanliness or redundancy.
Nonetheless, even with these caveats, BoR contributes one thing vital: it forces us to cease complicated protection with selectivity.
How this modifications analysis observe for me
The most important sensible shift is that I’d now consider retrieval methods extra like this:
- First, take a look at normal retrieval metrics. They nonetheless matter. You must ideally think about a bag-of-metrics strategy, leveraging a number of complementary metrics.
Then ask:
- What’s the random baseline at this depth?
- Is greater Success@Okay truly demonstrating ability, or simply simpler situations?
- How a lot additional context did we add to get that achieve?
- Did downstream reply high quality enhance, keep flat, or worsen?
- Are we making the mannequin motive, or merely making it learn extra?
For brokers, I’d go even additional:
- What number of instruments have been seen at choice time?
- How a lot overlap existed between candidate instruments?
- May the system have routed first and chosen second?
- Was the mannequin requested to select from a clear shortlist, or from a crowded menu?
That may be a extra real looking analysis setup for the sorts of methods many groups are literally deploying.
The broader lesson
The principle lesson I took from the ICLR poster is far broader than a single new metric: it’s that LLM system high quality relies upon closely on the cleanliness of the context we assemble across the mannequin. That has penalties throughout the Agentic stack:
- retrieval,
- reminiscence,
- instrument routing,
- agent planning,
- multi-step workflows,
- and even UI design for human-in-the-loop methods.
The very best LLM methods would be the ones that expose the proper data, on the proper second, within the smallest clear bundle that also helps the duty. That is the character of what good context engineering appears like.
Last thought
For years, retrieval was principally about discovering needles in haystacks. For LLM methods, that’s now not sufficient. Now the job can be to keep away from dragging half the haystack into the immediate together with the needle.
That’s the reason I feel the BoR concept issues and is so impactful. It provides practitioners a greater language for an actual manufacturing drawback: how one can measure when helpful context has quietly was polluted context. And when you begin your methods that approach, loads of acquainted agent failures start to make rather more sense.
BoR doesn’t straight measure what the mannequin “feels” when studying a immediate, nevertheless it does inform us when retrieval is ceasing to be meaningfully selective and beginning to resemble brute-force context stuffing. In observe, that’s typically precisely the regime the place LLMs start to learn extra junk, motive much less cleanly, and carry out worse downstream.
Extra broadly, I feel this factors to an vital rising sub-field: creating higher metrics for measuring LLM system efficiency in real looking settings, not simply mannequin functionality in isolation. Now we have change into moderately good at measuring accuracy, recall, and benchmark efficiency, however a lot much less good at measuring what occurs when a mannequin is compelled to motive via cluttered, overlapping, or weakly filtered context.
That, to me, exposes an actual hole. BoR helps measure selectivity relative to likelihood, which is effective. However there may be nonetheless a lacking idea round what I’d time period cognitive overload: the purpose at which a mannequin should still have the fitting data someplace in view, but performs worse as a result of too many competing choices, snippets, instruments, or cues are offered directly. In different phrases, the failure is now not simply retrieval failure. It’s a reasoning failure induced by immediate air pollution.
I think that higher methods of measuring this type of cognitive overload will change into more and more vital as agentic methods develop extra complicated. The subsequent leap ahead might not simply come from bigger fashions or greater context home windows, however from higher methods of quantifying when the mannequin’s working context has crossed the road from helpful breadth into dangerous overload.
Impressed by the ICLR 2026 blogpost/article, The 99% Success Paradox: When Close to-Excellent Retrieval Equals Random Choice.
Disclaimer: The views and opinions expressed on this article are solely my very own and don’t symbolize these of my employer or any affiliated organisations. The content material relies on private reflections and speculative serious about the way forward for science and expertise. It shouldn’t be interpreted as skilled, educational, or funding recommendation. These forward-looking views are supposed to spark dialogue and creativeness, to not make predictions with certainty.
