I used to be going by means of our agentic software dashboard and seen a spike in final week’s LLM utilization, with site visitors sitting at precisely the identical degree it had been for weeks. Initially we assumed it was a logging bug however it caught an actual subject.
It lined up virtually precisely with an architectural change we’d shipped just a few weeks earlier: we’d moved a bit of our pipeline from a single-agent setup to a multi-agent one. Totally different brokers dealing with completely different elements of a process, coordinated by means of LangGraph, with a supervisor node deciding what occurs subsequent. On paper this was a clear win conducting higher process decomposition, every agent doing one factor effectively as a substitute of 1 mannequin attempting to do all the pieces inside a single sprawling immediate. You’d anticipate utilization to go up considerably, extra calls per process is the price of that structure, and no person was pretending in any other case. However to our shock it had roughly tripled for duties that functionally hadn’t modified in any respect.
What’s on this article
- The instinct that stopped working
- Chasing the mistaken suspect
- The repair that seemed proper and wasn’t sufficient
- What truly labored
- What the numbers seemed like as soon as we stopped guessing
- What I’d nonetheless rethink
The instinct that stopped working
In the event you’ve solely ever run a single LLM name per process, your psychological mannequin of price is roughly linear, longer enter, longer output, extra tokens, extra spend. You may eyeball it and be shut sufficient.
That instinct falls aside the second you introduce orchestration, and it falls aside quietly, which is worse than falling aside loudly. A supervisor agent now decides about what occurs subsequent, and that call itself prices tokens. Every sub-agent carries its personal system immediate, its personal instrument schemas, usually its personal copy of context the earlier agent already had. None of this reveals up as “extra work being carried out”, from the person’s facet the duty remains to be the identical process. It reveals up as extra equipment wrapped across the process, and equipment isn’t free simply because it’s invisible to whoever’s utilizing the product.
We hadn’t budgeted for it. Our structure evaluations had centered on decomposition high quality and correctness, did every agent do its job effectively, did the handoffs make sense, and never as soon as, so far as I can bear in mind, on what the token graph would truly seem like as soon as it was reside and taking actual site visitors. That’s on us, and it’s in all probability on most groups making this transfer for the primary time.
Chasing the mistaken suspect
My first assumption was fan-out. Someplace, the supervisor was spawning extra sub-agent calls than the duty genuinely wanted, and trimming the branching logic would convey the quantity again down. It felt like the plain place to look, extra brokers, extra calls, easy arithmetic.
So I spent the higher a part of a day instrumenting name counts per supervisor choice, anticipating to catch some node calling three brokers when one would have carried out the job. I discovered virtually nothing. The fan-out was doing roughly what it was meant to do, department for department.
That was irritating within the particular method that being mistaken about an apparent reply is irritating. You’ve burned a day, and also you’re again the place you began with one speculation eradicated and no substitute.
The substitute turned up virtually accidentally, whereas I used to be studying by means of uncooked name logs searching for something uncommon fairly than testing a concept. One agent’s output would fail a validation step, a malformed instrument name, a schema mismatch, sometimes a mannequin deciding to clarify itself in prose as a substitute of simply calling the instrument and the retry wrapper would silently re-invoke that agent. Silently, as a result of it accomplished efficiently on the retry and nothing downstream ever noticed a failure. Which sounds tremendous, till you discover what re-invoking that agent truly required: rebuilding its upstream context, which in some paths meant re-running a sub-agent that had already produced a superbly good end result, purely to reconstruct the enter the failing agent wanted. One validation failure, three layers deep within the graph, might set off a cascade that re-ran work that had by no means been damaged.
Nothing about this threw an error a human would see. The pipeline accomplished and the reply that got here again was appropriate. The retry logic was doing exactly what it had been informed to do, retry on failure, it merely had no idea of how costly the factor it was retrying truly was, or whether or not re-running already-correct upstream work was vital in any respect.
In the event you’re constructing retry logic for an agent graph fairly than a single name, that is the half price sitting with: a retry decorator that behaves completely effectively wrapped round one LLM name turns into a value amplifier the second it sits inside a graph with shared upstream state.
The repair that seemed proper and wasn’t sufficient
The apparent response to “retries are costly” is to cap retry counts and add exponential backoff. We tried it however it solely helped a little bit.
from tenacity import retry, stop_after_attempt, wait_exponential
@retry(
cease=stop_after_attempt(3),
wait=wait_exponential(multiplier=1, min=2, max=10),
reraise=True,
)
def call_agent(agent, payload):
return agent.invoke(payload)It’s not the mistaken factor to have and you must have retry caps no matter anything on this article however it didn’t contact the precise drawback. Each retry, capped or not, was nonetheless paying full worth on probably the most succesful, costliest mannequin accessible within the graph, no matter what had truly gone mistaken.
A malformed instrument name from a summarisation step doesn’t want the identical firepower to repair as a genuinely onerous reasoning failure. Treating each retry as equally deserving of your strongest mannequin is the error, and it’s the one I see really helpful most frequently: add backoff, add a retry restrict, transfer on, as if the associated fee per retry weren’t the variable truly price controlling. Capping how usually you’re mistaken doesn’t show you how to if you happen to’re nonetheless paying premium costs to be mistaken.
What truly labored
The actual repair was three adjustments, they usually solely actually labored as soon as we’d made all three. Doing simply one in every of them left an apparent hole the others would have closed.
Routing by process complexity – Sub-agents bought assigned a mannequin based mostly on what the step genuinely required, fairly than one mannequin used in every single place by default. Summarisation, formatting, routine tool-call building, the mechanical steps, not the reasoning-heavy ones moved to a smaller, cheaper mannequin. Steps that wanted precise multi-step reasoning or judgement calls, the type of factor that had been inflicting the fascinating failures within the first place, stayed on the stronger mannequin.
It’s a easy desk, and the explanation it hadn’t existed from the beginning says extra about our priorities. The unique objective had been correctness-per-call and correctness-per-call quietly biases you in the direction of “simply use the very best mannequin in every single place and fear about price later.” I don’t assume that’s a defensible default when you’re previous a prototype. In the event you can’t articulate why a given step wants your costliest mannequin, it in all probability doesn’t, and writing the routing desk is the train that forces you to truly reply that query as a substitute of assuming it.
Context trimming between handoffs – This one’s simpler to skip as a result of it’s much less seen than mannequin selection, and we very almost did skip it. Each agent within the graph was inheriting the total amassed context from all the pieces upstream of it, by default, as a result of that’s what the framework offers you until you actively resolve in any other case. A formatting agent on the very finish of the chain was receiving the whole reasoning hint of three earlier brokers, most of which it had no use for by any means.
def trim_handoff_context(state, needed_fields):
return {
discipline: state[field]
for discipline in needed_fields
if discipline in state
}
# formatting agent solely wants the ultimate structured end result,
# not the total upstream reasoning hint
handoff = trim_handoff_context(
graph_state,
needed_fields=["final_result", "output_schema"],
)We trimmed handoffs all the way down to what the receiving agent truly wanted: remaining outputs and the precise fields related to its process, not the total upstream dialog. It doesn’t produce one dramatic before-and-after quantity however what it produces is a gradual discount throughout each single name within the graph, since you’ve stopped paying input-token price for context that was by no means going to be learn by something downstream.
Parallel execution – We’d initially parallelised impartial sub-agent branches purely to chop latency. Brokers doing separate jobs, pulling completely different information sources, operating impartial checks don’t must run sequentially simply because a sequential graph is simpler to attract on a whiteboard.
import asyncio
async def run_independent_branches(brokers, payload):
outcomes = await asyncio.collect(
*(agent.ainvoke(payload) for agent in brokers),
return_exceptions=False,
)
return outcomesWhat I hadn’t anticipated moving into was that parallel execution additionally shrank the retry blast radius, virtually as a facet impact. Sequential execution meant a downstream failure might set off re-running a complete upstream chain to rebuild shared state. Unbiased parallel branches meant a failure in a single department had nothing to tug down with it as a result of there was no shared state to rebuild within the first place. Latency was the explanation we did this however in hindsight, the lowered retry cascading is the half that mattered extra.
What the numbers seemed like as soon as we stopped guessing
I need to be upfront that these are tough figures, not precision-instrumented ones, as a result of we hadn’t wired up per-step price attribution earlier than any of this began which is itself the lesson, construct that first, not after you’ve already bought an issue to diagnose. With that caveat on the desk: token utilization per mannequin dropped by roughly 40% as soon as retries stopped silently re-running already-correct upstream work, and end-to-end latency dropped someplace within the 45–55% vary from the mix of task-based routing and parallel execution. The retry repair and the routing repair landed shut collectively in time, so I can’t cleanly separate how a lot every one contributed by itself. That’s a real limitation of how we measured this, not a hedge for the sake of sounding cautious.
What I’d nonetheless rethink
Complexity-based routing works, however it’s a desk somebody has to take care of by hand, and it goes stale the second you add a brand new agent sort or a mannequin will get deprecated out from beneath you. A light-weight classifier might predict process complexity and choose the mannequin at runtime as a substitute however that’s its personal mannequin name, with its personal price, sitting there guarding in opposition to a value drawback. Whether or not that commerce is price making relies upon completely on how usually your process combine truly shifts, and we haven’t run this lengthy sufficient, on a large sufficient number of duties, to know the reply for us but.
The most important lesson wasn’t “use smaller fashions” or “cap retries.” It was that multi-agent methods have failure modes that don’t exist in single-agent pipelines. As soon as work is distributed throughout a graph, each retry, each context handoff, and each routing choice turns into a part of your price mannequin. In the event you’re solely measuring latency and correctness, you’ll uncover these prices in manufacturing fairly than throughout design.
