inside your individual firm and nearly any failure is affordable: you retry, fall again, or doubtlessly even ignore it. Put that very same workflow behind a buyer’s API or MCP server and the grace is gone. Now just one factor issues: did the shopper get an accurate, usable consequence? Their course of is determined by yours delivering one. They, not you, now determine what counts as delivered. At Databook we course of billions of tokens for the world’s largest enterprises; this text relies on actual knowledge from manufacturing flows at scale. I hope it provides you some helpful insights.
Delivering that result’s more durable than it seems, as a result of LLMs are notoriously unreliable. They fail incessantly, in 4 flavors: an invalid reply (empty, unparseable, or just unsuitable), a tough error, no reply in any respect, or no reply in time. And the entire run solely succeeds if each step does, so the extra you chain collectively, the extra probabilities there are for certainly one of them to fail. A workflow of individually glorious steps can nonetheless come out a coin flip.
Inside your individual firm you may soak up each certainly one of these, as a result of you’ve gotten slack on each axis: retry the failed step, wait out the sluggish one, spend slightly extra, calm down the bar when you should. Put the identical workflow behind a buyer’s API and the slack vanishes, as a result of the run now has to clear three useful resource budgets on the similar time, none of which you set:
- Time — a window that closes whether or not or not you’re accomplished: a tough gateway timeout (one to 3 minutes, typically 5) that severs the connection mid-run, or one thing softer: an SLA, a caller blocked on the consequence, a course of that may solely wait so lengthy. And it doesn’t resume: when the window closes, the shopper simply retries, beginning the entire run over from zero.
- Value — now a margin, not a pool. Each run carries a worth the shopper already paid, so it has to return again worthwhile, not merely reasonably priced. And the shopper, not you, decides how typically it runs.
- Tokens and fee — a per-minute token funds (TPM) you share throughout each buyer directly, and so they are likely to name in the identical bursts. You hit the ceiling precisely when load is heaviest, which is strictly when latency is worst.
Below all three sits a tough flooring you by no means commerce beneath: high quality. The reply must be proper to depend in any respect. A quick, low-cost, on-time reply that’s unsuitable remains to be a failure. High quality isn’t a funds you spend down.
Any certainly one of these you could possibly handle by itself. The bind is that they apply collectively and pull in opposition to one another, so the plain repair for one spends one other. Wait out a sluggish step and also you blow the time window. Race a second copy to beat the clock and also you burn value and quota. Attain for a stronger mannequin to clear the standard flooring and also you get slower. Not one of the budgets are yours to loosen, so the one transfer left is to commerce intentionally throughout all of them directly — with out ever dropping under the ground.
That’s what makes a customer-facing workflow a genuinely totally different factor to construct, and it typically forces a playbook that, from the within, seems completely backwards:
- Kill a name that hasn’t failed
- Hearth a replica of a name you’re already paying for
- Drop to a weaker mannequin on function
Inside your individual partitions you’d by no means hassle. You’d simply let the sluggish step end. And the funds that punishes you most quietly is time: miss it and nothing seems damaged in your aspect. An ideal reply that lands a number of seconds late nonetheless reads as a hit in your dashboards and as a failure to the shopper, and it’s the one restrict nothing within the stack enforces for you.
Right here’s the thesis, up entrance, as a result of all the things else serves it: as soon as high quality clears the bar, dependable supply is a query of variance, not pace. A predictable completion time beats a quick one with an extended tail, as a result of your clients can’t run their infrastructure in your greatest case; they need to construct to your worst.
What that is — and isn’t: workflows, not free reasoning brokers
One distinction up entrance, as a result of it adjustments all the things. That is about an agentic workflow: a recognized course of circulation with LLM-powered steps inside it, run by a deterministic orchestrator. It’s not a reasoning agent that decides its personal subsequent transfer at runtime. For a similar job, a workflow is solely sooner: it already is aware of the plan, skips the deliberation, and runs each impartial step in parallel, so it reaches the identical reply in a fraction of the time and price a reasoning agent would take. Each have their place (reasoning brokers are way more versatile), however they fail in another way and also you repair them in another way. A reasoning agent’s downside is deciding what to do; a workflow’s downside (the one clients really feel) is delivering what it already is aware of how one can do, with high quality, and in time. This text is concerning the latter.
How our system is constructed
The findings under come from our structure, and they need to generalize. These are peculiar, direct API calls. Nonetheless, it helps to know the setup so you may evaluate it to yours.
We run a customized orchestrator over managed third-party APIs (no self-hosted fashions on this dataset), and we run flagship fashions each straight via their suppliers (OpenAI, Anthropic, …) and thru managed platforms (Bedrock, Databricks, …), so high fashions have greater than 1 supplier. That lets us evaluate serving paths and transfer work between them.
Our workloads are a combination: easy agent calls, deep reasoning, extractions, JSON and free textual content outputs. For a big fraction of calls we synthesize a big truth base into a solution, so massive enter and small to medium outputs. The analytics on this article maintain enter and output measurement fixed inside buckets (see appendix).
The sluggish tails we encounter are largely transient. Observe that in case your structure is self-hosted or on devoted capability the tail could behave in another way, and can warrant one other strategy. Secondly, working a number of suppliers is what makes routing a hedge to a separate funds sensible. With a single supplier, fewer of those strikes can be found.
The declare, and the receipts
So right here’s the transfer that sounds backwards: we minimize a step off at 20-30 seconds even once we comprehend it may need answered completely slightly later — and that makes the system extra dependable, not much less.
That isn’t a hunch. It’s true on paper — the mathematics of heavy-tailed retries is unambiguous — and it’s true within the knowledge: a scan of properly over one million latest manufacturing LLM calls throughout our enterprise workloads — actual buyer site visitors. The very first thing that site visitors tells you is how unusual a single name’s timing actually is. A typical longer-output name comes again in a couple of dozen seconds. However one in 100 takes thirty seconds, typically a full minute or extra — for no motive linked to how a lot work it was doing.
That hole between the standard name and the sluggish one underlies a lot of this text. The remainder of the article critiques what to do about it.
Why the clock is unforgiving
A workflow isn’t judged on its common. It’s judged in opposition to a deadline. On common our flows end comfortably; nevertheless outlier runs in lengthy tails don’t. These tail runs aren’t damaged. They’d return an ideal reply a bit later, and on an inner run they’d depend as successes. On the shopper’s aspect, each certainly one of them is a failure. All the tail of your latency distribution, nevertheless appropriate, turns into an addition to your failure fee.
That’s why the quantity that issues right here isn’t common latency, it’s variance. A quick median buys you nothing in case your tail is lengthy.
The second squeeze is sunk value. The deeper you might be right into a workflow, the extra you’ve already spent: time, {dollars}, and your TPM quota. A failure on step 9 is much costlier than the identical failure on step two. You throw away all the things the workflow constructed and you’ve gotten much less of the clock left to shift gears. We by no means restart the entire workflow ourselves, however the buyer will. If we fail, they are going to nearly definitely retry, beginning the total circulation once more from the start. That compounds the issue on our aspect. It burns extra value, extra token funds, and the error funds on the SLA. And since the circumstances that made the run fail often haven’t modified, the retry has an identical probability of failing. Worse, it tends to occur throughout a high-TPM window. The worst potential time to pile further load onto an already-strained system, and precisely when the chances of failing once more are highest.
There’s a second multiplier, and it’s straightforward to overlook. The primary is the one from the opening: reliability compounds, so a sequence of individually glorious steps can nonetheless come out a coin flip1. However that failure is at all times informed as a narrative about correctness: getting a unsuitable reply.
Right here’s what you nearly by no means hear about: the very same compounding occurs on the clock. Each step provides its personal small probability of touchdown within the sluggish tail, and people probabilities stack. So the extra steps you chain, the extra possible it’s that a minimum of one of them blows the deadline, even when each step is individually quick. That’s the multiplier this text is about, and it’s the one the literature leaves out. So let’s take a look at the numbers.
What an LLM reply time really seems like
The everyday instances within the chart above sit in a reasonably tight band: each mannequin finishes a typical name someplace between eight and twenty seconds. The tails should not tight in any respect. One mannequin’s 99th-percentile name is available in round 30 seconds, one other’s previous 80. Comparable median, wildly totally different worst case. Promise a buyer your median and also you’re mendacity to the 1-in-20 and 1-in-100 calls within the tail, and a multi-step workflow hits these continuously. A quick typical time shouldn’t be a predictable one.
The apparent objection is that the sluggish calls are simply doing extra work: larger prompts, longer solutions. They aren’t. Pin each the immediate measurement and the response size and the tail barely strikes: inside a single measurement bucket (work held fastened), p99 nonetheless runs two to seven instances the median (Determine 4). The slowness isn’t about how a lot the decision has to do — in our site visitors it’s largely transient (queueing, scheduling, mid-stream competition, a supplier hiccup), which is strictly what makes it value interrupting.
One sluggish step sinks the entire run
You’d assume a workflow misses its deadline as a result of many steps have been every slightly sluggish. It nearly by no means occurs that approach. When a sequence blows its funds, it’s often one step that wandered into its tail whereas all the things else behaved fantastic. Mathematically, a sequence’s overrun is dominated by its single worst step, not by the buildup of mildly sluggish ones. The whole behaves like its most, not its sum.2
That’s excellent news. You don’t want each step quick. It’s essential to cease any single step from working away. Which is the cutoff.
Sidebar — The maths, briefly (skip except you want math)
Three outcomes sit beneath the argument:
- Compounding. Simply the arithmetic of impartial steps: n steps every succeeding with chance p provides pⁿ end-to-end. At p = 0.95, ten steps ≈ 60% and twenty ≈ 36% — multiplication, no modeling. The identical compounding hits the clock: every added step is one other impartial draw in opposition to the latency tail (the 2-7× p99/p50 we measure per name), so the chances that a minimum of one step blows its funds solely rise with size. Independence is the simplification — shared capability correlates actual steps — however it’s the conservative, illustrative case.
- The one massive soar. LLM latency is heavy-tailed (lognormal-ish), and the lognormal is subexponential. For impartial subexponential steps the tail of the sum is simply the sum of the tails — `P(ΣX_i > t) ≈ Σ P(X_i > t) ≈ P(maxᵢ X_i > t)` as t grows. In phrases: a sequence overruns as a result of one step hit its tail, not as a result of many have been mildly sluggish.2
- Hedging, and why it really works for any failure. Hearth n impartial makes an attempt and take the primary good one: if a single try fails with chance q, all n fail with chance qⁿ. That arithmetic doesn’t care what “fail” means — a blown deadline, a tough error, or a unsuitable reply all purchase down the identical approach, which is why the identical retry/race/fallback transfer serves each taste. For the timing taste particularly it additionally shrinks unfold: for the reason that variances of impartial steps add, `Var(ΣX_i) = Σ Var(X_i)`, capping every step’s tail shrinks the entire chain’s. All of it rests on the makes an attempt being impartial (contemporary attracts, contemporary queue) — which is strictly why a parallel re-draw collapses a transient tail (or an unfortunate dangerous reply) and does nothing for a deterministic one.3
The transfer: minimize early, then race
If a step has wandered into its tail, ready is the worst factor you are able to do — you’re spending your scarcest useful resource in your least possible payoff. So that you hand over early and check out once more in parallel: hearth a contemporary try and take whichever returns first. A contemporary try not often lands in the identical pothole, so two of them match contained in the time one caught name would have eaten — and the chances of each being sluggish are tiny (if one is sluggish with chance q, two are each sluggish with chance q²).3
The median barely strikes: about 10 seconds as a substitute of 12. The tail does the other: the 99th percentile drops from roughly 60 seconds to 25, and the run-to-run unfold is greater than minimize in half. You purchase predictability for the worth of some further tokens.
That worth is actual, and it pushes again. Racing doubles the token invoice on that step, and tokens are a shared, capped funds. So value is a real downward drive on how freely you retry and race. However run the arithmetic and it’s lopsided. Doubling one step prices you that step’s tokens, as soon as. Blowing the deadline throws away all the things you’ve already paid for, and the shopper nearly at all times retries, re-running all N steps of the workflow, a minimum of as soon as, typically extra. The deeper into the circulation you might be, the extra one-sided the commerce: a redundant try on step 9 is affordable subsequent to discarding steps one via 9 and watching them run once more. So that you hedge anyway. You simply don’t hedge indiscriminately, as a result of that shared token funds bites again hardest precisely once you most wish to spend it (extra on that stress shortly).
One nuance that decides which fallback to succeed in for: the course has to match why the step is failing.
- Sluggish for transient causes → re-draw, ideally in parallel. A contemporary try escapes the stall. (A plain serial retry is weaker right here on an extended step — you’d pay the lengthy era time twice.)
- Sluggish as a result of the work is genuinely massive → don’t re-run the identical name. Fall down to a sooner mannequin, or to an alternate path that reaches the identical consequence extra cheaply.
- Flawed, not sluggish → fall up to a extra succesful mannequin. Velocity gained’t repair a foul reply; functionality would possibly. (That is the standard flooring from earlier, enforced at runtime.)
Lower on the suitable sign
A solution time is admittedly two phases.4 The anticipate the first token is generally queueing and scheduling; the era that follows, token by token, is the remainder. Which section carries the tail decides what you place the cutoff on. And that is determined by how a lot the step writes.
For the longer steps this text is about (those that press in opposition to a deadline), the tail lives in era, not the first-token wait. A sluggish queue is a small slice of a forty-second name; the unfold that blows the funds is within the tokens. So minimize these on whole elapsed time, or on tokens emitted thus far in opposition to the time you’ve gotten left, not on time-to-first-token. (For brief steps the stability flips: with little to generate, the first-token wait is many of the name, and time-to-first-token turns into the cleaner minimize. Measure your individual steps to see which aspect you’re on.)
Two alerts are value wiring in regardless:
- No first token in any respect, previous the cutoff? That’s caught, not sluggish. Quit and hedge. A contemporary parallel try will get newly scheduled and nearly at all times wins.
- Tokens flowing however it’ll blow the funds? Don’t re-run it. You’d simply regenerate the identical size on the similar pace. Fall to a sooner mannequin.
And one failure no clock can catch: a step that returns on time however returns junk (e.g. it’s empty, truncated, or unparseable). A latency cutoff sails proper previous it; solely a high quality verify downstream will. For any step that’s alleged to return a selected form, the most cost effective such verify is a strict validation proper after the decision. Parse the consequence in opposition to the anticipated schema or object, and deal with a validation failure precisely like another: minimize and fall again (re-draw, or fall up to a extra succesful mannequin). It catches a significant slice of dangerous solutions earlier than they attain the following step. Slicing early buys you predictability, not correctness. Preserve these two jobs separate.
The catch: hedging spends the funds you’re shortest on
Racing has a clumsy property. The tail is worst when the system is busy. And “busy” is strictly when your tokens-per-minute funds has the least room left. So the one transfer that fixes the tail needs to spend tokens on the exact second they’re hardest to return by. Do it blindly and also you get a pile-on: sluggish calls set off hedges, hedges add load, load makes all the things slower, extra calls cross their cutoff. A latency downside turns into a rate-limit downside.
Two info make this much less forgiving than it first seems. The price is dedicated the moment you hearth the second name. Cancelling the loser frees your connection, however the supplier retains producing, and billing, the deserted try. There’s no clawback, so all of the management has to reside on the resolution to hedge, not after. And also you often can’t see how a lot funds is left. Estimating it’s potential however concerned, so any scheme that “eases off because the quota fills” is difficult to run in follow.
What works in follow is cruder and extra structural:
- Ship the hedge someplace with its personal funds. Token limits are per-model and per-provider, and most of us run a couple of (as famous in How our system is constructed). Routing the retry to a totally different mannequin or supplier will get a separate quota and an impartial draw. The identical transfer that escapes the stall additionally avoids spending the scarce funds twice.
- Preserve hedges uncommon by development. That is what the precomputed cutoffs already purchase you: with the edge set at every step’s measured p95, a hedge fires solely on the sluggish minority, so the additional spend stays small with no runtime accounting in any respect. (Identical cutoffs as the following part, no new equipment.)
- React to the alerts you really get. You most likely can’t learn headroom, however you may learn 429s and climbing latency. Deal with these because the cue to hedge much less and minimize later, no more.
- At actual saturation, cease hedging. As soon as the supplier is already returning rate-limit errors, extra makes an attempt solely deepen the opening. Downshift to a smaller, cheaper mannequin or shed the work as a substitute.
One lever we haven’t constructed, and provide solely as a course: an specific international cap that holds hedged calls to a small fraction of whole site visitors, impartial of the per-step selections. It’s the principled backstop the tail-at-scale work factors to;3 we set conservative cutoffs as a substitute and haven’t wanted it, however at larger hedge charges that’s the place we’d go subsequent.
Sidebar — A budget strikes you make first
Cutoffs and hedging are insurance coverage. You purchase much less of it if the workflow is constructed properly to start with. The defaults that fireside on each request, earlier than any reactive trick:
- Parallelism by design. Lay the circulation out as a dependency graph and run each step the second its inputs exist. Then go additional — design the dependencies out. Fewer dependencies means extra steps are leaves, and a leaf can fail cheaply with out taking the remainder of the graph down.
- Don’t name the mannequin in any respect once you don’t need to. Probably the most dependable name is the one you don’t make — use code, lookups, and validators wherever the work doesn’t really want a mannequin.
- Combine fashions per step, not per workflow. Quick and low-cost the place it’s sufficient; succesful the place it isn’t.
- Cache the deterministic elements. Don’t pay an LLM twice for a solution that may’t change.
The purpose right here: spend your reliability funds on construction first, so the clock work has much less to repair.
When do you really pull the set off?
The cutoff is a knob, not a relentless. How onerous you flip it comes down to 3 plain questions on every step:
- How a lot does the reply want this step? Good-to-have: let it go. Should-have: shield it.
- How a lot is ready on it? If nothing is determined by it, let it run to the deadline. If half the workflow is queued behind it, end it sooner, and ensure it’s proper, as a result of a unsuitable reply right here poisons all the things downstream.
- How a lot time is left? A lot: retry calmly. Nearly out: minimize quick and fall again.
The extra a step is must-have, load-bearing, and quick on time, the sooner you hearth the backup and the extra you’ll spend to hedge it. An non-obligatory, terminal, early step will get none of that. (“Early or late within the circulation” was by no means the actual axis. It was a proxy for a way a lot nonetheless is determined by this step.)
And also you don’t guess the quantity. You run the workflow many instances, measure every step’s latency curve (P95), and set the cutoff from that curve. Under the step’s worst case, weighted by the three questions. A step that often solutions in 20 seconds will get minimize at 30, although it may need succeeded at 60.
Why nearly no person does this
This isn’t onerous. It’s nuanced, and most groups don’t have the engine for it.
The favored workflow instruments, the Airflows and Temporals, have been constructed to make pipelines sturdy: retry, resume, don’t lose state, and so they’re superb at it. Their timeout recommendation follows from that purpose: set a per-step timeout longer than the slowest run and retry till it succeeds.5 That’s the suitable intuition when the job is to sturdy completion, and it’s precisely the unsuitable recommendation when the job is to complete in time. Your workflow engine will fortunately retry a step many instances; it has no notion of a step’s measured typical time and downstream implications, so it could actually’t minimize early and swap fashions. That isn’t a flaw. It’s by design.
The distributed-systems fundamentals are already on our aspect: work from a deadline funds, match every timeout to measured latency.6 We’re not contradicting that. We’re making use of it to a case these instruments don’t assume: a brief, non-resumable funds the place the suitable transfer on the cutoff is a sooner different, not the identical name once more. Identical precept, inverted course.
Takeaway
One factor, when you preserve nothing else: a predictable completion time beats a quick one with an extended tail. Low variance beats low latency. You may’t promise a buyer a median, solely a sure. All the things right here serves that sure. Slicing early, hedging, racing, designing out dependencies: every trades slightly common pace for lots much less variance. You hand over the suitable tail to purchase the left.
In a customer-facing agentic workflow, reliability is the product. The craft isn’t proudly owning a bag of retries and fallbacks, these are desk stakes. It’s deciding, per step, whether or not to hedge and when to surrender, from the constraints and the measured habits of your individual system.
APPENDIX
In regards to the creator
Frank Wittkampf is Head of Utilized AI Engineering at Databook. His staff architects, builds, and operates a totally customized AI stack together with deep reasoning, an agentic workflow engine, AI asset era, agentic harnesses, data base & context graph, AI pre-processing, multi-tenant AI configuration administration, and many others. This AI infrastructure powers the GTM groups of high Enterprise corporations like Microsoft, Salesforce, Amazon, Databricks, and lots of others.
A be aware on the information
The latency figures right here come from latest (June 2026), anonymized manufacturing site visitors throughout enterprise buyer workloads — roughly 1.2 million LLM calls over a 30-day window, not artificial benchmarks or a public hint. As described in How our system is constructed, these are direct calls to managed third-party APIs, which is a part of why the sluggish tail is basically transient. The numbers within the textual content describe the longer calls (output ≥ 600 tokens), since these are those that truly press in opposition to a deadline; shorter calls are sooner and fewer variable. All through, a “tail ratio” (p99/p50) holds name measurement fastened inside a bucket except acknowledged in any other case. Fashions are labeled by household and serving path solely; predictability is determined by the serving path (e.g. a managed API vs. a direct one), not simply the mannequin, so these are intentionally not a mannequin rating. Durations have been bucketed in one-second bins; a tough 90-second ceiling truncates solely the final ~0.2% of longer calls, so the tail you see is actual, not an artifact of the cap.
Isn’t the tail simply the larger calls?
The truthful objection to Determine 4: every row is a token bucket, not a hard and fast token depend, so perhaps the sluggish calls inside a cell are merely the bigger ones — extra to prefill, extra to generate — and the tail is simply measurement, not something transient.
It isn’t, and the information’s personal form exhibits why. If measurement drove the within-cell tail, two issues would comply with: the tail ratio would develop with the quantity of labor, and probably the most tightly bounded cells would have nearly no tail. Neither holds.
Two issues to learn off it. First, the tail ratio is flat at roughly 2–4× throughout each output-size column — it doesn’t climb because the work grows, so the tail doesn’t scale with the work. Second, and decisively, take a look at the leftmost column: these calls emit at most 50 output tokens, so era time bodily can’t range by greater than a couple of second — but the tail there may be nonetheless ~3.5×. There is no such thing as a measurement variable massive sufficient to supply that. The residual unfold is transient (queueing, scheduling, a momentary supplier hiccup), which is strictly what a contemporary try escapes.
Why these numbers look smaller than the two–7× quoted earlier: the column figures listed here are volume-weighted averages throughout many cells, which easy out the unfold, whereas the two–7× within the physique is the per-call envelope — the vary particular person cells really span. Identical knowledge, two totally different cuts: the averages present the tail doesn’t scale with work; the envelope exhibits how extensive it will get on any given name.
Notes & Footnotes
Observe: All photographs created by the creator.
1: Ten steps at 95% every ≈ 60% end-to-end; twenty ≈ 36% (assuming independence).
2: The lognormal lies within the subexponential class, the place the tail of a sum of impartial phrases is asymptotically the sum of the person tails: `P(S_n > t) ∼ Σ_i P(X_i > t) ∼ P(max_i X_i > t)` as t → ∞ — the “single massive soar” precept (Foss, Korshunov & Zachary, An Introduction to Heavy-Tailed and Subexponential Distributions, Springer, 2nd ed. 2013, eqs. 1.3 & 1.6). It’s an asymptotic assertion and assumes independence, so deal with it because the instinct for why one sluggish step dominates, not a plug-in formulation.
3: If every impartial try is sluggish with chance q, two parallel makes an attempt are each sluggish with chance q²; n makes an attempt, qⁿ. The basic hedged-request consequence (Dean & Barroso, “The Tail at Scale,” CACM 2013); in an agent setting, Winston et al. (arXiv:2605.21470, ICML 2026) select between serial, parallel, and hedged execution from measured latency curves. On our manufacturing knowledge, racing two makes an attempt minimize p99 on longer steps by greater than half (≈60s→25s) whereas sequential re-draw on whole time didn’t.
4: The cut up is customary in inference work: “time to first token” (queue + prefill) versus per-token era. See e.g. Agrawal et al., Taming the Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve (arXiv:2403.02310, 2024). In our manufacturing site visitors the tail for longer calls sits within the era section, not the first-token wait — which is why we minimize lengthy steps on whole elapsed time reasonably than time-to-first-token.
5: Temporal’s exercise timeouts are designed to complete ultimately, together with retries — therefore Begin-To-Shut set above the sluggish tail.
6: Google SRE, gRPC deadlines, and Spanner all advise propagating a complete funds and dropping work that may not assist the caller. We prolong the identical precept to a sync, non-resumable buyer funds.
inside your individual firm and nearly any failure is affordable: you retry, fall again, or doubtlessly even ignore it. Put that very same workflow behind a buyer’s API or MCP server and the grace is gone. Now just one factor issues: did the shopper get an accurate, usable consequence? Their course of is determined by yours delivering one. They, not you, now determine what counts as delivered. At Databook we course of billions of tokens for the world’s largest enterprises; this text relies on actual knowledge from manufacturing flows at scale. I hope it provides you some helpful insights.
Delivering that result’s more durable than it seems, as a result of LLMs are notoriously unreliable. They fail incessantly, in 4 flavors: an invalid reply (empty, unparseable, or just unsuitable), a tough error, no reply in any respect, or no reply in time. And the entire run solely succeeds if each step does, so the extra you chain collectively, the extra probabilities there are for certainly one of them to fail. A workflow of individually glorious steps can nonetheless come out a coin flip.
Inside your individual firm you may soak up each certainly one of these, as a result of you’ve gotten slack on each axis: retry the failed step, wait out the sluggish one, spend slightly extra, calm down the bar when you should. Put the identical workflow behind a buyer’s API and the slack vanishes, as a result of the run now has to clear three useful resource budgets on the similar time, none of which you set:
- Time — a window that closes whether or not or not you’re accomplished: a tough gateway timeout (one to 3 minutes, typically 5) that severs the connection mid-run, or one thing softer: an SLA, a caller blocked on the consequence, a course of that may solely wait so lengthy. And it doesn’t resume: when the window closes, the shopper simply retries, beginning the entire run over from zero.
- Value — now a margin, not a pool. Each run carries a worth the shopper already paid, so it has to return again worthwhile, not merely reasonably priced. And the shopper, not you, decides how typically it runs.
- Tokens and fee — a per-minute token funds (TPM) you share throughout each buyer directly, and so they are likely to name in the identical bursts. You hit the ceiling precisely when load is heaviest, which is strictly when latency is worst.
Below all three sits a tough flooring you by no means commerce beneath: high quality. The reply must be proper to depend in any respect. A quick, low-cost, on-time reply that’s unsuitable remains to be a failure. High quality isn’t a funds you spend down.
Any certainly one of these you could possibly handle by itself. The bind is that they apply collectively and pull in opposition to one another, so the plain repair for one spends one other. Wait out a sluggish step and also you blow the time window. Race a second copy to beat the clock and also you burn value and quota. Attain for a stronger mannequin to clear the standard flooring and also you get slower. Not one of the budgets are yours to loosen, so the one transfer left is to commerce intentionally throughout all of them directly — with out ever dropping under the ground.
That’s what makes a customer-facing workflow a genuinely totally different factor to construct, and it typically forces a playbook that, from the within, seems completely backwards:
- Kill a name that hasn’t failed
- Hearth a replica of a name you’re already paying for
- Drop to a weaker mannequin on function
Inside your individual partitions you’d by no means hassle. You’d simply let the sluggish step end. And the funds that punishes you most quietly is time: miss it and nothing seems damaged in your aspect. An ideal reply that lands a number of seconds late nonetheless reads as a hit in your dashboards and as a failure to the shopper, and it’s the one restrict nothing within the stack enforces for you.
Right here’s the thesis, up entrance, as a result of all the things else serves it: as soon as high quality clears the bar, dependable supply is a query of variance, not pace. A predictable completion time beats a quick one with an extended tail, as a result of your clients can’t run their infrastructure in your greatest case; they need to construct to your worst.
What that is — and isn’t: workflows, not free reasoning brokers
One distinction up entrance, as a result of it adjustments all the things. That is about an agentic workflow: a recognized course of circulation with LLM-powered steps inside it, run by a deterministic orchestrator. It’s not a reasoning agent that decides its personal subsequent transfer at runtime. For a similar job, a workflow is solely sooner: it already is aware of the plan, skips the deliberation, and runs each impartial step in parallel, so it reaches the identical reply in a fraction of the time and price a reasoning agent would take. Each have their place (reasoning brokers are way more versatile), however they fail in another way and also you repair them in another way. A reasoning agent’s downside is deciding what to do; a workflow’s downside (the one clients really feel) is delivering what it already is aware of how one can do, with high quality, and in time. This text is concerning the latter.
How our system is constructed
The findings under come from our structure, and they need to generalize. These are peculiar, direct API calls. Nonetheless, it helps to know the setup so you may evaluate it to yours.
We run a customized orchestrator over managed third-party APIs (no self-hosted fashions on this dataset), and we run flagship fashions each straight via their suppliers (OpenAI, Anthropic, …) and thru managed platforms (Bedrock, Databricks, …), so high fashions have greater than 1 supplier. That lets us evaluate serving paths and transfer work between them.
Our workloads are a combination: easy agent calls, deep reasoning, extractions, JSON and free textual content outputs. For a big fraction of calls we synthesize a big truth base into a solution, so massive enter and small to medium outputs. The analytics on this article maintain enter and output measurement fixed inside buckets (see appendix).
The sluggish tails we encounter are largely transient. Observe that in case your structure is self-hosted or on devoted capability the tail could behave in another way, and can warrant one other strategy. Secondly, working a number of suppliers is what makes routing a hedge to a separate funds sensible. With a single supplier, fewer of those strikes can be found.
The declare, and the receipts
So right here’s the transfer that sounds backwards: we minimize a step off at 20-30 seconds even once we comprehend it may need answered completely slightly later — and that makes the system extra dependable, not much less.
That isn’t a hunch. It’s true on paper — the mathematics of heavy-tailed retries is unambiguous — and it’s true within the knowledge: a scan of properly over one million latest manufacturing LLM calls throughout our enterprise workloads — actual buyer site visitors. The very first thing that site visitors tells you is how unusual a single name’s timing actually is. A typical longer-output name comes again in a couple of dozen seconds. However one in 100 takes thirty seconds, typically a full minute or extra — for no motive linked to how a lot work it was doing.
That hole between the standard name and the sluggish one underlies a lot of this text. The remainder of the article critiques what to do about it.
Why the clock is unforgiving
A workflow isn’t judged on its common. It’s judged in opposition to a deadline. On common our flows end comfortably; nevertheless outlier runs in lengthy tails don’t. These tail runs aren’t damaged. They’d return an ideal reply a bit later, and on an inner run they’d depend as successes. On the shopper’s aspect, each certainly one of them is a failure. All the tail of your latency distribution, nevertheless appropriate, turns into an addition to your failure fee.
That’s why the quantity that issues right here isn’t common latency, it’s variance. A quick median buys you nothing in case your tail is lengthy.
The second squeeze is sunk value. The deeper you might be right into a workflow, the extra you’ve already spent: time, {dollars}, and your TPM quota. A failure on step 9 is much costlier than the identical failure on step two. You throw away all the things the workflow constructed and you’ve gotten much less of the clock left to shift gears. We by no means restart the entire workflow ourselves, however the buyer will. If we fail, they are going to nearly definitely retry, beginning the total circulation once more from the start. That compounds the issue on our aspect. It burns extra value, extra token funds, and the error funds on the SLA. And since the circumstances that made the run fail often haven’t modified, the retry has an identical probability of failing. Worse, it tends to occur throughout a high-TPM window. The worst potential time to pile further load onto an already-strained system, and precisely when the chances of failing once more are highest.
There’s a second multiplier, and it’s straightforward to overlook. The primary is the one from the opening: reliability compounds, so a sequence of individually glorious steps can nonetheless come out a coin flip1. However that failure is at all times informed as a narrative about correctness: getting a unsuitable reply.
Right here’s what you nearly by no means hear about: the very same compounding occurs on the clock. Each step provides its personal small probability of touchdown within the sluggish tail, and people probabilities stack. So the extra steps you chain, the extra possible it’s that a minimum of one of them blows the deadline, even when each step is individually quick. That’s the multiplier this text is about, and it’s the one the literature leaves out. So let’s take a look at the numbers.
What an LLM reply time really seems like
The everyday instances within the chart above sit in a reasonably tight band: each mannequin finishes a typical name someplace between eight and twenty seconds. The tails should not tight in any respect. One mannequin’s 99th-percentile name is available in round 30 seconds, one other’s previous 80. Comparable median, wildly totally different worst case. Promise a buyer your median and also you’re mendacity to the 1-in-20 and 1-in-100 calls within the tail, and a multi-step workflow hits these continuously. A quick typical time shouldn’t be a predictable one.
The apparent objection is that the sluggish calls are simply doing extra work: larger prompts, longer solutions. They aren’t. Pin each the immediate measurement and the response size and the tail barely strikes: inside a single measurement bucket (work held fastened), p99 nonetheless runs two to seven instances the median (Determine 4). The slowness isn’t about how a lot the decision has to do — in our site visitors it’s largely transient (queueing, scheduling, mid-stream competition, a supplier hiccup), which is strictly what makes it value interrupting.
One sluggish step sinks the entire run
You’d assume a workflow misses its deadline as a result of many steps have been every slightly sluggish. It nearly by no means occurs that approach. When a sequence blows its funds, it’s often one step that wandered into its tail whereas all the things else behaved fantastic. Mathematically, a sequence’s overrun is dominated by its single worst step, not by the buildup of mildly sluggish ones. The whole behaves like its most, not its sum.2
That’s excellent news. You don’t want each step quick. It’s essential to cease any single step from working away. Which is the cutoff.
Sidebar — The maths, briefly (skip except you want math)
Three outcomes sit beneath the argument:
- Compounding. Simply the arithmetic of impartial steps: n steps every succeeding with chance p provides pⁿ end-to-end. At p = 0.95, ten steps ≈ 60% and twenty ≈ 36% — multiplication, no modeling. The identical compounding hits the clock: every added step is one other impartial draw in opposition to the latency tail (the 2-7× p99/p50 we measure per name), so the chances that a minimum of one step blows its funds solely rise with size. Independence is the simplification — shared capability correlates actual steps — however it’s the conservative, illustrative case.
- The one massive soar. LLM latency is heavy-tailed (lognormal-ish), and the lognormal is subexponential. For impartial subexponential steps the tail of the sum is simply the sum of the tails — `P(ΣX_i > t) ≈ Σ P(X_i > t) ≈ P(maxᵢ X_i > t)` as t grows. In phrases: a sequence overruns as a result of one step hit its tail, not as a result of many have been mildly sluggish.2
- Hedging, and why it really works for any failure. Hearth n impartial makes an attempt and take the primary good one: if a single try fails with chance q, all n fail with chance qⁿ. That arithmetic doesn’t care what “fail” means — a blown deadline, a tough error, or a unsuitable reply all purchase down the identical approach, which is why the identical retry/race/fallback transfer serves each taste. For the timing taste particularly it additionally shrinks unfold: for the reason that variances of impartial steps add, `Var(ΣX_i) = Σ Var(X_i)`, capping every step’s tail shrinks the entire chain’s. All of it rests on the makes an attempt being impartial (contemporary attracts, contemporary queue) — which is strictly why a parallel re-draw collapses a transient tail (or an unfortunate dangerous reply) and does nothing for a deterministic one.3
The transfer: minimize early, then race
If a step has wandered into its tail, ready is the worst factor you are able to do — you’re spending your scarcest useful resource in your least possible payoff. So that you hand over early and check out once more in parallel: hearth a contemporary try and take whichever returns first. A contemporary try not often lands in the identical pothole, so two of them match contained in the time one caught name would have eaten — and the chances of each being sluggish are tiny (if one is sluggish with chance q, two are each sluggish with chance q²).3
The median barely strikes: about 10 seconds as a substitute of 12. The tail does the other: the 99th percentile drops from roughly 60 seconds to 25, and the run-to-run unfold is greater than minimize in half. You purchase predictability for the worth of some further tokens.
That worth is actual, and it pushes again. Racing doubles the token invoice on that step, and tokens are a shared, capped funds. So value is a real downward drive on how freely you retry and race. However run the arithmetic and it’s lopsided. Doubling one step prices you that step’s tokens, as soon as. Blowing the deadline throws away all the things you’ve already paid for, and the shopper nearly at all times retries, re-running all N steps of the workflow, a minimum of as soon as, typically extra. The deeper into the circulation you might be, the extra one-sided the commerce: a redundant try on step 9 is affordable subsequent to discarding steps one via 9 and watching them run once more. So that you hedge anyway. You simply don’t hedge indiscriminately, as a result of that shared token funds bites again hardest precisely once you most wish to spend it (extra on that stress shortly).
One nuance that decides which fallback to succeed in for: the course has to match why the step is failing.
- Sluggish for transient causes → re-draw, ideally in parallel. A contemporary try escapes the stall. (A plain serial retry is weaker right here on an extended step — you’d pay the lengthy era time twice.)
- Sluggish as a result of the work is genuinely massive → don’t re-run the identical name. Fall down to a sooner mannequin, or to an alternate path that reaches the identical consequence extra cheaply.
- Flawed, not sluggish → fall up to a extra succesful mannequin. Velocity gained’t repair a foul reply; functionality would possibly. (That is the standard flooring from earlier, enforced at runtime.)
Lower on the suitable sign
A solution time is admittedly two phases.4 The anticipate the first token is generally queueing and scheduling; the era that follows, token by token, is the remainder. Which section carries the tail decides what you place the cutoff on. And that is determined by how a lot the step writes.
For the longer steps this text is about (those that press in opposition to a deadline), the tail lives in era, not the first-token wait. A sluggish queue is a small slice of a forty-second name; the unfold that blows the funds is within the tokens. So minimize these on whole elapsed time, or on tokens emitted thus far in opposition to the time you’ve gotten left, not on time-to-first-token. (For brief steps the stability flips: with little to generate, the first-token wait is many of the name, and time-to-first-token turns into the cleaner minimize. Measure your individual steps to see which aspect you’re on.)
Two alerts are value wiring in regardless:
- No first token in any respect, previous the cutoff? That’s caught, not sluggish. Quit and hedge. A contemporary parallel try will get newly scheduled and nearly at all times wins.
- Tokens flowing however it’ll blow the funds? Don’t re-run it. You’d simply regenerate the identical size on the similar pace. Fall to a sooner mannequin.
And one failure no clock can catch: a step that returns on time however returns junk (e.g. it’s empty, truncated, or unparseable). A latency cutoff sails proper previous it; solely a high quality verify downstream will. For any step that’s alleged to return a selected form, the most cost effective such verify is a strict validation proper after the decision. Parse the consequence in opposition to the anticipated schema or object, and deal with a validation failure precisely like another: minimize and fall again (re-draw, or fall up to a extra succesful mannequin). It catches a significant slice of dangerous solutions earlier than they attain the following step. Slicing early buys you predictability, not correctness. Preserve these two jobs separate.
The catch: hedging spends the funds you’re shortest on
Racing has a clumsy property. The tail is worst when the system is busy. And “busy” is strictly when your tokens-per-minute funds has the least room left. So the one transfer that fixes the tail needs to spend tokens on the exact second they’re hardest to return by. Do it blindly and also you get a pile-on: sluggish calls set off hedges, hedges add load, load makes all the things slower, extra calls cross their cutoff. A latency downside turns into a rate-limit downside.
Two info make this much less forgiving than it first seems. The price is dedicated the moment you hearth the second name. Cancelling the loser frees your connection, however the supplier retains producing, and billing, the deserted try. There’s no clawback, so all of the management has to reside on the resolution to hedge, not after. And also you often can’t see how a lot funds is left. Estimating it’s potential however concerned, so any scheme that “eases off because the quota fills” is difficult to run in follow.
What works in follow is cruder and extra structural:
- Ship the hedge someplace with its personal funds. Token limits are per-model and per-provider, and most of us run a couple of (as famous in How our system is constructed). Routing the retry to a totally different mannequin or supplier will get a separate quota and an impartial draw. The identical transfer that escapes the stall additionally avoids spending the scarce funds twice.
- Preserve hedges uncommon by development. That is what the precomputed cutoffs already purchase you: with the edge set at every step’s measured p95, a hedge fires solely on the sluggish minority, so the additional spend stays small with no runtime accounting in any respect. (Identical cutoffs as the following part, no new equipment.)
- React to the alerts you really get. You most likely can’t learn headroom, however you may learn 429s and climbing latency. Deal with these because the cue to hedge much less and minimize later, no more.
- At actual saturation, cease hedging. As soon as the supplier is already returning rate-limit errors, extra makes an attempt solely deepen the opening. Downshift to a smaller, cheaper mannequin or shed the work as a substitute.
One lever we haven’t constructed, and provide solely as a course: an specific international cap that holds hedged calls to a small fraction of whole site visitors, impartial of the per-step selections. It’s the principled backstop the tail-at-scale work factors to;3 we set conservative cutoffs as a substitute and haven’t wanted it, however at larger hedge charges that’s the place we’d go subsequent.
Sidebar — A budget strikes you make first
Cutoffs and hedging are insurance coverage. You purchase much less of it if the workflow is constructed properly to start with. The defaults that fireside on each request, earlier than any reactive trick:
- Parallelism by design. Lay the circulation out as a dependency graph and run each step the second its inputs exist. Then go additional — design the dependencies out. Fewer dependencies means extra steps are leaves, and a leaf can fail cheaply with out taking the remainder of the graph down.
- Don’t name the mannequin in any respect once you don’t need to. Probably the most dependable name is the one you don’t make — use code, lookups, and validators wherever the work doesn’t really want a mannequin.
- Combine fashions per step, not per workflow. Quick and low-cost the place it’s sufficient; succesful the place it isn’t.
- Cache the deterministic elements. Don’t pay an LLM twice for a solution that may’t change.
The purpose right here: spend your reliability funds on construction first, so the clock work has much less to repair.
When do you really pull the set off?
The cutoff is a knob, not a relentless. How onerous you flip it comes down to 3 plain questions on every step:
- How a lot does the reply want this step? Good-to-have: let it go. Should-have: shield it.
- How a lot is ready on it? If nothing is determined by it, let it run to the deadline. If half the workflow is queued behind it, end it sooner, and ensure it’s proper, as a result of a unsuitable reply right here poisons all the things downstream.
- How a lot time is left? A lot: retry calmly. Nearly out: minimize quick and fall again.
The extra a step is must-have, load-bearing, and quick on time, the sooner you hearth the backup and the extra you’ll spend to hedge it. An non-obligatory, terminal, early step will get none of that. (“Early or late within the circulation” was by no means the actual axis. It was a proxy for a way a lot nonetheless is determined by this step.)
And also you don’t guess the quantity. You run the workflow many instances, measure every step’s latency curve (P95), and set the cutoff from that curve. Under the step’s worst case, weighted by the three questions. A step that often solutions in 20 seconds will get minimize at 30, although it may need succeeded at 60.
Why nearly no person does this
This isn’t onerous. It’s nuanced, and most groups don’t have the engine for it.
The favored workflow instruments, the Airflows and Temporals, have been constructed to make pipelines sturdy: retry, resume, don’t lose state, and so they’re superb at it. Their timeout recommendation follows from that purpose: set a per-step timeout longer than the slowest run and retry till it succeeds.5 That’s the suitable intuition when the job is to sturdy completion, and it’s precisely the unsuitable recommendation when the job is to complete in time. Your workflow engine will fortunately retry a step many instances; it has no notion of a step’s measured typical time and downstream implications, so it could actually’t minimize early and swap fashions. That isn’t a flaw. It’s by design.
The distributed-systems fundamentals are already on our aspect: work from a deadline funds, match every timeout to measured latency.6 We’re not contradicting that. We’re making use of it to a case these instruments don’t assume: a brief, non-resumable funds the place the suitable transfer on the cutoff is a sooner different, not the identical name once more. Identical precept, inverted course.
Takeaway
One factor, when you preserve nothing else: a predictable completion time beats a quick one with an extended tail. Low variance beats low latency. You may’t promise a buyer a median, solely a sure. All the things right here serves that sure. Slicing early, hedging, racing, designing out dependencies: every trades slightly common pace for lots much less variance. You hand over the suitable tail to purchase the left.
In a customer-facing agentic workflow, reliability is the product. The craft isn’t proudly owning a bag of retries and fallbacks, these are desk stakes. It’s deciding, per step, whether or not to hedge and when to surrender, from the constraints and the measured habits of your individual system.
APPENDIX
In regards to the creator
Frank Wittkampf is Head of Utilized AI Engineering at Databook. His staff architects, builds, and operates a totally customized AI stack together with deep reasoning, an agentic workflow engine, AI asset era, agentic harnesses, data base & context graph, AI pre-processing, multi-tenant AI configuration administration, and many others. This AI infrastructure powers the GTM groups of high Enterprise corporations like Microsoft, Salesforce, Amazon, Databricks, and lots of others.
A be aware on the information
The latency figures right here come from latest (June 2026), anonymized manufacturing site visitors throughout enterprise buyer workloads — roughly 1.2 million LLM calls over a 30-day window, not artificial benchmarks or a public hint. As described in How our system is constructed, these are direct calls to managed third-party APIs, which is a part of why the sluggish tail is basically transient. The numbers within the textual content describe the longer calls (output ≥ 600 tokens), since these are those that truly press in opposition to a deadline; shorter calls are sooner and fewer variable. All through, a “tail ratio” (p99/p50) holds name measurement fastened inside a bucket except acknowledged in any other case. Fashions are labeled by household and serving path solely; predictability is determined by the serving path (e.g. a managed API vs. a direct one), not simply the mannequin, so these are intentionally not a mannequin rating. Durations have been bucketed in one-second bins; a tough 90-second ceiling truncates solely the final ~0.2% of longer calls, so the tail you see is actual, not an artifact of the cap.
Isn’t the tail simply the larger calls?
The truthful objection to Determine 4: every row is a token bucket, not a hard and fast token depend, so perhaps the sluggish calls inside a cell are merely the bigger ones — extra to prefill, extra to generate — and the tail is simply measurement, not something transient.
It isn’t, and the information’s personal form exhibits why. If measurement drove the within-cell tail, two issues would comply with: the tail ratio would develop with the quantity of labor, and probably the most tightly bounded cells would have nearly no tail. Neither holds.
Two issues to learn off it. First, the tail ratio is flat at roughly 2–4× throughout each output-size column — it doesn’t climb because the work grows, so the tail doesn’t scale with the work. Second, and decisively, take a look at the leftmost column: these calls emit at most 50 output tokens, so era time bodily can’t range by greater than a couple of second — but the tail there may be nonetheless ~3.5×. There is no such thing as a measurement variable massive sufficient to supply that. The residual unfold is transient (queueing, scheduling, a momentary supplier hiccup), which is strictly what a contemporary try escapes.
Why these numbers look smaller than the two–7× quoted earlier: the column figures listed here are volume-weighted averages throughout many cells, which easy out the unfold, whereas the two–7× within the physique is the per-call envelope — the vary particular person cells really span. Identical knowledge, two totally different cuts: the averages present the tail doesn’t scale with work; the envelope exhibits how extensive it will get on any given name.
Notes & Footnotes
Observe: All photographs created by the creator.
1: Ten steps at 95% every ≈ 60% end-to-end; twenty ≈ 36% (assuming independence).
2: The lognormal lies within the subexponential class, the place the tail of a sum of impartial phrases is asymptotically the sum of the person tails: `P(S_n > t) ∼ Σ_i P(X_i > t) ∼ P(max_i X_i > t)` as t → ∞ — the “single massive soar” precept (Foss, Korshunov & Zachary, An Introduction to Heavy-Tailed and Subexponential Distributions, Springer, 2nd ed. 2013, eqs. 1.3 & 1.6). It’s an asymptotic assertion and assumes independence, so deal with it because the instinct for why one sluggish step dominates, not a plug-in formulation.
3: If every impartial try is sluggish with chance q, two parallel makes an attempt are each sluggish with chance q²; n makes an attempt, qⁿ. The basic hedged-request consequence (Dean & Barroso, “The Tail at Scale,” CACM 2013); in an agent setting, Winston et al. (arXiv:2605.21470, ICML 2026) select between serial, parallel, and hedged execution from measured latency curves. On our manufacturing knowledge, racing two makes an attempt minimize p99 on longer steps by greater than half (≈60s→25s) whereas sequential re-draw on whole time didn’t.
4: The cut up is customary in inference work: “time to first token” (queue + prefill) versus per-token era. See e.g. Agrawal et al., Taming the Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve (arXiv:2403.02310, 2024). In our manufacturing site visitors the tail for longer calls sits within the era section, not the first-token wait — which is why we minimize lengthy steps on whole elapsed time reasonably than time-to-first-token.
5: Temporal’s exercise timeouts are designed to complete ultimately, together with retries — therefore Begin-To-Shut set above the sluggish tail.
6: Google SRE, gRPC deadlines, and Spanner all advise propagating a complete funds and dropping work that may not assist the caller. We prolong the identical precept to a sync, non-resumable buyer funds.
