TL;DR
don’t simply pause your brokers. They wreck your knowledge construction for those who swap fashions with out altering the payload.
A primary fallback router reveals a 100% completion fee in your dashboard however drops schema integrity to 0%. The pipeline finishes, however the output is damaged.
To repair it, the engine catches the error, rebuilds the payload for the backup mannequin, and saves the agent’s progress earlier than the swap. The benchmarks beneath run on normal Python 3.12 with zero exterior dependencies.
The Second The whole lot Broke
I used to be constructing a three-agent pipeline for EmiTechLogic: a Planner, an Executor, and a Validator working sequentially, every feeding structured JSON output to the subsequent. The pipeline labored advantageous in testing. Small hundreds, clear responses, no surprises.
Then I ran it underneath life like situations.
The first step, the Planner, completed. Step two, the Executor, hit a 429 fee restrict halfway via. My primary retry loop caught the error, swapped to a fallback mannequin, and stored working. The pipeline reported 100% completion. No exceptions thrown. No error logs.
However after I checked the downstream output, the boldness key was lacking. The outcome area was only a string: “incomplete – schema mismatch throughout swap.” The Validator obtained structurally damaged enter and had no approach to understand it. The pipeline completed on paper, however the knowledge was ineffective.
That’s the failure mode this text is about. Not the 429 itself, and never the retry loop. The problem is what occurs while you hand a fallback mannequin an unchanged payload formatted for a distinct engine completely.
The Full implementation accessible on GitHub: https://github.com/Emmimal/async-router-engine
Why This Failure Is So Laborious to See
Normal monitoring dashboards conceal this bug as a result of they solely observe course of completion. They examine if the API returned a 200 and if the thread exited cleanly. If the script finishes, the dashboard turns inexperienced. For multi-agent techniques, uptime is the mistaken metric.
The metric that issues is schema integrity. A pipeline that silently completes with corrupted fields is commonly worse than a tough crash. A crash forces an instantaneous repair, whereas silent knowledge corruption slips instantly into your database unnoticed [1].
These brokers are tightly coupled. The Executor expects the Planner’s actual JSON keys, and the Validator expects the Executor’s keys. When a mannequin swap breaks the construction at step two, that step doesn’t throw an error. It simply passes the malformed knowledge down the road, ruining the ultimate output someplace downstream the place you aren’t trying [2].
Price limiting isn’t a primary community infrastructure situation. It’s a knowledge integrity downside.
The Anatomy of a Silent Pipeline Failure
The failure mechanism is quiet however harmful.
API contracts are utterly inconsistent throughout totally different mannequin tiers. A premium mannequin enforces strict JSON mode and makes use of a devoted system immediate array. A less expensive fallback tier won’t assist an remoted system area in any respect, forcing you to merge directions instantly into the person textual content. It additionally not often ensures structured JSON outputs.
When a primary router catches a 429 and swaps the mannequin ID, it forwards the unique request payload unchanged. The fallback mannequin will get a configuration it may well’t parse. The community request succeeds as a result of the API technically returned textual content. No exception is thrown. The pipeline retains shifting, however the knowledge construction is already ruined. The subsequent agent simply will get uncooked textual content or lacking keys as a substitute of legitimate JSON.
Right here is the payload format throughout the three tiers in my system:
That final block is Technique A in my benchmark. The router swaps the mannequin ID, however the payload by no means adapts. The incoming response breaks structurally, however the pipeline logs a clear success anyway.
Constructing a Restoration Layer That Really Understands Context
I break up the logic into 4 components. Every one has a single job and nothing else.
The primary realization: not all failures are the identical
A primary router treats each API error as a set off to swap or retry. That logic fails immediately on context overflows or billing points.
It’s a must to separate the foundation causes. A 429 means the mannequin is briefly throttled, so that you swap and retry elsewhere. A context overflow means the immediate itself is simply too huge, so a retry is only a waste of tokens as a result of the payload must be trimmed first. A billing quota drop means all the supplier is lifeless for the session, so burning retries towards it’s pointless.
The detector handles this by parsing the uncooked error string towards particular sample lists. As an alternative of a generic crash, it returns a typed ThrottleEvent containing a clear motive code and a backoff window tied to the precise error:
The detector tracks supplier home windows utilizing time.monotonic() for cooldown decay. It retains observe of remaining backoff occasions and displays the request fee over a rolling 60-second window. Each routing try calls is_throttled() first. If a supplier is in backoff, the router skips it completely.
Normalizing Payloads: Cease Schema Corruption
The mannequin registry and the adapt_payload() technique separate Technique B from Technique A.
The registry holds a ModelProfile for every engine. This profile explicitly defines goal capabilities, together with native system immediate assist, JSON mode flags, schema buildings, and particular formatting templates.
When a swap occurs, the router calls adapt_payload() for the brand new goal. The adapter builds a very recent request dictionary as a substitute of forwarding the previous one. If the backup mannequin lacks a devoted system immediate area, the adapter injects these directions straight into the primary person message. It solely applies the response_format key or structural schemas if the goal mannequin natively helps them.
Right here is the payload transformation when dropping from model_a to model_c:
The three traces in adapt_payload() that examine supports_system_prompt earlier than deciding the place to inject the system content material are, within the benchmark, the distinction between 0% schema integrity and 100%.
Maintaining the pipeline alive throughout a swap
The state preserver prevents context loss throughout a mid-task swap.
When the Executor hits a 429 and the router switches fashions, the fallback engine begins chilly. It sees the uncooked message historical past however has no thought the Planner already ran, the place it sits within the execution sequence, or what schema it must return.
The state preserver fixes this by snapshotting all the execution context the second the throttle occasion fires, proper earlier than the swap. It logs the message historical past, system immediate, step indexes, present partial outputs, and the goal schema.
After the swap, build_resume_message() turns that snapshot right into a structured textual content block and appends it to the messages array. The fallback mannequin receives the context instantly:
[RESUME] Activity 'pipeline_run_3' interrupted at step 2/3 (Execute deliberate steps).
Earlier mannequin: model_a.
Progress: 67% full.
Partial output to this point:
{
"planner": {
"outcome": "Pipeline step accomplished with full structured evaluation.",
"confidence": 0.94,
"metadata": {"tokens_used": 312, "model_tier": "major"}
}
}
Proceed from the place the earlier mannequin stopped.
Required output schema: {"sort": "object", "required": ["result", "confidence"]}The fallback mannequin now is aware of precisely the place it’s, what got here earlier than, and what it wants to supply. That is what the 100% state preservation fee within the benchmark displays.
The Router
The router coordinates the detector, registry, and state preserver. The whole lot runs inside a bounded retry loop, executing these steps so as on each try:
Two configuration values matter most right here.
max_swaps limits what number of occasions a single name can swap fashions. With out this cover, back-to-back throttling throughout a number of suppliers would loop endlessly till max_retries runs out.
swap_delay_seconds provides a tiny 0.05-second pause earlier than hitting the brand new mannequin. This window is sufficiently small to keep away from hurting latency, however giant sufficient to cease you from slamming a supplier that’s already struggling. The max_swaps cap and swap_delay_seconds pause implement a light-weight model of the bulkhead and throttling patterns described by Nygard [3].
The Three-Agent Pipeline
The WorkflowOrchestrator runs three sequential steps: Planner, Executor, and Validator. Every step requires its personal system immediate, person message, and anticipated output schema. The output from one step feeds instantly into the subsequent, constructing a rising message historical past.
The orchestrator retains a shared messages checklist and a partial_output dictionary throughout all three steps. When a mid-step swap occurs, the state preserver packs each into the resume message. As an alternative of simply getting the present dialog, the fallback mannequin receives the complete context of what all the pipeline has produced as much as that time.
My previous fallback setups solely dealt with swaps on the mannequin stage and utterly ignored the pipeline. The backup mannequin obtained the brand new ID, but it surely had no thought the place it landed within the sequence. The state preserver fixes that disconnect.
The Benchmark
I ran three situations throughout ten runs every utilizing seed=42 for actual reproducibility. A mock supplier forces model_a to throttle at the 1st step each single time, forcing the fallback logic to kick in.
NO_ROUTER is the baseline with zero fallback logic. When model_a throttles, the pipeline kills the run. The mock returns a 503 for any secondary mannequin calls. That is what occurs while you simply wrap an API name in a primary strive/besides block, log the failure, and quit.
STRATEGY_A is primary routing. The router catches the 429 and swaps the mannequin ID, but it surely forwards the very same payload with out altering something. The mock supplier returns a degraded response with lacking keys and a schema error string. This matches how an actual backup mannequin behaves while you feed it an incompatible request format and it tries to guess its method via.
STRATEGY_B is this method. The router intercepts the 429,
snapshots the execution state, normalizes the payload for
the backup engine, injects the resume context, and carries on.
The benchmark isolates payload adaptation failures independently
of provider-specific latency, pricing, or mannequin high quality variations.
Technique A and Technique B differ solely in payload normalization and
state preservation logic. Utilizing a deterministic MockProvider permits
a direct causal comparability between these restoration methods with out introducing variability from community situations or variations in mannequin capabilities. Actual-world APIs will produce totally different latencies and outputs, however the structural failure measured right here — forwarding incompatible payloads throughout mannequin swaps — stays the identical.
Schema integrity was measured as the proportion of runs by which the ultimate agent output happy the anticipated JSON schema, together with all required fields and proper structural varieties.
BENCHMARK RESULTS
seed=42 | 10 runs per situation | throttle_at_step=1
Latency = simulated (seeded, deterministic, OS-independent)
═══════════════════════════════════════════════════════════════════════════
Metric NO_ROUTER STRATEGY_A STRATEGY_B
─────────────────────────────────────────────────────────────────────
Completion Price 0.0% 100.0% 100.0%
Schema Integrity Price 100.0% 0.0% 100.0%
State Preserved Price N/A 100.0% 100.0%
Supplier Swap Price 100.0% 100.0% 100.0%
Avg Simulated Latency (ms) 57.50 77.12 77.12
Avg Steps Accomplished 0.00 3.00 3.00
─────────────────────────────────────────────────────────────────────
Completion enchancment: +100.0% (NO_ROUTER -> STRATEGY_B)
Schema integrity enchancment: +100.0% (STRATEGY_A -> STRATEGY_B)Technique A and Technique B execute the identical variety of API calls. The benchmark reviews an identical simulated supplier latency as a result of the seeded MockProvider fashions solely API response time. The extra 50 ms swap delay configured in RouterConfig is an express operational overhead launched by Technique B throughout failover occasions.
Take a look at Technique A’s schema integrity: 0.0%. Each single run completed. Each single run returned damaged knowledge. The pipeline cleared all three steps and the orchestrator logged a hit, however the ultimate output was utterly unusable. In case your dashboards solely observe completion charges, this failure is totally invisible.
Technique B provides a 50ms swap delay per failover occasion (swap_delay_seconds=0.05 in RouterConfig). This configurable pause avoids hammering a supplier already underneath load earlier than switching to the fallback. Simulated latency for Technique A and Technique B is an identical within the benchmark as a result of each make the identical variety of API calls. The overhead is strictly the swap delay, not the snapshot, payload rebuild, or resume injection. In manufacturing, a 50 ms delay is usually negligible relative to end-to-end LLM latencies that always vary from a number of hundred milliseconds to a number of seconds. It’s a necessary trade-off.
State preservation just isn’t relevant to NO_ROUTER as a result of execution terminates earlier than restoration happens.
Sincere Design Selections
The payload adapter is strictly rule-based, not discovered. Each ModelProfile is hand-written. If you wish to add a brand new mannequin, you manually map out its capabilities, templates, and schemas.
This design is intentional. A rule-based setup is 100% auditable. You’ll be able to learn the profile and know the precise transformation that may occur. A discovered adapter creates an opaque black field proper while you want transparency most throughout a stay fallback.
The resume message isn’t a structured area both. It’s simply plain textual content. build_resume_message() merely drops a uncooked string into an everyday person message.
If a mannequin helps system prompts, injecting the context there could be cleaner. However the present setup works throughout all three mannequin tiers, together with model_c which has no system immediate assist in any respect. Compatibility received over magnificence.
Utilizing a mock supplier retains the experiment managed. Actual APIs introduce community lag, billing prices, and timing variables that make benchmark outcomes unpredictable.
Technique A’s failure is completely structural. It occurs as a result of the payloads aren’t normalized, not due to a random timing fluke. The mock isolates this flaw cleanly and retains the take a look at utterly reproducible.
The benchmark runs with max_retries=4. The default of three is conservative for a two-provider setup — elevate it in case your registry has greater than three tiers. The cap exists to keep away from runaway prices on genuinely unavailable suppliers.
What This Means for How You Construct Agentic Methods
You can not delegate fee restrict dealing with to a generic retry library. Generic libraries catch exceptions and retry. They don’t perceive payload contracts between mannequin tiers, they don’t snapshot agent state, and so they can’t normalize system prompts for suppliers that don’t assist a devoted system area. In case your fallback logic is simply catching an exception, swapping the mannequin ID, and retrying, you’re working Technique A. Your dashboards will present a wholesome completion fee, however your schema integrity might be zero with out you realizing it.
The repair begins with error classification. A 429, a quota exhaustion, a context overflow, and a supplier timeout are 4 totally different issues that want 4 totally different responses. Treating them identically burns retries on failures a retry won’t ever repair.
Payload normalization is the place Technique A breaks down. The request must be rebuilt from scratch for the goal mannequin, not forwarded unchanged. The one examine on supports_system_prompt earlier than deciding the place to inject system content material is all the distinction between 0% and 100% schema integrity within the benchmark. That’s one conditional. It prices nothing.
State must be snapshotted earlier than the swap, not after. If the fallback mannequin additionally throttles, you want the context from the unique failure level. A snapshot taken after a failed restoration try captures the mistaken state.
The final piece is the resume message. The fallback mannequin begins chilly. After I examined this with out the resume message, model_b picked up the message historical past and tried to re-execute the Planner’s step as a substitute of continuous from the Executor. It had no approach to know the place it landed. The pipeline accomplished, the output was mistaken, and nothing flagged it. Injecting the resume context explicitly is the one approach to inform the fallback mannequin what already occurred and what it nonetheless wants to supply.
What’s Lacking and What Comes Subsequent
The StatePreserver is the half I’m least happy with. Snapshots stay in reminiscence and disappear the second the method crashes, which implies a restart loses the whole lot. I need to swap the dictionary for a SQLite backend — the interface stays the identical, however the state survives. The mannequin choice can also be too inflexible proper now. The registry picks the subsequent mannequin by precedence order and that’s it. What I really need is for it to take a look at which fallback has the very best schema integrity observe file for a given schema and route there as a substitute — the stats() technique already collects sufficient knowledge to make that decision. And the mock supplier must go. Wiring in an actual Anthropic or OpenAI shopper is a one-function change, however I haven’t achieved it but as a result of the benchmark wanted to remain managed and reproducible.
Closing
I constructed this as a result of my pipeline was silently damaged. The 429 errors and mannequin swaps had been seen, and completion charges appeared clear. What went unnoticed was that each fallback response had a null confidence area and an “incomplete” outcome string. The validator was processing damaged knowledge each time the first mannequin throttled. Throughout load testing, that was more often than not.
The code requires zero exterior dependencies and makes use of solely the usual library (asyncio, dataclasses, enum, hashlib, json, random, time):
- Price restrict detector: ~160 traces
- Payload adapter: A single technique within the mannequin registry
- State preserver: ~140 traces (together with the resume message builder)
Writing the code wasn’t the troublesome half. The exhausting half was realizing {that a} accomplished pipeline just isn’t the identical as a working pipeline. Normal mannequin swapping confuses these two metrics. The completion counter goes up, the output is damaged, and no one notices till a downstream system fails three steps faraway from the trigger.
The Takeaway
Construct your fallback logic for manufacturing actuality. Deal with a mannequin swap as a knowledge integrity occasion, not an infrastructure retry.
Snapshot earlier than you swap, adapt the payload earlier than you ship it, and inform the fallback mannequin explicitly the place it landed.
Full code: https://github.com/Emmimal/async-router-engine
References
[1] Sculley, D., Holt, G., Golovin, D., Davydov, E., Phillips, T., Ebner, D., Chaudhary, V., Younger, M., Crespo, J.-F., & Dennison, D. (2015). Hidden technical debt in machine studying techniques. Advances in Neural Data Processing Methods, 28. https://papers.nips.cc/paper/2015/hash/86df7dcfd896fcaf2674f757a2463eba-Summary.html
[2] Anthropic. (2024, December 19). Constructing efficient brokers. https://www.anthropic.com/engineering/building-effective-agents
[3] Nygard, M. T. (2018). Launch It!: Design and Deploy Manufacturing-Prepared Software program (2nd ed.). Pragmatic Bookshelf. [Circuit breaker and bulkhead patterns]
Disclosure
All code on this article was written by me and is unique work, developed and examined on Python 3.12 (Home windows 11, CPU solely). Benchmark numbers are from precise runs of benchmarks/benchmark.py utilizing MockProvider with seed=42 and are absolutely reproducible by working the file on an ordinary Python set up with no packages to put in. Latency figures mirror deterministic simulated latency gathered by the seeded mock supplier — not wall-clock measurement — guaranteeing an identical outcomes throughout all machines and runs. The MockProvider simulates supplier habits deterministically: no actual LLM API calls are made within the benchmark. I’ve no monetary relationship with any instrument, library, or firm talked about on this article.
