Most manufacturing AI brokers nonetheless ship each LLM name to the identical costly frontier mannequin. Classification steps, easy instrument calls, progress checks, and exhausting reasoning all hit the identical endpoint. The result’s pointless price and latency. NVIDIA NeMo Switchyard solves this.
It’s an open-source routing layer (proxy + library) that sits between your agent and the fashions. It decides, request by request or flip by flip, which mannequin ought to deal with the work. On this tutorial, we’ll construct a working two-model router and progressively transfer from random routing to content-aware routing. So, let’s get began.
What Precisely Does Switchyard Do?
A traditional LLM utility would possibly seem like this:
Utility
|
v
GPT / Claude / Native LLMSwitchyard provides a routing layer:
Utility
|
v
Switchyard
/
v v
Low cost Highly effective
Mannequin MannequinThe applying doesn’t have to know which upstream mannequin finally serves the request. Switchyard selects the precise goal and forwards the request. Let’s examine this virtually.
Step 1: Putting in Switchyard
For the CLI/server path, the mission documentation gives a uv set up route:
uv instrument set up "nemo-switchyard[cli,server]"Confirm the set up:
switchyard --versionOutput:
switchyard 0.2.0
nemo-switchyard v0.2.0Alternatively, the native Rust server may be put in immediately with Cargo:
cargo set up --locked switchyard-serverFor this tutorial, we’ll route fashions via OpenRouter, so export your API key:
export OPENROUTER_API_KEY="your-key-here"Don’t retailer the API key immediately within the configuration file.
Step 2: Understanding a Switchyard Configuration
Let’s begin with the best potential setup: two fashions and random routing.
Create a YAML file named routes.random.yaml and add:
defaults:
base_url: https://openrouter.ai/api/v1
api_key: ${OPENROUTER_API_KEY}
routes:
ab-test:
kind: random_routing
robust:
mannequin: openai/gpt-4o
weak:
mannequin: openai/gpt-4o-mini
strong_probability: 0.3
rng_seed: 42
fallback_target_on_evict: weakThe important thing setting is:
strong_probability: 0.3Switchyard interprets this as roughly:
30% -> robust mannequin
70% -> weak mannequinRandom routing will not be clever routing, however it’s helpful for A/B exams and for validating the proxy earlier than introducing a classifier. fallback_target_on_evict is required for this route kind and refers to a tier ID similar to robust or weak.
Step 3: Beginning the Routing Server
Begin Switchyard with:
switchyard serve
-c routes.random.yaml
--host 127.0.0.1
--port 4000There isn’t any --dry-run possibility within the examined serve CLI. Beginning the server is successfully the validation step: an invalid routing bundle fails throughout startup. You’ll be able to confirm that the proxy is alive with:
curl -s http://127.0.0.1:4000/well beingOutput:
{"standing":"okay"}Step 4: Sending a Request Via the Router
Now ship an OpenAI-compatible request:
curl http://localhost:4000/v1/chat/completions
-H "Content material-Kind: utility/json"
-d '{"mannequin":"ab-test","messages":[{"role":"user","content":"Explain gradient descent in simple terms."}]}'Discover this subject:
"mannequin": "ab-test"Your shopper will not be asking for a selected mannequin (gpt-4o or gpt-4o-mini). Switchyard chooses the precise mannequin. For this instance, the request landed on the weak tier:
"mannequin": "openai/gpt-4o-mini",
"utilization": { "prompt_tokens": 14, "completion_tokens": 247, "price": 0.0001503 }Response:
Gradient descent is a technique utilized in optimization to search out the minimal of a perform. Think about you are on a hilly panorama, and your objective is to get to the bottom level within the valley. This is the way it works, step-by-step:
1) Begin at a Random Level: You start at a random location on the hill.
2) Discover the Slope: You go searching and decide the steepness of the hill (the gradient) at your present location. This tells you which of them route is downhill.
3) Take a Step Downhill: You're taking a step within the route that goes down the steepest slope. The size of your step is named the "studying charge" — in case you take small steps, you are cautious, whereas bigger steps will get you there sooner however would possibly lead you off target.
4) Repeat: You retain repeating this course of, recalculating the slope and stepping down till you possibly can't go any decrease — that is the underside of the valley or the minimal of the perform.
In easy phrases, gradient descent is about marching down the hill step-by-step till you attain the bottom level. It is broadly utilized in machine studying to regulate fashions in order that they make higher predictions.Step 5: Upgrading to Clever Routing
Random routing is nice for experiments, however suppose we wish this conduct:
Easy request — low cost mannequin
Exhausting request — robust mannequin
Switchyard gives a classifier route for precisely this objective. The classifier estimates whether or not the weaker mannequin can remedy the duty, then applies a configured threshold. Create routes.sensible.yaml and write this configuration:
defaults:
base_url: https://openrouter.ai/api/v1
api_key: ${OPENROUTER_API_KEY}
routes:
sensible:
kind: deterministic
classifier:
mannequin: openai/gpt-4o-mini
robust:
mannequin: openai/gpt-4o
weak:
mannequin: openai/gpt-4o-mini
profile: normal
session_affinity: true
fallback_target_on_evict: weakAnd begin it:
switchyard serve
-c routes.sensible.yaml
--host 127.0.0.1
--port 4000Now there are three roles:
classifier
|
| predicts weak-model functionality
v
+-------------------+
| Ought to weak remedy?|
+-------------------+
/
/
sure no
| |
v v
weak robustThe classifier produces a structured estimate containing a price referred to as p_solve: an estimate of the chance that the weak mannequin can efficiently full the request.
Step 6: Testing the Sensible Route
Strive a straightforward query:
curl http://localhost:4000/v1/chat/completions
-H "Content material-Kind: utility/json"
-d '{
"mannequin": "sensible",
"messages": [
{
"role": "user",
"content": "What is 15% of 200?"
}
]
}'Output:
To search out 15% of 200, you possibly can multiply 200 by 0.15:
200 × 0.15 = 30
So, 15% of 200 is 30.Then strive a more durable one:
curl http://localhost:4000/v1/chat/completions
-H "Content material-Kind: utility/json"
-d '{
"mannequin": "sensible",
"max_tokens": 1500,
"messages": [
{
"role": "user",
"content": "Find the race condition in a distributed job queue where workers acquire leases using non-transactional Redis operations, then propose a failure-safe redesign."
}
]
}'Output:
In a distributed job queue system utilizing Redis to handle and lease
jobs to employees, race circumstances can happen if a number of employees
try to accumulate a lease for a similar job concurrently utilizing
non-transactional operations. This may result in a number of employees
incorrectly believing they've efficiently acquired the lease,
leading to duplicate processing of the identical job.
### Typical Race Situation Situation
...
By incorporating these redesign parts into the distributed job
queue structure, race circumstances may be considerably decreased
and job leases may be dealt with extra reliably and safely.We did not hard-code the mannequin choice right here. As a substitute, the classifier determines the suitable tier for every immediate and routes the request accordingly. If you happen to have a look at the logs, you possibly can see which mannequin was finally chosen for every request.
| Immediate | Served Mannequin | Tier | Latency |
|---|---|---|---|
| “What’s 15% of 200?” | openai/gpt-4o-mini |
weak | 1,428 ms |
| Redis race-condition redesign | openai/gpt-4o |
robust | 4,475 ms |
Step 7: Routing Coding Brokers Based mostly on Their Progress
Immediate problem will not be the one helpful routing sign.
Contemplate a coding agent working for 30 turns. It could spend early turns exploring recordsdata, debugging failures, and reasoning about structure. Later turns could merely apply a longtime plan or make repetitive edits. Utilizing the strongest mannequin for each flip wastes inference price range. Switchyard’s stage_router is designed for this type of multi-turn workload. It makes use of dialog and tool-result alerts to resolve whether or not a flip ought to go to a succesful or environment friendly tier.
You’ll be able to create a configuration like this:
routes:
stage:
kind: stage_router
robust:
mannequin: openai/gpt-4o
weak:
mannequin: openai/gpt-4o-mini
picker: efficient_first
confidence_threshold: 0.5
signal_recent_window: 3
fallback_target_on_evict: weakThe thought is:
Agent flip
|
v
Latest progress / failure alerts
|
v
Is further functionality helpful now?
/
/
weak robustRight here, the router seems for alerts related to issues similar to errors, repeated unproductive conduct, exploration, and up to date productive adjustments. The objective is to order the stronger mannequin for turns the place further functionality seems helpful.
Step 8: Escalating Solely After the Weak Mannequin Struggles
One other technique is to keep away from predicting problem up entrance.
Let a budget mannequin strive first, then escalate when proof of sustained hassle seems. The movement turns into:
Request
|
v
Weak mannequin
|
v
Choose consequence
/
okay struggling
| |
v v
keep robust mannequinSwitchyard calls this escalation routing. You’ll be able to create a configuration like this:
routes:
agent:
kind: escalation_router
robust:
mannequin: openai/gpt-4o
weak:
mannequin: openai/gpt-4o-mini
choose:
mannequin: openai/gpt-4o-mini
confirmations: 2
recent_turn_window: 28
window_message_chars: 500
fallback_target_on_evict: weakThat is conceptually totally different from up-front deterministic classification. Deterministic/functionality routing asks:
How troublesome does this request seem?Escalation routing asks:
Is the weak mannequin really moving into hassle?This makes escalation helpful for long-running agent periods the place activity problem can change over time.
Step 9: Measuring Whether or not Routing Is Truly Serving to
A router is just helpful if it improves the quality-cost trade-off. Switchyard exposes Prometheus metrics and statistics round requests, errors, latency, tokens, and routing conduct. The mission additionally helps structured request telemetry and optionally available routing logs.
You may get server metrics with:
curl -s http://localhost:4000/metrics | headand mixture JSON statistics:
curl -s http://localhost:4000/v1/stats | python3 -m json.instrumentFor experiments, examine at the least three runs:
| Configuration | Objective |
|---|---|
| At all times robust | High quality ceiling and price baseline |
| At all times weak | Low cost baseline |
| Switchyard router | Check whether or not routing captures most strong-model high quality at decrease price |
The extra helpful query will not be whether or not the router was 85% correct, however how a lot of the robust mannequin’s high quality did routing protect, and the way a lot price and latency did it cut back? For instance:
Sturdy-only:
$20
92% activity success
Weak-only:
$5
71% activity success
Router:
$9
89% activity successThis tells you whether or not routing is economically helpful.
Ultimate Ideas
As LLM programs grow to be extra agentic, the query is shifting from:
Which mannequin ought to I exploit?
to:
Which mannequin ought to I exploit for this request, at this level within the workflow, below this price price range?
Switchyard is NVIDIA’s try to show that call into reusable infrastructure.
For a primary experiment, do not bounce immediately into stage routing or advanced agent escalation.
Begin with two fashions.
Measure them independently.
Use weighted random routing to confirm your setup.
Then introduce capability-based routing and measure whether or not it preserves a lot of the robust mannequin’s high quality whereas shifting a significant share of requests to the cheaper tier.
This experiment provides you one thing way more helpful than one other LLM benchmark:
a quality-versus-cost curve to your precise workload.And that’s finally what clever mannequin routing is making an attempt to optimize.
Kanwal Mehreen is a machine studying engineer and a technical author with a profound ardour for knowledge science and the intersection of AI with drugs. She co-authored the book “Maximizing Productiveness with ChatGPT”. As a Google Technology Scholar 2022 for APAC, she champions range and tutorial excellence. She’s additionally acknowledged as a Teradata Variety in Tech Scholar, Mitacs Globalink Analysis Scholar, and Harvard WeCode Scholar. Kanwal is an ardent advocate for change, having based FEMCodes to empower girls in STEM fields.
Most manufacturing AI brokers nonetheless ship each LLM name to the identical costly frontier mannequin. Classification steps, easy instrument calls, progress checks, and exhausting reasoning all hit the identical endpoint. The result’s pointless price and latency. NVIDIA NeMo Switchyard solves this.
It’s an open-source routing layer (proxy + library) that sits between your agent and the fashions. It decides, request by request or flip by flip, which mannequin ought to deal with the work. On this tutorial, we’ll construct a working two-model router and progressively transfer from random routing to content-aware routing. So, let’s get began.
What Precisely Does Switchyard Do?
A traditional LLM utility would possibly seem like this:
Utility
|
v
GPT / Claude / Native LLMSwitchyard provides a routing layer:
Utility
|
v
Switchyard
/
v v
Low cost Highly effective
Mannequin MannequinThe applying doesn’t have to know which upstream mannequin finally serves the request. Switchyard selects the precise goal and forwards the request. Let’s examine this virtually.
Step 1: Putting in Switchyard
For the CLI/server path, the mission documentation gives a uv set up route:
uv instrument set up "nemo-switchyard[cli,server]"Confirm the set up:
switchyard --versionOutput:
switchyard 0.2.0
nemo-switchyard v0.2.0Alternatively, the native Rust server may be put in immediately with Cargo:
cargo set up --locked switchyard-serverFor this tutorial, we’ll route fashions via OpenRouter, so export your API key:
export OPENROUTER_API_KEY="your-key-here"Don’t retailer the API key immediately within the configuration file.
Step 2: Understanding a Switchyard Configuration
Let’s begin with the best potential setup: two fashions and random routing.
Create a YAML file named routes.random.yaml and add:
defaults:
base_url: https://openrouter.ai/api/v1
api_key: ${OPENROUTER_API_KEY}
routes:
ab-test:
kind: random_routing
robust:
mannequin: openai/gpt-4o
weak:
mannequin: openai/gpt-4o-mini
strong_probability: 0.3
rng_seed: 42
fallback_target_on_evict: weakThe important thing setting is:
strong_probability: 0.3Switchyard interprets this as roughly:
30% -> robust mannequin
70% -> weak mannequinRandom routing will not be clever routing, however it’s helpful for A/B exams and for validating the proxy earlier than introducing a classifier. fallback_target_on_evict is required for this route kind and refers to a tier ID similar to robust or weak.
Step 3: Beginning the Routing Server
Begin Switchyard with:
switchyard serve
-c routes.random.yaml
--host 127.0.0.1
--port 4000There isn’t any --dry-run possibility within the examined serve CLI. Beginning the server is successfully the validation step: an invalid routing bundle fails throughout startup. You’ll be able to confirm that the proxy is alive with:
curl -s http://127.0.0.1:4000/well beingOutput:
{"standing":"okay"}Step 4: Sending a Request Via the Router
Now ship an OpenAI-compatible request:
curl http://localhost:4000/v1/chat/completions
-H "Content material-Kind: utility/json"
-d '{"mannequin":"ab-test","messages":[{"role":"user","content":"Explain gradient descent in simple terms."}]}'Discover this subject:
"mannequin": "ab-test"Your shopper will not be asking for a selected mannequin (gpt-4o or gpt-4o-mini). Switchyard chooses the precise mannequin. For this instance, the request landed on the weak tier:
"mannequin": "openai/gpt-4o-mini",
"utilization": { "prompt_tokens": 14, "completion_tokens": 247, "price": 0.0001503 }Response:
Gradient descent is a technique utilized in optimization to search out the minimal of a perform. Think about you are on a hilly panorama, and your objective is to get to the bottom level within the valley. This is the way it works, step-by-step:
1) Begin at a Random Level: You start at a random location on the hill.
2) Discover the Slope: You go searching and decide the steepness of the hill (the gradient) at your present location. This tells you which of them route is downhill.
3) Take a Step Downhill: You're taking a step within the route that goes down the steepest slope. The size of your step is named the "studying charge" — in case you take small steps, you are cautious, whereas bigger steps will get you there sooner however would possibly lead you off target.
4) Repeat: You retain repeating this course of, recalculating the slope and stepping down till you possibly can't go any decrease — that is the underside of the valley or the minimal of the perform.
In easy phrases, gradient descent is about marching down the hill step-by-step till you attain the bottom level. It is broadly utilized in machine studying to regulate fashions in order that they make higher predictions.Step 5: Upgrading to Clever Routing
Random routing is nice for experiments, however suppose we wish this conduct:
Easy request — low cost mannequin
Exhausting request — robust mannequin
Switchyard gives a classifier route for precisely this objective. The classifier estimates whether or not the weaker mannequin can remedy the duty, then applies a configured threshold. Create routes.sensible.yaml and write this configuration:
defaults:
base_url: https://openrouter.ai/api/v1
api_key: ${OPENROUTER_API_KEY}
routes:
sensible:
kind: deterministic
classifier:
mannequin: openai/gpt-4o-mini
robust:
mannequin: openai/gpt-4o
weak:
mannequin: openai/gpt-4o-mini
profile: normal
session_affinity: true
fallback_target_on_evict: weakAnd begin it:
switchyard serve
-c routes.sensible.yaml
--host 127.0.0.1
--port 4000Now there are three roles:
classifier
|
| predicts weak-model functionality
v
+-------------------+
| Ought to weak remedy?|
+-------------------+
/
/
sure no
| |
v v
weak robustThe classifier produces a structured estimate containing a price referred to as p_solve: an estimate of the chance that the weak mannequin can efficiently full the request.
Step 6: Testing the Sensible Route
Strive a straightforward query:
curl http://localhost:4000/v1/chat/completions
-H "Content material-Kind: utility/json"
-d '{
"mannequin": "sensible",
"messages": [
{
"role": "user",
"content": "What is 15% of 200?"
}
]
}'Output:
To search out 15% of 200, you possibly can multiply 200 by 0.15:
200 × 0.15 = 30
So, 15% of 200 is 30.Then strive a more durable one:
curl http://localhost:4000/v1/chat/completions
-H "Content material-Kind: utility/json"
-d '{
"mannequin": "sensible",
"max_tokens": 1500,
"messages": [
{
"role": "user",
"content": "Find the race condition in a distributed job queue where workers acquire leases using non-transactional Redis operations, then propose a failure-safe redesign."
}
]
}'Output:
In a distributed job queue system utilizing Redis to handle and lease
jobs to employees, race circumstances can happen if a number of employees
try to accumulate a lease for a similar job concurrently utilizing
non-transactional operations. This may result in a number of employees
incorrectly believing they've efficiently acquired the lease,
leading to duplicate processing of the identical job.
### Typical Race Situation Situation
...
By incorporating these redesign parts into the distributed job
queue structure, race circumstances may be considerably decreased
and job leases may be dealt with extra reliably and safely.We did not hard-code the mannequin choice right here. As a substitute, the classifier determines the suitable tier for every immediate and routes the request accordingly. If you happen to have a look at the logs, you possibly can see which mannequin was finally chosen for every request.
| Immediate | Served Mannequin | Tier | Latency |
|---|---|---|---|
| “What’s 15% of 200?” | openai/gpt-4o-mini |
weak | 1,428 ms |
| Redis race-condition redesign | openai/gpt-4o |
robust | 4,475 ms |
Step 7: Routing Coding Brokers Based mostly on Their Progress
Immediate problem will not be the one helpful routing sign.
Contemplate a coding agent working for 30 turns. It could spend early turns exploring recordsdata, debugging failures, and reasoning about structure. Later turns could merely apply a longtime plan or make repetitive edits. Utilizing the strongest mannequin for each flip wastes inference price range. Switchyard’s stage_router is designed for this type of multi-turn workload. It makes use of dialog and tool-result alerts to resolve whether or not a flip ought to go to a succesful or environment friendly tier.
You’ll be able to create a configuration like this:
routes:
stage:
kind: stage_router
robust:
mannequin: openai/gpt-4o
weak:
mannequin: openai/gpt-4o-mini
picker: efficient_first
confidence_threshold: 0.5
signal_recent_window: 3
fallback_target_on_evict: weakThe thought is:
Agent flip
|
v
Latest progress / failure alerts
|
v
Is further functionality helpful now?
/
/
weak robustRight here, the router seems for alerts related to issues similar to errors, repeated unproductive conduct, exploration, and up to date productive adjustments. The objective is to order the stronger mannequin for turns the place further functionality seems helpful.
Step 8: Escalating Solely After the Weak Mannequin Struggles
One other technique is to keep away from predicting problem up entrance.
Let a budget mannequin strive first, then escalate when proof of sustained hassle seems. The movement turns into:
Request
|
v
Weak mannequin
|
v
Choose consequence
/
okay struggling
| |
v v
keep robust mannequinSwitchyard calls this escalation routing. You’ll be able to create a configuration like this:
routes:
agent:
kind: escalation_router
robust:
mannequin: openai/gpt-4o
weak:
mannequin: openai/gpt-4o-mini
choose:
mannequin: openai/gpt-4o-mini
confirmations: 2
recent_turn_window: 28
window_message_chars: 500
fallback_target_on_evict: weakThat is conceptually totally different from up-front deterministic classification. Deterministic/functionality routing asks:
How troublesome does this request seem?Escalation routing asks:
Is the weak mannequin really moving into hassle?This makes escalation helpful for long-running agent periods the place activity problem can change over time.
Step 9: Measuring Whether or not Routing Is Truly Serving to
A router is just helpful if it improves the quality-cost trade-off. Switchyard exposes Prometheus metrics and statistics round requests, errors, latency, tokens, and routing conduct. The mission additionally helps structured request telemetry and optionally available routing logs.
You may get server metrics with:
curl -s http://localhost:4000/metrics | headand mixture JSON statistics:
curl -s http://localhost:4000/v1/stats | python3 -m json.instrumentFor experiments, examine at the least three runs:
| Configuration | Objective |
|---|---|
| At all times robust | High quality ceiling and price baseline |
| At all times weak | Low cost baseline |
| Switchyard router | Check whether or not routing captures most strong-model high quality at decrease price |
The extra helpful query will not be whether or not the router was 85% correct, however how a lot of the robust mannequin’s high quality did routing protect, and the way a lot price and latency did it cut back? For instance:
Sturdy-only:
$20
92% activity success
Weak-only:
$5
71% activity success
Router:
$9
89% activity successThis tells you whether or not routing is economically helpful.
Ultimate Ideas
As LLM programs grow to be extra agentic, the query is shifting from:
Which mannequin ought to I exploit?
to:
Which mannequin ought to I exploit for this request, at this level within the workflow, below this price price range?
Switchyard is NVIDIA’s try to show that call into reusable infrastructure.
For a primary experiment, do not bounce immediately into stage routing or advanced agent escalation.
Begin with two fashions.
Measure them independently.
Use weighted random routing to confirm your setup.
Then introduce capability-based routing and measure whether or not it preserves a lot of the robust mannequin’s high quality whereas shifting a significant share of requests to the cheaper tier.
This experiment provides you one thing way more helpful than one other LLM benchmark:
a quality-versus-cost curve to your precise workload.And that’s finally what clever mannequin routing is making an attempt to optimize.
Kanwal Mehreen is a machine studying engineer and a technical author with a profound ardour for knowledge science and the intersection of AI with drugs. She co-authored the book “Maximizing Productiveness with ChatGPT”. As a Google Technology Scholar 2022 for APAC, she champions range and tutorial excellence. She’s additionally acknowledged as a Teradata Variety in Tech Scholar, Mitacs Globalink Analysis Scholar, and Harvard WeCode Scholar. Kanwal is an ardent advocate for change, having based FEMCodes to empower girls in STEM fields.
