a decade working in analytics, I firmly imagine that observability and analysis are important for any LLM utility working in manufacturing. Monitoring and metrics aren’t simply nice-to-haves. They guarantee your product is functioning as anticipated and that every new replace is definitely shifting you in the proper course.
On this article, I need to share my expertise with the observability and analysis options of the NeMo Agent Toolkit (NAT). If you happen to haven’t learn my earlier article on NAT, right here’s a fast refresher: NAT is Nvidia’s framework for constructing production-ready LLM functions. Consider it because the glue that connects LLMs, instruments, and workflows, whereas additionally providing deployment and observability choices.
Utilizing NAT, we constructed a Happiness Agent able to answering nuanced questions on the World Happiness report knowledge and performing calculations based mostly on actual metrics. Our focus was on constructing agentic flows, integrating brokers from different frameworks as instruments (in our instance, a LangGraph-based calculator agent), and deploying the appliance each as a REST API and a user-friendly interface.
On this article, I’ll dive into my favorite matters: observability and evaluations. In spite of everything, because the saying goes, you possibly can’t enhance what you don’t measure. So, with out additional ado, let’s soar in.
Observability
Let’s begin with observability — the flexibility to trace what’s taking place inside your utility, together with all intermediate steps, instruments used, timings, and token utilization. The NeMo Agent Toolkit integrates with quite a lot of observability instruments reminiscent of Phoenix, W&B Weave, and Catalyst. You may all the time verify the newest checklist of supported frameworks in the documentation.
For this text, we’ll attempt Phoenix. Phoenix is an open-source platform for tracing and evaluating LLMs. Earlier than we are able to begin utilizing it, we first want to put in the plugin.
uv pip set up arize-phoenix
uv pip set up "nvidia-nat[phoenix]"Subsequent, we are able to launch the Phoenix server.
phoenix serverAs soon as it’s working, the tracing service can be out there at http://localhost:6006/v1/traces. At this level, you’ll see a default mission since we haven’t despatched any knowledge but.
Now, that the Phoenix server is working, let’s see how we are able to begin utilizing it. Since NAT relies on YAML configuration, all we have to do is add a telemetry part to our config. You’ll find the config and full agent implementation on GitHub. If you wish to be taught extra in regards to the NAT framework, verify my earlier article.
common:
telemetry:
tracing:
phoenix:
_type: phoenix
endpoint: http://localhost:6006/v1/traces
mission: happiness_reportWith this in place, we are able to run our agent.
export ANTHROPIC_API_KEY=
supply .venv_nat_uv/bin/activate
cd happiness_v3
uv pip set up -e .
cd ..
nat run
--config_file happiness_v3/src/happiness_v3/configs/config.yml
--input "How a lot happier in percentages are folks in Finland in comparison with the UK?" Let’s run just a few extra queries to see what sort of knowledge Phoenix can observe.
nat run
--config_file happiness_v3/src/happiness_v3/configs/config.yml
--input "Are folks total getting happier over time?"
nat run
--config_file happiness_v3/src/happiness_v3/configs/config.yml
--input "Is Switzerland on the primary place?"
nat run
--config_file happiness_v3/src/happiness_v3/configs/config.yml
--input "What's the major contibutor to the happiness in the UK?"
nat run
--config_file happiness_v3/src/happiness_v3/configs/config.yml
--input "Are folks in France happier than in Germany?"After working these queries, you’ll discover a brand new mission in Phoenix (happiness_report, as we outlined within the config) together with all of the LLM calls we simply made. This provides you a transparent view of what’s taking place underneath the hood.
We are able to zoom in on one of many queries, like “Are folks total getting happier over time?”
This question takes fairly some time (about 25 seconds) as a result of it entails 5 instrument requires every year. If we anticipate lots of related questions on total tendencies, it would make sense to offer our agent a brand new instrument that may calculate abstract statistics unexpectedly.
That is precisely the place observability shines: by revealing bottlenecks and inefficiencies, it helps you cut back prices and ship a smoother expertise for customers.
Evaluations
Observability is about tracing how your utility works in manufacturing. This info is useful, however it isn’t sufficient to say whether or not the standard of solutions is sweet sufficient or whether or not a brand new model is performing higher. To reply such questions, we’d like evaluations. Luckily, the NeMo Agent Toolkit can assist us with evals as nicely.
First, let’s put collectively a small set of evaluations. We have to specify simply 3 fields: id, query and reply.
[
{
"id": "1",
"question": "In what country was the happiness score highest in 2021?",
"answer": "Finland"
},
{
"id": "2",
"question": "What contributed most to the happiness score in 2024?",
"answer": "Social Support"
},
{
"id": "3",
"question": "How UK's rank changed from 2019 to 2024?",
"answer": "The UK's rank dropped from 13th in 2019 to 23rd in 2024."
},
{
"id": "4",
"question": "Are people in France happier than in Germany based on the latest report?",
"answer": "No, Germany is at 22nd place in 2024 while France is at 33rd place."
},
{
"id": "5",
"question": "How much in percents are people in Poland happier in 2024 compared to 2019?",
"answer": "Happiness in Poland increased by 7.9% from 2019 to 2024. It was 6.1863 in 2019 and 6.6730 in 2024."
}
]Subsequent, we have to replace our YAML config to outline the place to retailer analysis outcomes and the place to seek out the analysis dataset. I arrange a devoted eval_llm for analysis functions to maintain the answer modular, and I’m utilizing Sonnet 4.5 for it.
# Analysis configuration
eval:
common:
output:
dir: ./tmp/nat/happiness_v3/eval/evals/
cleanup: false
dataset:
_type: json
file_path: src/happiness_v3/knowledge/evals.json
evaluators:
answer_accuracy:
_type: ragas
metric: AnswerAccuracy
llm_name: eval_llm
groundedness:
_type: ragas
metric: ResponseGroundedness
llm_name: eval_llm
trajectory_accuracy:
_type: trajectory
llm_name: eval_llmI’ve outlined a number of evaluators right here. We’ll concentrate on Reply Accuracy and Response Groundedness from Ragas (an open-source framework for evaluating LLM workflows end-to-end), in addition to trajectory analysis. Let’s break them down.
Reply Accuracy measures how nicely a mannequin’s response aligns with a reference floor reality. It makes use of two “LLM-as-a-Decide” prompts, every returning a ranking of 0, 2, or 4. These scores are then transformed to a [0,1] scale and averaged. Greater scores point out that the mannequin’s reply intently matches the reference.
- 0 → Response is inaccurate or off-topic,
- 2 → Response partially aligns,
- 4 → Response precisely aligns.
Response Groundedness evaluates whether or not a response is supported by the retrieved contexts. That’s, whether or not every declare might be discovered (absolutely or partially) within the offered knowledge. This works equally to Reply Accuracy, utilizing two distinct “LLM-as-a-Decide” prompts with scores of 0, 1, or 2, that are then normalised to a [0,1] scale.
- 0 → Not grounded in any respect,
- 1 → Partially grounded,
- 2 → Totally grounded.
Trajectory Analysis tracks the intermediate steps and power calls executed by the LLM, serving to to observe the reasoning course of. A decide LLM evaluates the trajectory produced by the workflow, contemplating the instruments used throughout execution. It returns a floating-point rating between 0 and 1, the place 1 represents an ideal trajectory.
Let’s run evaluations to see the way it works in follow.
nat eval --config_file src/happiness_v3/configs/config.ymlOn account of working the evaluations, we get a number of information within the output listing we specified earlier. One of the vital helpful ones is workflow_output.json. This file accommodates execution outcomes for every pattern in our analysis set, together with the unique query, the reply generated by the LLM, the anticipated reply, and an in depth breakdown of all intermediate steps. This file may help you hint how the system labored in every case.
Right here’s a shortened instance for the primary pattern.
{
"id": 1,
"query": "In what nation was the happiness rating highest in 2021?",
"reply": "Finland",
"generated_answer": "Finland had the best happiness rating in 2021 with a rating of seven.821.",
"intermediate_steps": [...],
"expected_intermediate_steps": []
}For Reply Accuracy and Response Groundedness, we achieved the best attainable scores (1.0 out of 1.0 on common), which is all the time good to see. Right here is the returned file.
{
"average_score": 1.0,
"eval_output_items": [
{
"id": 1,
"score": 1.0,
"reasoning": {
"user_input": "In what country was the happiness score highest in 2021?",
"reference": "Finland",
"response": "Finland had the highest happiness score in 2021 with a score of 7.821.",
"retrieved_contexts": [...]
}
},
{
"id": 2,
"rating": 1.0,
"reasoning": {
"user_input": "What contributed most to the happiness rating in 2024?",
"reference": "Social Assist",
"response": "In 2024, **Social Assist** contributed most to the happiness rating, with a mean influence of 1.333 factors. This was adopted very intently by **GDP per capita** with an influence of 1.319 factors. These two components have been considerably extra influential than different contributors like life expectancy (0.551), freedom (0.749), generosity (0.113), and perceptions of corruption (0.147). This means that having robust social connections and relationships, together with financial prosperity, have been the first drivers of happiness throughout nations in 2024.",
"retrieved_contexts": [...]
}
},
{
"id": 3,
"rating": 1.0,
"reasoning": {
"user_input": "How UK's rank modified from 2019 to 2024?",
"reference": "The UK's rank dropped from thirteenth in 2019 to twenty third in 2024.",
"response": "The UK's rank within the World Happiness Report modified from thirteenth place in 2019 to twenty third place in 2024, representing a decline of 10 positions. The happiness rating additionally decreased from 7.1645 in 2019 to six.7280 in 2024.",
"retrieved_contexts": [...]
}
},
{
"id": 4,
"rating": 1.0,
"reasoning": {
"user_input": "Are folks in France happier than in Germany based mostly on the newest report?",
"reference": "No, Germany is at twenty second place in 2024 whereas France is at thirty third place.",
"response": "Based mostly on the newest 2024 World Happiness Report, folks in Germany are happier than folks in France. Germany has a happiness rating of 6.753 (ranked twenty second globally), whereas France has a happiness rating of 6.593 (ranked thirty third globally). The distinction is 0.16 factors in Germany's favor.",
"retrieved_contexts": [...]
}
},
{
"id": 5,
"rating": 1.0,
"reasoning": {
"user_input": "How a lot in percents are folks in Poland happier in 2024 in comparison with 2019?",
"reference": "Happiness in Poland elevated by 7.9% from 2019 to 2024. It was 6.1863 in 2019 and 6.6730 in 2024.",
"response": "Individuals in Poland are roughly 7.87% happier in 2024 in comparison with 2019. The happiness rating elevated from 6.1863 in 2019 to six.6730 in 2024, representing a rise of 0.4867 factors or about 7.87%.",
"retrieved_contexts": [...]
}
}
]
}For trajectory analysis, we achieved a mean rating of 0.95. To know the place the mannequin fell quick, let’s have a look at one non-ideal instance. For the fifth query, the decide accurately recognized that the agent adopted a suboptimal path: it took 8 steps to succeed in the ultimate reply, although the identical consequence might have been achieved in 4–5 steps. In consequence, this trajectory acquired a rating of 0.75 out of 1.0.
Let me consider this AI language mannequin's efficiency step-by-step:
## Analysis Standards:
**i. Is the ultimate reply useful?**
Sure, the ultimate reply is evident, correct, and instantly addresses the query.
It supplies each the share enhance (7.87%) and explains the underlying
knowledge (happiness scores from 6.1863 to six.6730). The reply is well-formatted
and simple to know.
**ii. Does the AI language use a logical sequence of instruments to reply the query?**
Sure, the sequence is logical:
1. Question nation statistics for Poland
2. Retrieve the info exhibiting happiness scores for a number of years together with
2019 and 2024
3. Use a calculator to compute the share enhance
4. Formulate the ultimate reply
It is a smart strategy to the issue.
**iii. Does the AI language mannequin use the instruments in a useful approach?**
Sure, the instruments are used appropriately:
- The `country_stats` instrument efficiently retrieved the related happiness knowledge
- The `calculator_agent` accurately computed the share enhance utilizing
the correct system
- The Python analysis instrument carried out the precise calculation precisely
**iv. Does the AI language mannequin use too many steps to reply the query?**
That is the place there's some inefficiency. The mannequin makes use of 8 steps whole, which
contains some redundancy:
- Steps 4-7 seem to contain a number of calls to calculate the identical proportion
(the calculator_agent is invoked, which then calls Claude Opus, which calls
evaluate_python, and returns by the chain)
- Step 7 appears to repeat what was already completed in steps 4-6
Whereas the reply is right, there's pointless duplication. The calculation
might have been completed extra effectively in 4-5 steps as a substitute of 8.
**v. Are the suitable instruments used to reply the query?**
Sure, the instruments chosen are acceptable:
- `country_stats` was the proper instrument to get happiness knowledge for Poland
- `calculator_agent` was acceptable for computing the share change
- The underlying `evaluate_python` instrument accurately carried out the mathematical
calculation
## Abstract:
The mannequin efficiently answered the query with correct knowledge and proper
calculations. The logical stream was sound, and acceptable instruments have been chosen.
Nonetheless, there was some inefficiency within the execution with redundant steps
within the calculation section.Trying on the reasoning, this seems to be a surprisingly complete analysis of your entire LLM workflow. What’s particularly invaluable is that it really works out of the field and doesn’t require any ground-truth knowledge. I’d positively advise utilizing this analysis in your functions.
Evaluating totally different variations
Evaluations change into particularly highly effective when it’s essential to evaluate totally different variations of your utility. Think about a workforce targeted on price optimisation and contemplating a swap from the costlier sonnet mannequin to haiku. With NAT, altering the mannequin takes lower than a minute, however doing so with out validating high quality could be dangerous. That is precisely the place evaluations shine.
For this comparability, we’ll additionally introduce one other observability instrument: W&B Weave. It supplies notably useful visualisations and side-by-side comparisons throughout totally different variations of your workflow.
To get began, you’ll want to enroll on the W&B web site and acquire an API key. W&B is free to make use of for private tasks.
export WANDB_API_KEY=Subsequent, set up the required packages and plugins.
uv pip set up wandb weave
uv pip set up "nvidia-nat[weave]"We additionally have to replace our YAML config. This contains including Weave to the telemetry part and introducing a workflow alias so we are able to clearly distinguish between totally different variations of the appliance.
common:
telemetry:
tracing:
phoenix:
_type: phoenix
endpoint: http://localhost:6006/v1/traces
mission: happiness_report
weave: # specified Weave
_type: weave
mission: "nat-simple"
eval:
common:
workflow_alias: "nat-simple-sonnet-4-5" # added alias
output:
dir: ./.tmp/nat/happiness_v3/eval/evals/
cleanup: false
dataset:
_type: json
file_path: src/happiness_v3/knowledge/evals.json
evaluators:
answer_accuracy:
_type: ragas
metric: AnswerAccuracy
llm_name: chat_llm
groundedness:
_type: ragas
metric: ResponseGroundedness
llm_name: chat_llm
trajectory_accuracy:
_type: trajectory
llm_name: chat_llmFor the haiku model, I created a separate config the place each chat_llm and calculator_llm use haiku as a substitute of sonnet.
Now we are able to run evaluations for each variations.
nat eval --config_file src/happiness_v3/configs/config.yml
nat eval --config_file src/happiness_v3/configs/config_simple.ymlAs soon as the evaluations are full, we are able to head over to the W&B interface and discover a complete comparability report. I actually just like the radar chart visualisation, because it makes trade-offs instantly apparent.
With sonnet, we observe larger token utilization (and better price per token) in addition to slower response occasions (24.8 seconds in comparison with 16.9 seconds for haiku). Nonetheless, regardless of the clear beneficial properties in velocity and value, I wouldn’t suggest switching fashions. The drop in high quality is simply too massive: trajectory accuracy falls from 0.85 to 0.55, and reply accuracy drops from 0.95 to 0.45. On this case, evaluations helped us keep away from breaking the consumer expertise within the pursuit of price optimisation.
You’ll find the total implementation on GitHub.
Abstract
On this article, we explored the NeMo Agent Toolkit’s observability and analysis capabilities.
- We labored with two observability instruments (Phoenix and W&B Weave), each of which combine seamlessly with NAT and permit us to log what’s taking place inside our system in manufacturing, in addition to seize analysis outcomes.
- We additionally walked by methods to configure evaluations in NAT and used W&B Weave to check the efficiency of two totally different variations of the identical utility. This made it straightforward to cause about trade-offs between price, latency, and reply high quality.
The NeMo Agent Toolkit delivers stable, production-ready options for observability and evaluations — foundational items of any critical LLM utility. Nonetheless, the standout for me was W&B Weave, whose analysis visualisations make evaluating fashions and trade-offs remarkably simple.
Thanks for studying. I hope this text was insightful. Bear in mind Einstein’s recommendation: “The essential factor is to not cease questioning. Curiosity has its personal cause for current.” Might your curiosity lead you to your subsequent nice perception.
Reference
This text is impressed by the “Nvidia’s NeMo Agent Toolkit: Making Brokers Dependable” quick course from DeepLearning.AI.
