Image a help inbox for a financial institution. Each message that is available in must be sorted right into a class, a misplaced card, a refund request, a mistaken cost, and despatched to the appropriate staff.
A financial institution help analyst routes incoming buyer messages about misplaced playing cards, refunds, and incorrect costs to separate help groups.
Now image that sorting job handed to an AI mannequin as an alternative of an individual. The mannequin reads the message and arms again a brief observe in a set format, somewhat like a type with the identical packing containers each time, so the remainder of this system can learn it routinely and resolve what to do subsequent. No human has to interpret free textual content.
That fastened format is often a small block of laptop readable textual content known as JSON, brief for JavaScript Object Notation. Consider it as labeled packing containers on a type. One field known as intent holds the class. One other known as precedence holds how pressing it’s.
This system studying the mannequin’s reply doesn’t perceive English. It appears for these actual packing containers, spelled precisely the way in which it expects, each single time. If a field goes lacking, or a label is spelled barely in another way than this system expects, this system has no approach to discover by itself. It simply quietly stops working for that one message, whereas the whole lot on the floor nonetheless appears high-quality.
AI firms launch new mannequin variations always, and deciding which LLM to make use of for a given job often comes down to 1 quantity from the AI testing groups already run, how typically it picks the appropriate class. That single rating can go up whereas one thing else, the precise form of the reply, quietly will get worse, and a rising common has no approach to warn you.
With plain prompting, you merely write:
“Return your reply as JSON.”
The mannequin should still return:
Certain, right here is the JSON:{"intent": "request_refund"}
That further sentence can break code that expects JSON solely. The mannequin might also pass over a area or use a price this system doesn’t count on.
Structured Outputs is a stricter characteristic that makes the mannequin observe a predefined JSON construction, resembling requiring intent, precedence, and needs_human. It could possibly stop many formatting issues, however the software nonetheless must test whether or not the values and resolution are appropriate.
I ran an actual LLM regression take a look at on one small, actual software, as an alternative of trusting the accuracy quantity alone. I constructed a help triage assistant, gave it 47 actual buyer messages from a public banking dataset, and ran the very same messages via three actual variations of an OpenAI mannequin, an older one, the one I’m treating because the mannequin at the moment in manufacturing, and a more recent candidate being thought of as a alternative.
I anticipated the newer mannequin to decide on the proper class extra typically, however I additionally apprehensive that it would often ignore the precise format this system requires. As a substitute, the newer mannequin adopted the format each time. The manufacturing mannequin made the formatting mistake. It did so quietly, on each refund query within the pattern, spelling one label Request_refund with a capital R as an alternative of the lowercase request_refund the remainder of the system expects.
A human studying the reply would name it appropriate. A program matching labels precisely would silently drop each a type of tickets.
That’s the drawback this text is about: a mannequin can sound appropriate to an individual and nonetheless be mistaken for the software program that makes use of its reply.
A schematic diagram displaying a bar chart of general accuracy rising from an outdated mannequin to a brand new mannequin on the left, subsequent to a paired comparability on the appropriate the place the outdated mannequin solutions one take a look at case accurately and the brand new mannequin solutions the identical case mistaken, labeled as a detrimental flip
What this mission builds, and why it makes use of Weave
Earlier than writing any code, it helps to have one clear image of what will get constructed and the way its items match collectively.
This mission does 5 issues:
Weave information what occurs when the applying runs: the query, the directions, the mannequin, the response, and the timing.
The directions given to the mannequin are saved with a model quantity, so older and newer directions may be in contrast.
The actual buyer questions are saved as a take a look at dataset, so each mannequin solutions the identical examples.
A strict checker exams every response for actual necessities, resembling legitimate JSON, required fields, allowed labels, and the proper class.
A second AI mannequin reads every response and offers it a high quality rating, extra like a human reviewer would.
The strict checker appears for actual machine necessities. The second AI choose evaluates the reply extra like a reader. Utilizing each helps reveal issues that both checker may miss.
Every of these concepts will get defined correctly because it comes up. For now, begin with Weave itself, since the whole lot else on this article is recorded inside it.
Weave is a instrument from Weights & Biases (W&B) for watching what an AI software really does whereas it runs. Add one line, @weave.op(), above any Python perform, and each single name to that perform will get saved routinely, the precise textual content that went in, the precise textual content that got here again, and the way lengthy it took.
Weave calls certainly one of these saved information a hint, and it shops each hint in a mission you may open and browse in an online web page, the identical means a photograph app retains a timeline of each photograph you are taking.
A hint is just not solely helpful for debugging a damaged run after the very fact. As soon as an software has been answering actual questions for some time, its saved traces are additionally a prepared made supply of actual examples, which issues later on this article, for the reason that similar 47 actual questions that hint the applying additionally develop into the dataset it will get examined in opposition to.
The appliance itself is intentionally small, one perform, triage_message(textual content, mannequin, prompt_ref), that reads one actual buyer message and asks a mannequin to reply with a JSON object formed like this:
4 packing containers, each time. intent names the class. precedence is low, medium, or excessive. needs_human is true or false, and it decides whether or not the message will get escalated to an individual as an alternative of dealt with routinely. reply is the brief message the shopper really sees.
Solely two issues change throughout the remainder of this text: which mannequin solutions, and which model of the directions or the grading guidelines is energetic. The appliance logic itself by no means modifications, which is what makes the comparisons later on this article honest.
One alternative about how the mannequin will get requested issues sufficient to clarify now. The request to OpenAI makes use of plain prompted JSON, that means the mannequin is solely instructed in phrases to answer on this form. It doesn’t use OpenAI’s stricter Structured Outputs characteristic, the one already talked about above, which may power a mannequin’s reply into a set form by development.
That’s deliberate, not an oversight. Utilizing the strict characteristic right here would have hidden among the very failures this text is constructed to search for, an invalid response, further textual content wrapped across the JSON, or a mislabeled area. Later within the article, after you have seen what really broke, there may be an sincere take a look at precisely which of these failures the strict characteristic would and wouldn’t have caught.
The actual buyer messages come from BANKING77, a public dataset from a 2020 analysis paper by Iñigo Casanueva and coauthors at PolyAI (CC BY 4.0 license). It incorporates 13,083 actual banking customer support questions, every labeled by hand with certainly one of 77 high-quality grained classes, a card that by no means arrived, a refund that by no means confirmed up, a fee the shopper doesn’t acknowledge, and so forth.
High-quality grained means lots of these 77 classes sound shut sufficient to genuinely confuse a mannequin, which is precisely the property that makes this dataset helpful right here. A mannequin that may solely inform the straightforward circumstances aside is just not being examined very arduous.
Setup
This mission was written and run with Python 3.11 and Weave:
python3.11 -m venv .venvsupply .venv/bin/activate # on Home windows: .venvScriptsactivatepip set up weave openai python-dotenv requests wandb
You want an OpenAI software programming interface (API) key, and a free Weights & Biases account for Weave. Run wandb login as soon as within the activated atmosphere, or set a WANDB_API_KEY atmosphere variable. Save an OPENAI_API_KEY the identical means, both as an atmosphere variable or in a .env file subsequent to the script beneath.
One small model observe. This mission was run in opposition to weave==0.52.40. Weave printed a discover on each run saying that actual model had been recalled over a technical challenge and recommending an improve. The recall didn’t change something within the outcomes right here, however set up the present launch as an alternative of pinning an outdated one, pip set up -U weave, except you’ve a selected cause to not.
The whole script
All the things on this article, the traced software, the 2 variations of its directions, the dataset, the strict rule primarily based grader, the AI grader, and the mannequin comparability, lives in a single script. Put it aside as banking77_regression.py:
"""BANKING77 structured output regression testing.Traces a small banking help triage software with Weave, variations itssystem immediate, turns actual BANKING77 questions right into a Weave Dataset, buildsa graded LLM choose subsequent to a deterministic contract scorer (legitimate JSON,required fields, allowed values, appropriate intent, appropriate escalation), runsa Weave Analysis throughout three OpenAI fashions, validates the choose in opposition tothe deterministic scorer, and checks whether or not a candidate mannequin that wins onintent accuracy nonetheless respects the output contract the applying relies uponon.Intentionally does NOT use response_format json_object or strict StructuredOutputs. Plain prompted JSON is the purpose, it's what makes invalid JSON,prose wrapped round JSON, and area worth drift reachable failure modes.Run modes, so as: fetch fetch and save the true BANKING77 take a look at questions used smoke smoke take a look at immediate v1 in opposition to the candidate mannequin dataset publish immediate v1/v2 and the Weave Dataset consider run the Weave Analysis for all three fashions (choose v1) judge_check examine the choose scores to the deterministic scores refine_judge publish an improved choose immediate and rerun the analysis judge_check_v2 examine the refined choose scores to the deterministic scores contract examine manufacturing vs candidate on accuracy vs contract sorted_diffs kind two fashions by absolute choose rating distinction"""import argparseimport asyncioimport csvimport ioimport jsonimport reimport statisticsimport timefrom pathlib import Pathimport requestsimport weavefrom dotenv import load_dotenvfrom openai import OpenAI, RateLimitErrorROOT = Path(__file__).resolve().mum or dadOUTPUTS_DIR = ROOT / "outputs"OUTPUTS_DIR.mkdir(exist_ok=True)load_dotenv()PROJECT = "wb-authors/model-regression-tests-weave" # exchange with your individual W&B entity/missionPROMPT_NAME = "banking77-solver-prompt"JUDGE_PROMPT_NAME = "banking77-judge-prompt"DATASET_NAME = "banking77-eval-set"shopper = OpenAI()def call_with_retry(fn, *args, max_attempts=5, **kwargs): """Weave's Analysis runs each row concurrently, which may burst previous a shared group tokens per minute restrict on the choose mannequin even although every particular person name is small. Retries with backoff as an alternative of letting a fee restrict error drop that row's rating.""" for try in vary(max_attempts): strive: return fn(*args, **kwargs) besides RateLimitError: if try == max_attempts - 1: increase time.sleep(2 ** try)BANKING77_TEST_CSV_URL = ( "https://uncooked.githubusercontent.com/PolyAI-LDN/task-specific-datasets/" "grasp/banking_data/take a look at.csv")# Escalation coverage, written earlier than taking a look at any mannequin output. Intents# that contain fraud, loss, blocked entry, unrecognized cash motion, or# compliance checks require a human. All the things else is routine.ESCALATE_INTENTS = { "lost_or_stolen_card", "lost_or_stolen_phone", "compromised_card", "card_payment_not_recognised", "direct_debit_payment_not_recognised", "cash_withdrawal_not_recognised", "unable_to_verify_identity", "verify_source_of_funds", "pin_blocked", "transaction_charged_twice",}# The 20 intents sampled for this text's dataset, chosen as 5# confusable clusters so fashions have actual room to disagree, not an# arbitrary slice of the 77 classes.SELECTED_INTENTS = { "card_arrival": 3, "card_not_working": 3, "lost_or_stolen_card": 3, "declined_transfer": 2, "failed_transfer": 2, "transfer_not_received_by_recipient": 2, "pending_transfer": 2, "request_refund": 3, "Refund_not_showing_up": 3, "card_payment_not_recognised": 3, "direct_debit_payment_not_recognised": 2, "cash_withdrawal_not_recognised": 2, "verify_my_identity": 2, "why_verify_identity": 2, "unable_to_verify_identity": 3, "verify_source_of_funds": 2, "compromised_card": 2, "lost_or_stolen_phone": 2, "pin_blocked": 2, "transaction_charged_twice": 2,}PROMPT_V1 = ( "You're a banking help triage assistant. A buyer will ship you " "one message. Learn it and reply with a JSON object with precisely these " "4 fields, intent, precedence, needs_human, reply. intent should be one " "of the allowed banking intent labels given beneath, copied precisely. " "precedence should be certainly one of low, medium, excessive. needs_human should be true " "or false. reply is a brief, pure reply to the shopper.nn" "Allowed intent labels: {intents}")# v2 fixes an actual hole discovered within the v1 smoke take a look at, outputs/# banking77_smoke_test_v1.json. v1 by no means instructed the mannequin the escalation# coverage, so it guessed at needs_human/precedence utilizing its personal judgment and# received escalation_correct on solely 2 of 6 smoke examples despite the fact that intent# and JSON format had been each already good. v2 states the coverage# explicitly. Re-run on the identical 6 examples afterward, 6 of 6 appropriate.PROMPT_V2 = ( "You're a banking help triage assistant. A buyer will ship you " "one message. Learn it and reply with a JSON object with precisely these " "4 fields, intent, precedence, needs_human, reply. intent should be one " "of the allowed banking intent labels given beneath, copied precisely. " "precedence should be certainly one of low, medium, excessive. needs_human should be true " "or false. reply is a brief, pure reply to the shopper.nn" "Escalation coverage, apply it precisely. If the intent is certainly one of these, " "set needs_human to true and precedence to excessive, no matter how the " "message is worded, card_payment_not_recognised, " "cash_withdrawal_not_recognised, compromised_card, " "direct_debit_payment_not_recognised, lost_or_stolen_card, " "lost_or_stolen_phone, pin_blocked, transaction_charged_twice, " "unable_to_verify_identity, verify_source_of_funds. For each different " "intent, set needs_human to false and precedence to low or medium, " "by no means excessive, even when the shopper sounds annoyed or the state of affairs " "sounds pressing.nn" "Allowed intent labels: {intents}")JUDGE_PROMPT_V1 = """You might be grading one mannequin's response to a banking buyer help message.You will note the shopper's message, the proper intent label, the properescalation coverage for that intent (whether or not it ought to escalate to a humanwith excessive precedence, or not escalate and never use excessive precedence), and themannequin's uncooked response.Apply this tough ceiling earlier than anything. If the response is just not legitimateJSON, is lacking any of the 4 required fields (intent, precedence,needs_human, reply), or makes use of an intent label that isn't an actual bankingintent, the rating should be 0 to 2, no matter how good the reply textual contentsounds. A damaged output can't be utilized by the downstream code that expectsthis actual form.If the output is legitimate and full, apply this subsequent ceiling. If theintent is mistaken, or the escalation resolution (needs_human and precedence)doesn't match the acknowledged coverage, the rating should be 3 to five, no matterreply high quality.Provided that the output is legitimate, full, has the proper intent, and followsthe escalation coverage accurately, rating the reply textual content itself from 6 to 10:- 6 to eight: the reply is generic, or solely loosely addresses the shopper's particular message- 9 to 10: the reply is evident, particular to the shopper's precise message, and appropriately tonedReturn your grade as strict JSON with this actual form and nothing else:{"rating": , "reasoning": ""}"""# v2 fixes an actual miscalibration discovered throughout choose validation, see# banking77_judge_check_v1.json. The choose learn the coverage phrase "not# escalate, and never use excessive precedence" as if it meant one particular# precedence worth was required, and penalized responses that used medium# as an alternative of low despite the fact that each are legitimate non escalating priorities.# This produced 4 to six false disagreements per mannequin. v2 states the rule# precisely (low and medium are each appropriate) and disagreements dropped to# 0 for 2 of the three fashions. The disagreements that remained for the# manufacturing mannequin afterward had been a distinct, actual, non hypothetical# challenge described within the article, not a choose bug.JUDGE_PROMPT_V2 = """You might be grading one mannequin's response to a banking buyer help message.You will note the shopper's message, the proper intent label, the properescalation coverage for that intent (whether or not it ought to escalate to a humanwith excessive precedence, or not escalate and never use excessive precedence), and themannequin's uncooked response.Apply this tough ceiling earlier than anything. If the response is just not legitimateJSON, is lacking any of the 4 required fields (intent, precedence,needs_human, reply), or makes use of an intent label that isn't an actual bankingintent, the rating should be 0 to 2, no matter how good the reply textual contentsounds. A damaged output can't be utilized by the downstream code that expectsthis actual form.If the output is legitimate and full, apply this subsequent ceiling. If theintent is mistaken, the rating should be 3 to five. Individually, test theescalation resolution in opposition to the acknowledged coverage precisely as follows. If thecoverage says escalate, needs_human should be true and precedence should beprecisely excessive, anything is a mismatch. If the coverage says don'tescalate, needs_human should be false and precedence should not be excessive, howeverlow and medium are BOTH totally appropriate values for a non escalating case,there is no such thing as a single required worth between them, and selecting mediumas an alternative of low is just not a mismatch and should not be scored as one. Solely amistaken needs_human worth or a precedence of excessive on a non escalating casecounts as an escalation mismatch. If the intent is correct however theescalation mismatches by this actual definition, the rating should be 3 to five.Provided that the output is legitimate, full, has the proper intent, and followsthe escalation coverage accurately by the precise definition above, rating thereply textual content itself from 6 to 10:- 6 to eight: the reply is generic, or solely loosely addresses the shopper's particular message- 9 to 10: the reply is evident, particular to the shopper's precise message, and appropriately tonedReturn your grade as strict JSON with this actual form and nothing else:{"rating": , "reasoning": ""}"""MODEL_SLOTS = { "older": "gpt-4o-mini", "manufacturing": "gpt-4.1-mini", "candidate": "gpt-5-mini",}JUDGE_MODEL = "gpt-4.1"# ---------------------------------------------------------------------------# Knowledge fetch# ---------------------------------------------------------------------------def fetch_all_rows(): r = requests.get(BANKING77_TEST_CSV_URL, timeout=30) r.raise_for_status() reader = csv.DictReader(io.StringIO(r.textual content)) return record(reader)def fetch_examples(): rows = fetch_all_rows() all_categories = sorted(set(row["category"] for row in rows)) by_cat = {} for row in rows: by_cat.setdefault(row["category"], []).append(row["text"]) chosen = [] qid = 0 for cat, n in SELECTED_INTENTS.objects(): texts = by_cat.get(cat, []) for textual content in texts[:n]: qid += 1 chosen.append({ "question_id": f"b77-{qid:03d}", "textual content": textual content, "ground_truth_intent": cat, "ground_truth_escalate": cat in ESCALATE_INTENTS, }) out = {"all_categories": all_categories, "chosen": chosen} out_path = OUTPUTS_DIR / "banking77_examples_selected.json" with open(out_path, "w") as f: json.dump(out, f, indent=2) print(f"Saved {len(chosen)} examples throughout {len(SELECTED_INTENTS)} intents to {out_path}") return outdef load_examples(): return json.load(open(OUTPUTS_DIR / "banking77_examples_selected.json"))ALLOWED_PRIORITIES = {"low", "medium", "excessive"}# ---------------------------------------------------------------------------# The traced software# ---------------------------------------------------------------------------@weave.op()def triage_message(textual content: str, mannequin: str, prompt_ref: str) -> str: """The one small software traced all through this text.""" immediate = weave.ref(prompt_ref).get() response = call_with_retry( shopper.chat.completions.create, mannequin=mannequin, messages=[ {"role": "system", "content": prompt.content}, {"role": "user", "content": text}, ], max_completion_tokens=1000, ) return response.selections[0].message.content material or ""class Banking77Solver(weave.Mannequin): model_name: str prompt_ref: str @weave.op() def predict(self, textual content: str) -> str: return triage_message(textual content, self.model_name, self.prompt_ref)# ---------------------------------------------------------------------------# Deterministic contract scorer# ---------------------------------------------------------------------------def parse_json_response(uncooked: str): """Attempt three actual methods a plain textual content pipeline would strive, in order, and report which one labored.""" textual content = uncooked.strip() strive: return json.masses(textual content), "direct" besides json.JSONDecodeError: go fenced = re.search(r"```(?:json)?s*({.*?})s*```", textual content, re.DOTALL) if fenced: strive: return json.masses(fenced.group(1)), "fenced" besides json.JSONDecodeError: go first = textual content.discover("{") final = textual content.rfind("}") if first != -1 and final != -1 and final > first: strive: return json.masses(textual content[first:last + 1]), "extracted_braces" besides json.JSONDecodeError: go return None, "unparsable"@weave.op()def contract_scorer(question_id: str, ground_truth_intent: str, ground_truth_escalate: bool, all_categories: record, output: str) -> dict: parsed, parse_method = parse_json_response(output) consequence = { "question_id": question_id, "valid_json": parsed is just not None, "parse_method": parse_method, "has_required_fields": False, "intent_allowed": False, "intent_correct": False, "priority_allowed": False, "escalation_correct": False, "parsed": parsed, } if parsed is None or not isinstance(parsed, dict): return consequence required = {"intent", "precedence", "needs_human", "reply"} consequence["has_required_fields"] = required.issubset(parsed.keys()) intent = parsed.get("intent") if isinstance(intent, str): consequence["intent_allowed"] = intent in all_categories consequence["intent_correct"] = intent == ground_truth_intent precedence = parsed.get("precedence") if isinstance(precedence, str): consequence["priority_allowed"] = precedence.decrease() in ALLOWED_PRIORITIES needs_human = parsed.get("needs_human") if isinstance(needs_human, bool): consequence["escalation_correct"] = ( needs_human == ground_truth_escalate and (not ground_truth_escalate or (isinstance(precedence, str) and precedence.decrease() == "excessive")) and (ground_truth_escalate or not (isinstance(precedence, str) and precedence.decrease() == "excessive")) ) return consequence# ---------------------------------------------------------------------------# Graded LLM choose, rubric stuffed in after studying actual smoke take a look at output# ---------------------------------------------------------------------------class LLMJudge(weave.Scorer): judge_model: str judge_prompt_ref: str @weave.op() def rating(self, question_id: str, textual content: str, ground_truth_intent: str, ground_truth_escalate: bool, output: str) -> dict: judge_prompt = weave.ref(self.judge_prompt_ref).get() policy_note = ( "escalate to a human, that means needs_human should be true and " "precedence should be precisely excessive" if ground_truth_escalate else "not escalate, that means needs_human should be false and precedence " "should not be excessive, however both low or medium is appropriate, there " "isn't any single required worth between the 2" ) user_content = ( f"Buyer message:n{textual content}nn" f"Appropriate intent label: {ground_truth_intent}n" f"Appropriate escalation coverage for this intent: ought to {policy_note}.nn" f"Mannequin's uncooked response:n{output}" ) response = call_with_retry( shopper.chat.completions.create, mannequin=self.judge_model, messages=[ {"role": "system", "content": judge_prompt.content}, {"role": "user", "content": user_content}, ], max_completion_tokens=500, response_format={"sort": "json_object"}, ) uncooked = response.selections[0].message.content material or "{}" strive: parsed = json.masses(uncooked) rating = int(parsed.get("rating", 0)) reasoning = str(parsed.get("reasoning", "")) besides (json.JSONDecodeError, ValueError): rating = 0 reasoning = f"choose returned unparsable output: {uncooked[:200]}" return {"question_id": question_id, "judge_score": rating, "judge_reasoning": reasoning}# ---------------------------------------------------------------------------# Smoke take a look at, run earlier than the choose rubric is written# ---------------------------------------------------------------------------def smoke_test_step(): weave.init(PROJECT) knowledge = load_examples() all_categories = knowledge["all_categories"] v1_ref = weave.publish( weave.StringPrompt(PROMPT_V1.format(intents=", ".be part of(all_categories))), identify=PROMPT_NAME, ) print("immediate v1:", v1_ref.uri()) smoke = [] for ex in knowledge["selected"][:6]: output = triage_message(ex["text"], MODEL_SLOTS["candidate"], v1_ref.uri()) scored = contract_scorer(ex["question_id"], ex["ground_truth_intent"], ex["ground_truth_escalate"], all_categories, output) smoke.append({"question_id": ex["question_id"], "textual content": ex["text"], "raw_output": output, "scored": scored}) print(ex["question_id"], "valid_json:", scored["valid_json"], "escalation_correct:", scored["escalation_correct"]) with open(OUTPUTS_DIR / "banking77_smoke_test_v1.json", "w") as f: json.dump(smoke, f, indent=2) with open(OUTPUTS_DIR / "banking77_prompt_v1_ref.json", "w") as f: json.dump({"prompt_v1": v1_ref.uri()}, f, indent=2)# ---------------------------------------------------------------------------# Dataset + immediate publishing, v2 stuffed in after inspecting the smoke take a look at# ---------------------------------------------------------------------------def build_dataset_step(): weave.init(PROJECT) knowledge = load_examples() all_categories = knowledge["all_categories"] v2_ref = weave.publish( weave.StringPrompt(PROMPT_V2.format(intents=", ".be part of(all_categories))), identify=PROMPT_NAME, ) print("immediate v2:", v2_ref.uri()) rows = [ { "question_id": ex["question_id"], "textual content": ex["text"], "ground_truth_intent": ex["ground_truth_intent"], "ground_truth_escalate": ex["ground_truth_escalate"], "all_categories": all_categories, "prompt_version": "v2", } for ex in knowledge["selected"] ] dataset = weave.Dataset(identify=DATASET_NAME, rows=rows) dataset_ref = weave.publish(dataset, identify=DATASET_NAME) print("dataset:", dataset_ref.uri()) v1_ref = json.load(open(OUTPUTS_DIR / "banking77_prompt_v1_ref.json"))["prompt_v1"] with open(OUTPUTS_DIR / "banking77_refs.json", "w") as f: json.dump({"prompt_v1": v1_ref, "prompt_v2": v2_ref.uri(), "dataset": dataset_ref.uri()}, f, indent=2)# ---------------------------------------------------------------------------# Analysis# ---------------------------------------------------------------------------def get_refs(): return json.load(open(OUTPUTS_DIR / "banking77_refs.json"))def collect_run_rows(client_obj, evaluate_call, examples_by_text): client_obj.flush() descendants = record(client_obj.get_calls(filter={"trace_ids": [evaluate_call.trace_id]})) by_question_id = {} for name in descendants: trace_name = (name.abstract or {}).get("weave", {}).get("trace_name", "") if trace_name == "triage_message": q_text = name.inputs.get("textual content") ex = examples_by_text.get(q_text) if ex: by_question_id.setdefault(ex["question_id"], {})["output"] = name.output elif trace_name == "contract_scorer": out = name.output or {} qid = out.get("question_id") if qid: by_question_id.setdefault(qid, {})["contract"] = out elif trace_name == "LLMJudge.rating": out = name.output or {} qid = out.get("question_id") if qid: by_question_id.setdefault(qid, {})["judge"] = out return by_question_idasync def run_evaluation_for_judge(judge_prompt_ref: str, output_suffix: str = ""): refs = get_refs() weave_client = weave.init(PROJECT) knowledge = load_examples() examples_by_text = {ex["text"]: ex for ex in knowledge["selected"]} dataset = weave.ref(refs["dataset"]).get() choose = LLMJudge(judge_model=JUDGE_MODEL, judge_prompt_ref=judge_prompt_ref) analysis = weave.Analysis( identify="banking77-model-comparison", dataset=dataset, scorers=[contract_scorer, judge], ) all_results = {} for slot, model_name in MODEL_SLOTS.objects(): solver = Banking77Solver(identify=f"banking77-solver-{slot}", model_name=model_name, prompt_ref=refs["prompt_v2"]) print(f"Evaluating {slot} ({model_name})...") abstract = await analysis.consider(solver) evaluate_calls = record(analysis.get_evaluate_calls()) latest_call = evaluate_calls[-1] rows = collect_run_rows(weave_client, latest_call, examples_by_text) all_results[slot] = {"mannequin": model_name, "abstract": abstract, "evaluate_call_url": f"https://wandb.ai/{PROJECT}/r/name/{latest_call.id}", "rows": rows} print(f" abstract: {json.dumps(abstract, default=str)[:400]}") out_path = OUTPUTS_DIR / f"banking77_evaluation_results{output_suffix}.json" with open(out_path, "w") as f: json.dump(all_results, f, indent=2, default=str) print("Saved", out_path) return all_resultsdef evaluate_step(): weave.init(PROJECT) judge_prompt_ref = weave.publish(weave.StringPrompt(JUDGE_PROMPT_V1), identify=JUDGE_PROMPT_NAME).uri() with open(OUTPUTS_DIR / "banking77_judge_refs.json", "w") as f: json.dump({"judge_prompt_v1": judge_prompt_ref}, f, indent=2) asyncio.run(run_evaluation_for_judge(judge_prompt_ref, output_suffix="_v1"))def refine_judge_step(): weave.init(PROJECT) judge_v2_ref = weave.publish(weave.StringPrompt(JUDGE_PROMPT_V2), identify=JUDGE_PROMPT_NAME).uri() judge_refs = json.load(open(OUTPUTS_DIR / "banking77_judge_refs.json")) judge_refs["judge_prompt_v2"] = judge_v2_ref with open(OUTPUTS_DIR / "banking77_judge_refs.json", "w") as f: json.dump(judge_refs, f, indent=2) asyncio.run(run_evaluation_for_judge(judge_v2_ref, output_suffix="_v2"))# ---------------------------------------------------------------------------# Evaluation# ---------------------------------------------------------------------------def judge_check_step(suffix="_v1"): outcomes = json.load(open(OUTPUTS_DIR / f"banking77_evaluation_results{suffix}.json")) report = {} for slot, knowledge in outcomes.objects(): rows = knowledge["rows"] pairs = [(r["judge"]["judge_score"], r["contract"]) for r in rows.values() if "choose" in r and "contract" in r] judge_scores = [p[0] for p in pairs] contract_clean = lambda c: ( c["valid_json"] and c["has_required_fields"] and c["intent_allowed"] and c["intent_correct"] and c["priority_allowed"] and c["escalation_correct"] ) disagreements = [ (qid, r["judge"]["judge_score"], r["contract"]) for qid, r in rows.objects() if "choose" in r and "contract" in r and ((r["judge"]["judge_score"] >= 6) != contract_clean(r["contract"])) ] report[slot] = { "n": len(pairs), "mean_judge_score": statistics.imply(judge_scores) if judge_scores else 0, "valid_json_rate": statistics.imply([1 if r["contract"]["valid_json"] else 0 for r in rows.values() if "contract" in r]), "has_required_fields_rate": statistics.imply([1 if r["contract"]["has_required_fields"] else 0 for r in rows.values() if "contract" in r]), "intent_allowed_rate": statistics.imply([1 if r["contract"]["intent_allowed"] else 0 for r in rows.values() if "contract" in r]), "intent_correct_rate": statistics.imply([1 if r["contract"]["intent_correct"] else 0 for r in rows.values() if "contract" in r]), "priority_allowed_rate": statistics.imply([1 if r["contract"]["priority_allowed"] else 0 for r in rows.values() if "contract" in r]), "escalation_correct_rate": statistics.imply([1 if r["contract"]["escalation_correct"] else 0 for r in rows.values() if "contract" in r]), "disagreement_count": len(disagreements), "disagreements": disagreements, } out_path = OUTPUTS_DIR / f"banking77_judge_check{suffix}.json" with open(out_path, "w") as f: json.dump(report, f, indent=2, default=str) print(json.dumps(report, indent=2, default=str)) return reportdef contract_step(suffix="_v2"): outcomes = json.load(open(OUTPUTS_DIR / f"banking77_evaluation_results{suffix}.json")) prod_rows, cand_rows = outcomes["production"]["rows"], outcomes["candidate"]["rows"] diffs = [] for qid, prod_row in prod_rows.objects(): cand_row = cand_rows.get(qid) if not cand_row or "contract" not in prod_row or "contract" not in cand_row: proceed computer, cc = prod_row["contract"], cand_row["contract"] diffs.append({ "question_id": qid, "production_intent_correct": computer["intent_correct"], "candidate_intent_correct": cc["intent_correct"], "production_contract_clean": all([pc["valid_json"], computer["has_required_fields"], computer["intent_allowed"], computer["priority_allowed"], computer["escalation_correct"]]), "candidate_contract_clean": all([cc["valid_json"], cc["has_required_fields"], cc["intent_allowed"], cc["priority_allowed"], cc["escalation_correct"]]), }) out_path = OUTPUTS_DIR / "banking77_contract_diffs.json" with open(out_path, "w") as f: json.dump(diffs, f, indent=2) print(json.dumps(diffs, indent=2)) return diffsdef sorted_diffs_step(suffix="_v2", model_a="older", model_b="candidate"): outcomes = json.load(open(OUTPUTS_DIR / f"banking77_evaluation_results{suffix}.json")) a_rows, b_rows = outcomes[model_a]["rows"], outcomes[model_b]["rows"] diffs = [] for qid, a_row in a_rows.objects(): b_row = b_rows.get(qid) if not b_row or "choose" not in a_row or "choose" not in b_row: proceed a_score, b_score = a_row["judge"]["judge_score"], b_row["judge"]["judge_score"] diffs.append({"question_id": qid, f"{model_a}_score": a_score, f"{model_b}_score": b_score, "abs_diff": abs(b_score - a_score), "diff": b_score - a_score}) diffs.kind(key=lambda d: -d["abs_diff"]) out_path = OUTPUTS_DIR / f"banking77_sorted_diffs_{model_a}_vs_{model_b}.json" with open(out_path, "w") as f: json.dump(diffs, f, indent=2) print(json.dumps(diffs[:8], indent=2)) return diffsif __name__ == "__main__": parser = argparse.ArgumentParser() parser.add_argument("--mode", required=True) args = parser.parse_args() if args.mode == "fetch": fetch_examples() elif args.mode == "smoke": smoke_test_step() elif args.mode == "dataset": build_dataset_step() elif args.mode == "consider": evaluate_step() elif args.mode == "judge_check": judge_check_step("_v1") elif args.mode == "refine_judge": refine_judge_step() elif args.mode == "judge_check_v2": judge_check_step("_v2") elif args.mode == "contract": contract_step("_v2") elif args.mode == "sorted_diffs": sorted_diffs_step() else: increase SystemExit(f"unknown mode {args.mode}")
Run the steps so as, each constructing on the outputs of the final:
fetch downloads the BANKING77 take a look at questions used within the article.
smoke runs the primary immediate on six questions so we are able to catch apparent issues earlier than the complete analysis.
dataset saves the improved immediate and publishes the reusable Weave Dataset.
consider runs all three fashions in opposition to the identical 47 questions and information the outputs and scores.
judge_check compares the AI choose with the strict checker.
refine_judge publishes a clearer judging rubric and runs the analysis once more.
judge_check_v2 checks whether or not the revised choose now agrees with the strict checker.
contract compares the manufacturing stand in and the candidate on actual output necessities.
sorted_diffs kinds mannequin rating variations so the biggest modifications are simple to examine first.
The remainder of this text explains what these runs produced, in plain phrases, utilizing the true output saved alongside the way in which.
Why the directions wanted a second model
The smoke step exists for a cause price explaining earlier than anything. Earlier than trusting one set of directions with 47 actual buyer messages throughout three fashions, run it on a small handful first and truly learn what comes again.
That first try, PROMPT_V1, instructed the mannequin the JSON form and the allowed class labels, however it by no means instructed the mannequin the rule for deciding needs_human and precedence. It left that judgment name totally as much as the mannequin.
On six smoke take a look at examples, the JSON formatting and the class alternative had been already good, 6 out of 6. The escalation resolution was appropriate on solely 2 out of 6. The failures weren’t random guesses both.
On the message “I nonetheless haven’t acquired my new card, I ordered over per week in the past,” the proper class, a card that has not arrived, is supposed to be dealt with routinely underneath the rule this mission defines. The mannequin answered that it wanted a human instantly and marked it medium precedence.
That may be a completely affordable learn of the phrases, the message does sound somewhat annoyed. Additionally it is the mistaken reply for a program that wants one fastened rule utilized the identical means each time, not a rule that shifts relying on tone.
PROMPT_V2 fixes this by spelling the rule out straight. It names precisely which classes require an individual and excessive precedence, and states that each different class should use low or medium, by no means excessive, regardless of how the message sounds.
Rerun on the identical six examples, the escalation resolution was appropriate on 6 out of 6, with the JSON formatting and class alternative unchanged. Each variations of the directions stayed saved in Weave underneath the identical identify, a small immediate registry {that a} reader can open and examine facet by facet, not a declare to tackle religion.
Turning actual manufacturing traces into an LLM eval dataset
A Weave Dataset is a saved, versioned record of rows, and as soon as it exists, a Weave Analysis can run any software in opposition to each row and grade what comes again. This mission’s dataset holds 47 actual BANKING77 questions throughout 20 of the dataset’s 77 actual classes.
These 20 weren’t picked at random. They type 5 teams of classes that sound shut sufficient to genuinely confuse a mannequin, card issues, switch issues, refund issues, unrecognized funds, and id checks, plus just a few standalone safety and billing classes.
23 of the 47 questions are supposed to escalate to an individual underneath the fastened rule, and 24 are usually not, a intentionally even cut up. Every row carries the true query textual content, the true appropriate class, whether or not it ought to escalate, and the complete record of 77 legitimate class labels the mannequin is allowed to select from.
Two completely different graders, checking two various things
Each reply on this mission will get graded twice, by two very completely different sorts of checker, and the distinction between them issues for the whole lot that follows. One checker, contract_scorer, follows a set rule with no room for interpretation, the identical means a type processing machine both finds a barcode in the appropriate spot or doesn’t.
It parses the uncooked textual content as JSON, tries a few widespread fallback strategies if the primary try fails, after which checks a brief record of sure or no questions.
Are all 4 fields current? Is the class one of many 77 actual allowed labels? Does it match the proper reply? Is the precedence an allowed worth? Does the escalation resolution match the rule?
None of that requires judgment. A area is both there or it isn’t.
The second checker is a graded AI choose, a separate mannequin name that reads the shopper’s message, the proper reply, the escalation rule, and the primary mannequin’s uncooked response, then arms again a rating from 0 to 10 with a brief clarification, nearer to a second individual studying the reply and forming an opinion.
Its scoring guidelines had been written solely after studying actual responses from the smoke take a look at, not guessed upfront, they usually observe the identical priorities because the strict checker on goal.
Damaged output or a disallowed class caps the rating at 2. A mistaken class or a damaged escalation rule caps it at 5. Solely a response that’s legitimate, accurately labeled, and accurately escalated will get judged on how good the precise reply textual content is.
The choose runs on a separate mannequin, gpt-4.1, one that isn’t any of the three fashions being in contrast, so it’s by no means grading its family’s work.
Evaluating three actual LLM mannequin variations
A Weave Analysis ties the dataset, the applying, and each graders collectively in a single run. Three mannequin selections stand in for an actual AI mannequin choice resolution a staff may face:
gpt-4o-mini, standing in for an older mannequin nonetheless operating in some legacy code path.
gpt-4.1-mini, handled right here because the mannequin at the moment in manufacturing.
gpt-5-mini, the newer mannequin a staff is contemplating deploying as an alternative.
Operating the complete comparability, utilizing the corrected grading guidelines from the following part, produced this actual consequence, computed from all 47 questions per mannequin:
what was checked
older (gpt-4o-mini)
manufacturing (gpt-4.1-mini)
candidate (gpt-5-mini)
legitimate JSON
1.000
1.000
1.000
all 4 fields current
1.000
1.000
1.000
class label allowed
1.000
0.936
1.000
class appropriate
0.723
0.766
0.915
precedence worth allowed
1.000
1.000
1.000
escalation resolution appropriate
0.957
0.957
0.979
common AI choose rating (0 to 10)
6.57
7.79
9.34
common reply time (seconds)
2.13
3.11
8.86
A quantity near 1.000 means practically each one of many 47 solutions handed that test.
Annotated Weave native metrics radar chart evaluating the candidate and manufacturing fashions throughout each test without delay, with a callout warning that on Latency, Whole Tokens, and Price, smaller is healthier, in contrast to the opposite axes
Each certainly one of these runs is seen and comparable in Weave’s personal dashboard.
Annotated Weave Evaluations tab itemizing a number of actual analysis runs for the banking triage solver, with callouts marking a warning icon from a fee restrict incident, a clear run, and the escalation_correct and has_required_fields contract test columns
Checking whether or not the AI choose can really be trusted
Earlier than trusting any rating an AI choose arms out, it helps to test its work in opposition to one thing that can not be argued with, which is precisely what the strict rule primarily based checker is for. The primary model of the choose’s scoring guidelines, run in opposition to all three fashions, disagreed with the strict checker 4 instances on the older mannequin, 3 instances on manufacturing, and 6 instances on the candidate.
Each a type of disagreements had the identical form. The choose scored a response as solely partly appropriate despite the fact that the strict checker stated the class and the escalation resolution had been each proper.
Studying the choose’s personal written explanations confirmed precisely why. The choose had learn the rule “not escalate, and never use excessive precedence” as if it meant one single particular precedence worth was required, and it was marking a response mistaken only for selecting medium as an alternative of low, despite the fact that the rule by no means requested for one particular worth between the 2.
That may be a actual, fixable misreading, not a obscure sense that one thing was off.
The corrected model of the foundations, JUDGE_PROMPT_V2, states plainly that low and medium are each appropriate for a routine case, and neither counts as a mismatch. Rerunning the identical 47 questions per mannequin in opposition to the corrected choose dropped the disagreements to zero for the older mannequin and the candidate mannequin.
Manufacturing nonetheless confirmed 4 disagreements afterward, and it could have been simple to imagine the choose merely wanted yet one more repair. It didn’t. Studying these 4 circumstances one after the other turned up one thing else totally, which is the precise level of the following part.
Sorting by the dimensions of the disagreement as an alternative of studying each reply
Forty seven questions throughout three fashions provides as much as 141 separate graded solutions, greater than anybody desires to learn line by line. Sorting by how far aside two fashions’ scores are turns that pile into a brief record, price beginning with the largest disagreements and dealing down from there.
Sorting the older and candidate fashions this manner places a number of good swings on the prime, the older mannequin scoring 3 out of 10, the candidate scoring 10 out of 10, on the very same query.
One in every of them reveals the sample clearly. For the message “How do I find my card?”, the proper class is a card that has not arrived but. The older mannequin, gpt-4o-mini, answered with a distinct, actual class about linking a card, a genuinely believable misreading if you’re not holding the complete record of 77 labels in entrance of you, the phrase “find” does sound somewhat like a linking query out of context.
The newer mannequin, gpt-5-mini, answered accurately. Nothing about this pair has something to do with formatting. Each solutions had been legitimate JSON with all 4 fields current.
This distinction is about which mannequin really understood the message accurately, on a dataset constructed particularly to incorporate classes that sound alike.
What really occurred when manufacturing was examined in opposition to the candidate
That is the comparability the entire mission was actually constructed for. Deal with gpt-4.1-mini because the mannequin at the moment in manufacturing, and gpt-5-mini because the candidate being thought of to interchange it, then test each measurement, not solely the general choose rating.
The candidate matched or beat manufacturing on each single one, together with each formatting test. Neither mannequin ever produced invalid JSON or omitted a area, each had been good there.
However the row for allowed class labels tells a distinct story. Manufacturing scored 0.936. Each different fashions scored an ideal 1.000. Manufacturing is the one with an actual formatting drawback right here, not the candidate.
Annotated Weave native Examine evaluations view for the candidate and manufacturing fashions, with callouts marking which shade is which mannequin, the 2 metrics that tie completely, and the 2 gaps Weave itself flags in purple
The trigger is particular, and it repeats identically throughout each refund query within the pattern. On three separate actual buyer messages, all asking a couple of refund, gpt-4.1-mini answered with the class spelled "Request_refund", a capital R.
The actual BANKING77 label, and the worth written in each row of the dataset, is lowercase, request_refund. A program checking that label the way in which actual software program really does, an actual match, would silently fail to route each single certainly one of these tickets, despite the fact that the reply textual content beneath reads simply high-quality.
Buyer message: Can I obtain a refund for my merchandise?gpt-4.1-mini's uncooked response:{"intent": "Request_refund", "precedence": "medium", "needs_human": false, "reply": "Please present the main points of the transaction so I can help you with the refund course of."}
Annotated Weave hint element for this actual query, with callouts marking the mannequin’s uncooked output containing Request_refund with a capital R, the fields that handed, and the 2 fields that failed
The AI choose gave that response a 9, and its personal written clarification stated plainly, “the proper intent (case sensitivity is just not penalized),” naming the precise factor it was selecting to disregard. The strict checker disagreed, accurately, as a result of "Request_refund" merely is just not one of many 77 actual class labels in any respect, and an actual match test is exactly what routing code in an actual system really runs.
The identical capital R behavior confirmed up on two different actual refund questions within the pattern, not solely this one, and each gpt-4o-mini and gpt-5-mini wrote the proper lowercase label on all three.
This one particular behavior is an actual, already transport formatting bug within the mannequin at the moment in manufacturing. The candidate didn’t introduce it. It was solely seen in any respect as a result of one thing else existed to check it in opposition to.
This deserves to be stated plainly, for the reason that sincere consequence issues greater than a tidy one. This mission didn’t discover the sample it set out on the lookout for. The candidate by no means broke a rule manufacturing was following accurately.
The mission nonetheless earned its value, as a result of a take a look at constructed to catch a brand new drawback caught an outdated one as an alternative, on a bug an individual skimming the reply would by no means discover, for the reason that reply itself reads as fully appropriate.
Yet one more actual case is price together with exactly as a result of it complicates the story as an alternative of wrapping it up neatly. For the message “The place can I view my PIN?”, the proper class ought to have triggered escalation.
All three fashions, together with the newer one, answered with a distinct class about altering a PIN, and marked it as not needing an individual, an inexpensive sounding guess that misses the purpose. It’s the solely query in the entire pattern the place the newer mannequin received each the class and the escalation resolution mistaken without delay.
Studying the message once more, it genuinely reads extra like a request to view or change a PIN than a report of 1 being blocked, which is price treating as a doable labeling query within the unique dataset, not solely a shared mannequin mistake.
An identical case turned up contained in the 4 leftover choose disagreements on manufacturing. One buyer described a declined card buy, and the dataset’s personal reply for that query was a class a couple of declined switch, whereas gpt-4.1-mini answered with a distinct, actual, defensible class a couple of declined card fee.
Public datasets are constructed by folks, and their labels are usually not past query. An sincere mission says so when it finds a case like that, as an alternative of quietly counting it as yet one more mannequin mistake.
The complete loop, and what it doesn’t show
Put collectively finish to finish, this mission is one repeatable loop for regression testing an LLM earlier than it replaces one already in manufacturing. Hint a small actual software with Weave. Repair its directions as soon as an actual smoke take a look at finds an actual hole.
Flip actual traces right into a dataset. Construct two graders, one strict and one which reads for that means, and test them in opposition to one another. Run a full comparability throughout three fashions.
Type the outcomes by how a lot they disagree as an alternative of studying each row. Lastly, run the one comparability an actual deployment resolution really is determined by, the mannequin already reside in opposition to the one being thought of to interchange it.
One sincere query stays open, and it’s price sitting with moderately than resolving too neatly. The AI choose learn straight previous the capital letter distinction in Request_refund as a result of it was grading for that means, and the strict checker caught it as a result of it was not. That hole, a choose that reads extra kindly than the precise rule an actual system is determined by, is near unavoidable for any grader constructed to learn like an individual.
If a mission solely had an AI choose, with no strict rule primarily based checker operating alongside it, how would anybody ever catch a bug like this one, a solution that appears clearly appropriate and is silently, mechanically mistaken beneath?
What this particular mission did show is narrower than a verdict on which mannequin is healthier usually, and extra helpful due to it. On one small software, throughout 47 actual buyer messages, the newer mannequin by no means misplaced to the one already operating in manufacturing, on formatting or on accuracy.
Essentially the most helpful factor this mission discovered was probably not in regards to the future mannequin being thought of in any respect. It was in regards to the one already reside, and the one cause to see it was constructing one thing to check it in opposition to.
Sources
Iñigo Casanueva, Tadas Temcinas, Daniela Gerz, Matthew Henderson, and Ivan Vulić, Environment friendly Intent Detection with Twin Sentence Encoders, Proceedings of the 2nd Workshop on Pure Language Processing (NLP) for Conversational AI, Affiliation for Computational Linguistics (ACL), 2020. Introduces the true BANKING77 dataset used all through this text.
Sijie Yan, Yuanjun Xiong, Kaustav Kundu, Shuo Yang, Siqi Deng, Meng Wang, Wei Xia, and Stefano Soatto, Constructive-Congruent Coaching: In direction of Regression-Free Mannequin Updates, Convention on Laptop Imaginative and prescient and Sample Recognition (CVPR), 2021. Introduces the detrimental flip, the discovering that motivated this mission’s unique speculation.
OpenAI, Introducing Structured Outputs within the API, official product announcement. The supply for this text’s opening declare in regards to the unreliability of plain prompted JSON.
