Can an Condo Search Agent Name the Mannequin Fewer Instances and Nonetheless Discover Good Matches?

Of their paper FrugalGPT: The way to Use Giant Language Fashions Whereas Decreasing Price and Enhancing Efficiency, Lingjiao Chen, Matei Zaharia, and James Zou describe 3 ways to chop the price of calling a big language mannequin (LLM). They name these immediate adaptation, LLM approximation, and LLM cascade. The authors report that their cascade tries cheaper fashions first. On the duties they examined, it can match the efficiency of the most effective particular person mannequin with as much as 98 p.c value discount. That quantity belongs to their duties, so an agent that reads rental listings wants its personal measurement.

I constructed a small agent I might examine name by name. It checks house listings in opposition to a renter’s necessities, corresponding to a two bed room in Austin below $1,500 that permits canines. My first model despatched each itemizing to a robust mannequin, even when a metropolis identify or a lease determine already dominated the itemizing out. After I opened the hint, that meant 2,500 mannequin calls to seek out 101 actual matches.

That first model is a straw man, so this text measures three beginning factors as a substitute of 1. The primary sends each pair to the mannequin. The second is an agent that already searches by metropolis. The third runs a database question on metropolis, bedrooms, and lease earlier than any mannequin sees an inventory, which is what a cautious engineer would construct first.

This text reveals what I ran. I took a set pattern of 500 listings from a public dataset of 2019 United States rental adverts. I wrote 5 renter necessities and scored each model of the agent in opposition to the identical solutions. The sturdy mannequin was OpenAI’s gpt-6-sol, which I name Sol, and the cheaper mannequin was gpt-6-luna, which I name Luna.

I traced each name with Weights & Biases (W&B) Weave, the W&B instrument that information every step of an utility. The entire script is within the article, and the logged tables are in a W&B Report.

Towards the database question place to begin, the ultimate model value about 25 instances much less. The estimated value was $0.008 in opposition to $0.20 for a similar 2,500 checks, and the ultimate model returned all 101 matches with no unsuitable ones. Letting code settle the pets subject made up about 44 p.c of that drop, and utilizing the cheaper mannequin with a robust backup made up about 39 p.c. A shorter immediate and reused solutions made up the remaining.

The totals weren’t essentially the most helpful a part of the take a look at. Three findings stunned me, and I stored all of them.

• Weave’s value column confirmed $0.0000 for each name. The true token counts had been saved within the uncooked report, however the utilization desk displayed zero.

• Sol missed a match 10 instances out of 10 with the total itemizing, and located it 10 instances out of 10 with solely the title and physique. Sending much less textual content gave a greater reply.

• Each fashions reported a confidence of 5 on virtually each reply. A rule that escalates beneath 5 virtually by no means fires, so it can’t do a lot work.

The article additionally says the place the take a look at is weak. Solely one of many 101 matches will depend on studying textual content, so the take a look at measures value much better than it measures understanding.

What does the agent learn, and the way did I construct the take a look at?

The agent solutions one sure or no query for every itemizing and every renter requirement. I name one itemizing checked in opposition to one requirement a pair, so 500 listings and 5 necessities make 2,500 pairs.

The listings come from the Condo for Lease Categorized dataset within the College of California, Irvine (UCI) Machine Studying Repository. It holds 10,000 United States rental adverts with 22 fields, together with the title, the physique textual content, facilities, bedrooms, loos, lease, sq. ft, metropolis, state, and pets allowed. The dataset is licensed below Inventive Commons Attribution 4.0, and its itemizing timestamps run from September to December 2019. These are historic adverts, so nothing right here describes right now’s rents.

The pattern has 500 listings chosen with a set random seed of 42. It holds 200 from Austin, 100 from Dallas, 100 from Houston, and 100 from different cities. 5 renter necessities run in opposition to it, and each requirement additionally asks for a spot that permits canines.

• Austin, 1 bed room, lease as much as $1,300.

• Austin, 2 bedrooms, lease as much as $1,800.

• Austin, 2 bedrooms, lease as much as $1,500.

• Dallas, 2 bedrooms, lease as much as $1,600.

• Houston, 1 bed room, lease as much as $1,200.

The combination of cities is deliberate, as a result of a feed that covers a number of cities offers a filter one thing to reject. It additionally flatters the filter, and I return to that restrict within the outcomes. The prompts and the traces pass over the road handle, the coordinates, and the supply itemizing ID. Solely a row quantity corresponding to row_3445 identifies an inventory.

How do I do know a match is right, and what counts as value?

A pair is a real match when the itemizing is in the precise metropolis, has the precise variety of bedrooms, prices not more than the lease restrict, and permits canines. The primary three checks use fields with express values. The pets subject says issues like “Cats,Canines” or “Cats”, and that decides the fourth verify each time the sphere is crammed in.

The pets subject is clean on 4,163 of the ten,000 listings, so the textual content has to resolve these. There are 128 sampled listings that go the town, bed room, and lease checks with a clean pets subject. An AI assistant (Claude) learn each title and physique of these listings in opposition to one written rule. The rule says canines are allowed solely when the textual content says canines or pets are allowed. Silence, or a touch that doesn’t say so straight, counts as not confirmed.

The end result was lopsided. One itemizing, row_3445, calls itself “a pet pleasant group” and provides a 25 pound weight restrict, so I depend it as a match. Two listings trace at pets with out saying so. row_1808 lists a “Pet park” and row_8465 mentions “a pet bar on your favourite 4 legged associates”, so each depend as not confirmed. The opposite 125 listings say nothing about pets.

Earlier than this studying, I had written a key phrase seek for phrases corresponding to “pet pleasant” and “no pets”. It agreed with the studying on all 128 listings.

Two limits comply with from this. The labels are one reader’s judgment, and the 2 trace listings are controversial. Additionally, only one of the 101 true matches will depend on studying textual content, as a result of the opposite 100 come from the pets subject. The take a look at due to this fact says far more about value than about how properly a mannequin understands rental adverts.

Precision and recall rating the end result. Precision is the share of returned matches which can be right, and better is best. Recall is the share of true matches that had been returned, and better is best. With 101 true matches, one miss strikes recall by about one share level.

Price right here means API inference value, which is what the supplier fees for tokens. A token is a small chunk of textual content, roughly a brief phrase, and suppliers invoice enter tokens and output tokens at completely different charges. Output tokens embrace the hidden reasoning tokens a mannequin spends considering earlier than it solutions. The OpenAI pricing web page listed these normal charges on September 27, 2026. Sol value $2.00 per million enter tokens and $10.00 per million output tokens, and Luna value $0.10 and $0.50.

Each greenback determine beneath is an estimate from these charges and the token counts every response reported. It isn’t an bill, and it leaves out infrastructure and labor. I additionally report mannequin time, which is the sum of the time each name took.

How do I run the experiment?

The steps beneath work on macOS and Linux. On Home windows, activate the surroundings with .venvScriptsactivate. I ran every little thing with Python 3.9.6 and these bundle variations, weave 0.52.17, openai 2.48.0, pandas 2.3.3, and py7zr 1.0.0. The script wants an OpenAI API key with billing enabled and a free W&B account with its API key. With out the W&B key, the script stops when it calls weave.init, so the traces and the report are unavailable.

Create the working folder, set up the packages, and set each keys within the terminal you’ll use for each command on this article.

mkdir apartment-agent-cost && cd apartment-agent-costpython3 -m venv .venvsupply .venv/bin/activatepip set up weave openai pandas py7zrexport OPENAI_API_KEY="paste your OpenAI key right here"export WANDB_API_KEY="paste your W&B key right here"

The dataset downloads as a zipper file that holds a compressed 7z file. These instructions fetch and unpack it right into a information folder, which leaves information/apartments_for_rent_classified_10K.csv in place.

mkdir informationcurl -L -o information/residences.zip "https://archive.ics.uci.edu/static/public/555/house+for+lease+categorised.zip"python -c "import zipfile; zipfile.ZipFile('information/residences.zip').extractall('information')"python -c "import py7zr; py7zr.SevenZipFile('information/apartments_for_rent_classified_10K.7z').extractall('information')"

Save the next script as apartment_agent.py within the apartment-agent-cost folder. It holds every little thing the experiment wants, together with the necessities, the sphere guidelines, the bottom fact labels, the prompts, the cache, and the six configurations. Feedback within the code mark the elements the later sections clarify.

"""Condo search agent value experiment (apartment_agent.py).Run from the folder that comprises information/ and this file:    python apartment_agent.py baseline    python apartment_agent.py city_baseline    python apartment_agent.py field_query_baseline    python apartment_agent.py filter    python apartment_agent.py diminished    python apartment_agent.py cached    python apartment_agent.py luna    python apartment_agent.py remaining --threshold 5    python apartment_agent.py repeatRequires OPENAI_API_KEY and WANDB_API_KEY within the surroundings."""import argparseimport concurrent.futures as cfimport contextvarsimport hashlibimport jsonimport timefrom pathlib import Pathimport pandas as pdimport weavefrom openai import OpenAIDATA = "information/apartments_for_rent_classified_10K.csv"PROJECT = "apartment-agent-cost"SOL, LUNA = "gpt-6-sol", "gpt-6-luna"# USD per a million tokens (enter, output). Normal quick context charges, checked 2026-09-27.RATES = {SOL: (2.00, 10.00), LUNA: (0.10, 0.50)}PROMPT_VERSION = "v1"MAX_OUTPUT_TOKENS = 400MAX_ATTEMPTS = 2  # one name plus at most one retry when the reply isn't legitimate JSONREQUIREMENTS = [    {"id": "austin_1br", "city": "Austin", "state": "TX", "bedrooms": 1, "max_rent": 1300},    {"id": "austin_2br_a", "city": "Austin", "state": "TX", "bedrooms": 2, "max_rent": 1800},    {"id": "austin_2br_b", "city": "Austin", "state": "TX", "bedrooms": 2, "max_rent": 1500},    {"id": "dallas_2br", "city": "Dallas", "state": "TX", "bedrooms": 2, "max_rent": 1600},    {"id": "houston_1br", "city": "Houston", "state": "TX", "bedrooms": 1, "max_rent": 1200},]# ---------------------------------------------------------------- information and labelsdef load_sample(seed=42):    """Fastened analysis pattern: 200 Austin, 100 Dallas, 100 Houston, 100 different listings."""    df = pd.read_csv(DATA, sep=";", encoding="cp1252")    df["row_key"] = ["row_%d" % i for i in df.index]    elements = []    for metropolis, n in [("Austin", 200), ("Dallas", 100), ("Houston", 100)]:        elements.append(df[df.cityname == city].pattern(n, random_state=seed))    relaxation = df[~df.cityname.isin(["Austin", "Dallas", "Houston"])]    elements.append(relaxation.pattern(100, random_state=seed))    pattern = pd.concat(elements).sort_index()    pattern["split"] = ["dev" if int(k.split("_")[1]) % 2 == 0 else "take a look at" for okay in pattern.row_key]    return pattern.set_index("row_key", drop=False)# Labels for listings whose pets subject is clean and that go the exhausting subject checks for at the least one# requirement. Rubric: true solely when the textual content says canines or pets are allowed. Silence, or a touch corresponding to# "Pet park" (row_1808) or "a pet bar on your favourite 4 legged associates" (row_8465), counts as not confirmed.# The opposite 125 listings on this group say nothing about pets.BLANK_PETS_DOGS_ALLOWED = {"row_3445"}  # "a pet pleasant group", weight restrict 25 kilosdef dogs_ok(row):    """Floor fact pet rule. The pets subject wins. When it's clean, the reviewed labels above resolve."""    if isinstance(row.pets_allowed, str):        return "Canines" in row.pets_allowed    return row.row_key in BLANK_PETS_DOGS_ALLOWEDdef hard_fields_ok(row, req):    return (row.cityname == req["city"] and row.state == req["state"]            and row.bedrooms == req["bedrooms"] and row.worth <= req["max_rent"])def label(row, req):    return bool(hard_fields_ok(row, req) and dogs_ok(row))def prefilter(row, req):    """Deterministic checks. Returns reject, settle for, or ask (the mannequin should learn the textual content)."""    if not hard_fields_ok(row, req):        return "reject", "hard_fields"    if isinstance(row.pets_allowed, str):        return ("settle for" if "Canines" in row.pets_allowed else "reject"), "pets_field"    return "ask", "pets_text"# ---------------------------------------------------------------- prompts and mannequin callsdef requirement_text(req):    return ("Condo in %s, %s. %d bed room(s). Month-to-month lease at most $%d. Canines allowed."            % (req["city"], req["state"], req["bedrooms"], req["max_rent"]))def full_listing(row):    pets = row.pets_allowed if isinstance(row.pets_allowed, str) else "not listed"    return ("Title: %snBody: %snAmenities: %snBedrooms: %snBathrooms: %snRent: $%sn"            "Sq. ft: %snCity: %s, %snPets subject: %s"            % (row.title, row.physique, row.facilities if isinstance(row.facilities, str) else "not listed",               row.bedrooms, row.loos, row.worth, row.square_feet, row.cityname, row.state, pets))FULL_SYSTEM = ('You display screen house listings for a renter. Reply with JSON solely, like '               '{"match": true, "confidence": 5}. match is true provided that the itemizing meets each requirement. '               'confidence is 1 (guessing) to five (sure). If the itemizing doesn't say whether or not canines are '               'allowed, canines will not be confirmed and match is fake.')PETS_SYSTEM = ('Learn an house itemizing. Reply with JSON solely, like {"dogs_allowed": true, "confidence": 5}. '               'dogs_allowed is true provided that the textual content says canines or pets are allowed. Use false if it says no '               'pets, no canines, or doesn't say. confidence is 1 (guessing) to five (sure).')consumer = OpenAI()LISTINGS = {}def redact(inputs):    """Weave hook. Replaces immediate textual content so uncooked itemizing copy by no means reaches uploaded traces."""    out = dict(inputs)    if "messages" in out:        out["messages"] = [{"role": m["role"], "content material": "[redacted %d chars]" % len(m["content"])}                           for m in out["messages"]]    return out@weave.opdef call_model(mannequin, form, row_key, requirement_id):    """One traced mannequin name. Inputs are keys solely. Itemizing textual content is seemed up from reminiscence."""    row = LISTINGS[row_key]    if form == "full":        system = FULL_SYSTEM        consumer = "Necessities: %snnListing:npercents" % (            requirement_text(subsequent(r for r in REQUIREMENTS if r["id"] == requirement_id)), full_listing(row))    else:        system = PETS_SYSTEM        consumer = "Title: %snBody: %snPets subject: %s" % (row.title, row.physique,                                                        row.pets_allowed if isinstance(row.pets_allowed, str) else "not listed")    report = {"mannequin": mannequin, "form": form, "row_key": row_key, "requirement_id": requirement_id,              "prompt_tokens": 0, "completion_tokens": 0, "reasoning_tokens": 0, "latency_s": 0.0,              "makes an attempt": 0, "legitimate": False, "reply": None, "confidence": None}    for _ in vary(MAX_ATTEMPTS):        begin = time.perf_counter()        resp = consumer.chat.completions.create(            mannequin=mannequin, response_format={"sort": "json_object"}, max_completion_tokens=MAX_OUTPUT_TOKENS,            messages=[{"role": "system", "content": system}, {"role": "user", "content": user}])        report["latency_s"] += time.perf_counter() - begin        report["attempts"] += 1        report["prompt_tokens"] += resp.utilization.prompt_tokens        report["completion_tokens"] += resp.utilization.completion_tokens        particulars = resp.utilization.completion_tokens_details        report["reasoning_tokens"] += (particulars.reasoning_tokens or 0) if particulars else 0        strive:            information = json.masses(resp.selections[0].message.content material)            report["answer"] = bool(information["match"] if form == "full" else information["dogs_allowed"])            report["confidence"] = int(information.get("confidence", 0))            report["valid"] = True            break        besides (ValueError, KeyError, TypeError):            proceed    return reportdef cost_usd(rec):    rate_in, rate_out = RATES[rec["model"]]    return (rec["prompt_tokens"] * rate_in + rec["completion_tokens"] * rate_out) / 1e6# ---------------------------------------------------------------- configurationsclass Cache:    """Shops pet choices. The important thing consists of itemizing textual content, immediate model, and mannequin, so an edited    itemizing, a brand new immediate, or a special mannequin by no means reuses an previous reply."""    def __init__(self):        self.retailer, self.hits = {}, 0    def key(self, row, mannequin):        digest = hashlib.sha1(("%s|%s|%s|%s" % (row.title, row.physique, row.pets_allowed, PROMPT_VERSION)).encode())        return "%s:%s" % (mannequin, digest.hexdigest())def resolve(cfg, row, req, cache, threshold):    """Returns (predicted_match, list_of_call_records)."""    calls = []    if cfg == "baseline":        rec = call_model(SOL, "full", row.row_key, req["id"])        return bool(rec["answer"]), [rec]    if cfg == "field_query_baseline":        # The strongest extraordinary place to begin. A database question on metropolis, bedrooms, and lease runs first.        if not hard_fields_ok(row, req):            return False, calls        rec = call_model(SOL, "full", row.row_key, req["id"])        return bool(rec["answer"]), [rec]    if cfg == "city_baseline":        # A fairer place to begin. The feed is already searched by metropolis, so solely identical metropolis pairs attain Sol.        if not (row.cityname == req["city"] and row.state == req["state"]):            return False, calls        rec = call_model(SOL, "full", row.row_key, req["id"])        return bool(rec["answer"]), [rec]    verdict, _ = prefilter(row, req)    if verdict != "ask":        return verdict == "settle for", calls    form = "full" if cfg == "filter" else "pets"    mannequin = LUNA if cfg in ("luna", "remaining") else SOL    key = cache.key(row, mannequin) if cfg in ("cached", "remaining", "luna") else None    if key and key in cache.retailer:        cache.hits += 1        reply = cache.retailer[key]    else:        rec = call_model(mannequin, form, row.row_key, req["id"])        calls.append(rec)        reply = bool(rec["answer"])        if cfg == "remaining" and (not rec["valid"] or rec["confidence"] < threshold):            rec2 = call_model(SOL, "pets", row.row_key, req["id"])            calls.append(rec2)            reply = bool(rec2["answer"])        if key:            cache.retailer[key] = reply    return reply, calls@weave.opdef run_config(cfg, threshold=5, employees=8):    pattern = load_sample()    LISTINGS.replace({okay: r for okay, r in pattern.iterrows()})    pairs = [(row, req) for _, row in sample.iterrows() for req in REQUIREMENTS]    cache = Cache()    began = time.perf_counter()    def work(pair):        row, req = pair        pred, calls = resolve(cfg, row, req, cache, threshold)        return {"row_key": row.row_key, "requirement_id": req["id"], "break up": row.break up,                "label": label(row, req), "pred": pred, "calls": calls}    # Cached configurations run one pair at a time, so a repeated pet query is a assured cache hit.    n_workers = 1 if cfg in ("cached", "remaining", "luna") else employees    with cf.ThreadPoolExecutor(n_workers) as pool:        futures = [pool.submit(contextvars.copy_context().run, work, p) for p in pairs]        outcomes = [f.result() for f in futures]    wall = time.perf_counter() - began    out = {"config": cfg, "threshold": threshold, "n_pairs": len(outcomes), "wall_s": wall,           "cache_hits": cache.hits, "outcomes": outcomes}    Path("outputs").mkdir(exist_ok=True)    identify = "%s_tpercents" % (cfg, threshold) if cfg == "remaining" else cfg    Path("outputs/%s.json" % identify).write_text(json.dumps(out))    return summarize(out)@weave.opdef repeat_hard_cases(n=10):    """Asks the identical query n instances for the 2 hardest listings, to see how a lot solutions range."""    pattern = load_sample()    LISTINGS.replace({okay: r for okay, r in pattern.iterrows()})    rows = {"row_3445": "dallas_2br", "row_8465": "dallas_2br"}    out = []    for mannequin, form in [(SOL, "full"), (SOL, "pets"), (LUNA, "pets")]:        for key, req_id in rows.objects():            for _ in vary(n):                rec = call_model(mannequin, form, key, req_id)                out.append(rec)    Path("outputs/repeat.json").write_text(json.dumps(out))    return len(out)def summarize(out):    rows = out["results"]    calls = [c for r in rows for c in r["calls"]]    tp = sum(r["label"] and r["pred"] for r in rows)    fp = sum((not r["label"]) and r["pred"] for r in rows)    fn = sum(r["label"] and (not r["pred"]) for r in rows)    return {"config": out["config"], "pairs": len(rows), "model_calls": len(calls),            "cache_hits": out["cache_hits"], "tp": tp, "fp": fp, "fn": fn,            "precision": tp / (tp + fp) if tp + fp else None, "recall": tp / (tp + fn) if tp + fn else None,            "cost_usd": sum(cost_usd(c) for c in calls), "model_time_s": sum(c["latency_s"] for c in calls),            "wall_s": out["wall_s"],            "mean_call_latency_s": sum(c["latency_s"] for c in calls) / len(calls) if calls else 0.0}if __name__ == "__main__":    parser = argparse.ArgumentParser()    parser.add_argument("config", selections=["baseline", "city_baseline", "field_query_baseline", "filter", "reduced", "cached", "luna", "final", "repeat"])    parser.add_argument("--threshold", sort=int, default=5)    args = parser.parse_args()    weave.init(PROJECT, global_postprocess_inputs=redact)    if args.config == "repeat":        print(repeat_hard_cases())    else:        print(json.dumps(run_config(args.config, args.threshold), indent=2))

Run the configurations from the apartment-agent-cost folder, one command every. The baseline run makes 2,500 Sol calls and takes a number of minutes, and the others are faster.

python apartment_agent.py baselinepython apartment_agent.py city_baselinepython apartment_agent.py field_query_baselinepython apartment_agent.py filterpython apartment_agent.py diminishedpython apartment_agent.py cachedpython apartment_agent.py lunapython apartment_agent.py remaining --threshold 5python apartment_agent.py repeat

Every command saves its full outcomes to an outputs folder and prints a abstract. The block beneath is captured output from a rerun of the luna configuration in a contemporary folder with the precise script above.

{  "config": "luna",  "pairs": 2500,  "model_calls": 128,  "cache_hits": 29,  "tp": 101,  "fp": 0,  "fn": 0,  "precision": 1.0,  "recall": 1.0,  "cost_usd": 0.004947700000000001,  "model_time_s": 145.34897142399993,  "wall_s": 145.813283333,  "mean_call_latency_s": 1.1355388392499994}

The fashions don’t settle for a temperature setting, so each run makes use of the default and solutions can differ from run to run. Your numbers shall be near those on this article and won’t match digit for digit. The rerun above already reveals a distinction, as a result of it made no unsuitable match, and the run I captured for the comparability made one. The part on the cheaper mannequin explains why.

What does the baseline hint present?

The hint of the ship each pair baseline reveals one mannequin name per pair and no signal of which calls had been wanted. Weave information this by wrapping a operate with @weave.op, and the OpenAI integration provides a baby name for each request to the mannequin. Every hint is a tree of calls with inputs, outputs, timing, and token counts, and it really works like a receipt that lists each step the agent took.

The left view of the screenshot comprises essentially the most stunning results of the setup. Weave counted the request accurately, but its Utilization desk confirmed zero tokens and a complete value of $0.0000 for each fashions. The Weave value documentation says Weave applies inbuilt pricing for supported integrations. It additionally describes an add_cost() methodology for fashions and not using a worth, and these runs didn’t use it.

A zero isn’t a worth, so I calculated value from the token counts saved in every name report. The one name within the screenshot used 141 enter and 63 output tokens.

The best view reveals the redaction. The script passes a global_postprocess_inputs operate to weave.init, and that operate replaces each immediate with a be aware corresponding to [redacted 245 chars] earlier than the hint uploads. I then searched each saved name within the mission for distinctive itemizing phrases and for 400 avenue addresses from the pattern, and located none.

That first baseline is straightforward to summarize. Sol made 2,500 calls, one per pair, utilizing 618,635 enter tokens and 63,782 output tokens, of which 18,424 had been reasoning tokens. The estimated value was $1.88, and the calls added as much as 2,991 seconds of mannequin time. The baseline ran eight calls directly, so its wall clock time was 376 seconds.

Is sending each itemizing to a mannequin a good place to begin?

No, and a fairer comparability shrinks the financial savings. Sending each pair to Sol is the model I constructed first, and few actual brokers do it. Most slender the feed earlier than a mannequin sees something, so I measured two extra beginning factors on the identical 2,500 pairs with the identical full itemizing immediate.

The second place to begin sends solely pairs whose metropolis matches. It made 796 Sol calls and price an estimated $0.61, so it’s about 3 instances cheaper than sending every little thing. The third runs a database question on metropolis, bedrooms, and lease, then sends every surviving pair to Sol. It made 264 calls and price an estimated $0.20, about 9 instances cheaper than sending every little thing. That question is extraordinary database work and has no mannequin value.

All three runs missed the identical match, row_3445, which the subsequent sections clarify. I measure each later saving in opposition to the $0.20 question place to begin, as a result of a cautious engineer would construct it first. It nonetheless spends mannequin calls on 264 pairs, and people pairs are the place the remainder of the article begins.

Which mannequin calls can the pets subject settle and not using a mannequin?

One other 107 of these 264. A mail room clerk types envelopes by postal code, and the identical clerk may see that some envelopes already carry an approval stamp. The pets subject works like that stamp. When it’s crammed in, it says canines are allowed or it says they don’t seem to be, and no mannequin must learn the itemizing.

The diagram reveals the place the pairs went. Discipline checks rejected 2,236 pairs, and 1,704 of these had the unsuitable metropolis. That first department is similar work the database question does, and it covers 89 p.c of all pairs in my blended metropolis pattern.

The pets subject settled the subsequent 107 pairs. Seven had been rejected as a result of the sphere stated cats solely, and 100 had been accepted as a result of it listed canines. That left 157 pairs, which cowl 128 completely different listings, for a mannequin to learn.

The filter configuration sends solely these 157 pairs to Sol with the identical full immediate. Its estimated value fell from $0.20 for the question place to begin to $0.11, about 43 p.c decrease, and its mannequin time fell from 352 to 203 seconds. Precision stayed at 1.0 and recall stayed at 0.990, because it missed the identical match because the beginning factors. That’s what an accurate rule ought to do, as a result of it modifications which pairs attain the mannequin and leaves the reply for the remaining pairs alone.

Does sending much less textual content change the reply?

It modified the reply on this take a look at. The diminished configuration asks Sol one query, whether or not the title and physique say canines are allowed. It drops the facilities, lease, metropolis, and different fields, as a result of the sphere checks already dealt with them.

Enter tokens fell from 34,194 to 23,624, about 31 p.c decrease, and the estimated value fell from $0.114 to $0.098. The saving is smaller than the token drop suggests. Output tokens rose from 4,589 to five,039, and every output token prices 5 instances as a lot as an enter token.

The diminished run additionally discovered the match that every one three beginning factors missed. With the total itemizing, Sol answered no for row_3445, the pet pleasant group, with the best confidence. To verify that this was not luck, I requested every model of the query 10 instances. The total itemizing immediate gave 0 right solutions out of 10, and the title and physique immediate gave 10 out of 10.

I have no idea why. One potential cause is that the total immediate reveals a line saying the pets subject isn’t listed, proper subsequent to the instruction that unspoken canines depend as not confirmed. I didn’t take a look at that concept.

The repeat verify additionally lined the second exhausting itemizing, row_8465, the one with the “pet bar”. The desk beneath reveals how lots of the 10 solutions had been right for every mannequin and immediate. For row_8465, an accurate reply isn’t confirmed, following my label.

Each fashions break up on row_8465, answering sure two or 3 times in ten. That itemizing is a coin flip, and the desk reveals how a lot a single run can depend upon one such itemizing.

Can the agent reuse a solution safely?

Sure, if every saved reply remembers what produced it. A cache is a pocket book of previous solutions, and its threat is studying an previous reply for a query that has modified. The script keys every reply by the mannequin, the itemizing’s title, physique, and pets subject, and a immediate model string. Enhancing an inventory, altering the immediate model, or switching fashions due to this fact modifications the important thing, and the previous reply isn’t discovered.

The renter’s requirement isn’t a part of the important thing, as a result of the pet query doesn’t depend upon the requirement. The query is similar when two renters each desire a canine pleasant place in Austin. That’s the reason the cache helped, because the identical itemizing reveals up below a couple of requirement. It answered 29 of the 157 pairs from reminiscence, so mannequin calls fell from 157 to 128 and the estimated value fell from $0.098 to $0.082.

The cache within the script lives in reminiscence for one run. A manufacturing cache wants an expiry time and a approach to drop entries when an inventory is eliminated or edited. These elements had been exterior this take a look at.

Is the cheaper mannequin protected to make use of?

For this workload, virtually. Luna reads the identical 128 listings with the identical title and physique immediate, and the estimated value fell from $0.082 to $0.005, about 16 instances decrease once more. Within the captured run Luna made one unsuitable match, and in a rerun it made none. It stated sure to the “pet bar” itemizing with a confidence of 4 on a scale from 1 to five, so precision was 0.990 and recall was 1.0.

Escalation offers the agent a second opinion. Like a junior clerk who passes unsure instances to a senior one, Luna solutions first and Sol reads once more solely when Luna’s confidence falls beneath a threshold. Luna answered 5 for 127 of its 128 questions, and Sol answered 5 for all 157 of its questions. The arrogance scores barely range, so a threshold has little to work with.

I’ve to be trustworthy concerning the threshold. I break up the listings right into a growth half and a take a look at half, so I might tune the edge on one half and report on the opposite. The event half had no errors from any configuration, and each exhausting listings fell within the take a look at half. That left nothing to tune on, and I selected a threshold of 5 after seeing which reply had a confidence of 4. The ultimate result’s due to this fact an illustration and never a validated setting.

With that caveat, the ultimate configuration escalated one reply. Sol learn the “pet bar” itemizing once more, stated no with a confidence of 4, and the run had no unsuitable matches. It made 129 calls, 128 to Luna and 1 to Sol, and the estimated value was $0.0079.

The rerun within the setup part provides a warning. Luna alone made no unsuitable match there, and it answered the identical itemizing accurately with a confidence of 5. So one unsuitable match in opposition to none is noise on a coin flip itemizing, and my run can’t present that escalation fastened something.

Yet another value element issues. Luna’s 128 calls used 6,105 output tokens, greater than Sol’s 4,319 for a similar questions, as a result of Luna spent extra tokens on reasoning. About 61 p.c of Luna’s estimated value was output tokens, so limiting reasoning effort is a lever I didn’t take a look at.

A key phrase search would have matched my labels on all 128 clean listings, so is a mannequin price conserving on this loop in any respect? The repeat verify above reveals the place the 2 would differ, and the final part says what I might monitor to resolve.

What did the mixed model value, and what does the proof assist?

The bottom value model that made no errors within the captured runs was Luna with a Sol backup, at an estimated $0.0079 for two,500 pairs. The question place to begin value $0.20 for a similar pairs, about 25 instances extra, and the desk lists each configuration.

The interactive model of this desk, the fee chart, and each mannequin name are within the W&B Report.

The chart reveals the place the cash went. Ranging from the $0.20 question place to begin, the entire drop was about $0.19. The pets subject verify accounts for 44 p.c of it and the cheaper mannequin with a backup for 39 p.c. The shorter immediate accounts for 9 p.c, and reused solutions for 8 p.c. Mannequin time fell from 352 to 139 seconds.

Per 1,000 pairs, the estimated value went from $0.080 to $0.0032. That’s arithmetic on the speed card, and it doesn’t forecast a invoice for some other feed.

The primary two steps down the chart, from sending every little thing to a metropolis search after which to a full question, are extraordinary question work. They account for a lot of the distance between $1.88 and $0.008, and so they want no mannequin in any respect.

What this take a look at helps is slender. It ranks the fee drivers for one agent on one pattern of two,500 pairs with 101 true matches and one run per configuration. It can’t say how the agent behaves for actual renters, in different cities, on adverts from one other 12 months, or on listings with longer textual content. It additionally can’t decide a common finest setting.

It helps one conclusion. On this agent most mannequin calls had been avoidable, and code might settle most of them. The cheaper mannequin dealt with the remaining at about 4 p.c of the question place to begin’s value.

What ought to preserve working after launch?

A recurring verify ought to watch the identical numbers I used right here. Visitors, prompts, fashions, and retries all transfer value after launch, and a hint makes every of them seen. 4 numbers from this instance are sufficient to start out.

• Mannequin calls per pair. Sending each pair makes 1.0, the question place to begin makes 0.106, and the ultimate configuration makes 0.052. A soar means extra pairs are reaching the mannequin.

• The share of pairs the sphere checks settle. It was 93.7 p.c right here, and a drop means the feed or the necessities modified.

• Tokens per name, together with reasoning tokens. A immediate change or a mannequin change reveals up right here first.

• Precision and recall on a set set of pairs with recognized solutions. Rerunning these 2,500 pairs with the ultimate configuration prices lower than a cent on the charges above.

Two setup particulars will assist. Register the charges with add_cost() in order that Weave reveals actual numbers in its Utilization desk as a substitute of zero. Hold the immediate model string within the cache key, so a immediate edit can’t reuse an previous reply.

The arrogance scores deserve a watch too. If virtually each reply scores 5, an escalation rule that fires beneath 5 will not often hearth, and the backup mannequin will sit idle whereas the cheaper mannequin carries each determination.

Optimize for helpful matches, not the smallest invoice

The tactic on this article has 4 steps. Hint the run, discover the mannequin calls that code might settle, change one factor, and rating the identical pairs once more. On this agent that sequence lower the estimated value about 25 instances in opposition to a database question place to begin and left the solutions intact. Towards sending each pair, the lower was about 237 instances, however few actual brokers begin there. The identical steps might have uncovered a high quality loss, and the runs had been constructed to indicate one if it appeared.

Open one actual hint from your personal agent and depend the calls {that a} subject verify or a lookup might have answered. Then take a look at the most important value driver first, on a set set of examples with solutions you belief. The subsequent technical query I might ask is whether or not a decrease reasoning effort retains the identical solutions at a cheaper price, since output tokens drove most of Luna’s invoice.

Chosen Sources

1. FrugalGPT: The way to Use Giant Language Fashions Whereas Decreasing Price and Enhancing Efficiency, Lingjiao Chen, Matei Zaharia, and James Zou, 2023. This paper names the three value discount methods and reviews the cascade end result quoted within the opening. Its findings belong to the duties the authors examined.

2. Condo for Lease Categorized, UCI Machine Studying Repository, 2019, DOI 10.24432/C5X623, licensed below Inventive Commons Attribution 4.0. That is the supply of all 10,000 listings and the sphere descriptions.

3. W&B Weave value monitoring documentation. It describes how Weave reads token utilization, applies inbuilt costs for supported integrations, and provides customized prices with add_cost().

4. OpenAI API pricing, checked on September 27, 2026, for the Sol and Luna normal charges utilized in each value estimate.

5. OpenAI’s mannequin pages for Sol and Luna, checked on September 27, 2026. Sol is described as constructed for complicated coding and agentic workflows, and Luna as essentially the most environment friendly mannequin for centered, excessive quantity duties. The pages checklist gpt-6-sol because the default snapshot and gpt-6-luna as the one snapshot.