Coding Brokers Do not Want Longer Historical past — They Want Intent Continuity

TL;DR

• I constructed a whole, working implementation in pure Python and shared precise benchmark numbers from actual runs (no simulated information).

• The core lesson: simply pulling up previous historical past is not the identical as realizing what’s truly nonetheless correct.

• A primary search setup solely grabbed 57% of the necessities a coding agent wanted. Including a verification layer pushed that to 100%.

• Out of 8 duties, the baseline bought zero proper, primary search bought 4, and intent-aware search nailed all 8.

• I did all of this with zero embeddings, zero vector databases, and completely no LLM calls within the pipeline.

• I additionally come clean with a bug in my authentic experiment design that nearly made my outcomes look means higher than they really have been.

Why Extra Historical past Is not Sufficient

I arrange a coding agent workflow that labored completely at first. However as soon as a undertaking bought lengthy sufficient, it began inflicting issues.

When a undertaking handed a couple of dozen steps, core guidelines started vanishing. Nobody deleted them. The context window was not full. These guidelines have been nonetheless technically sitting within the chat logs. They simply dropped off the radar as a result of new requests didn’t set off the agent to examine if an older resolution nonetheless mattered.

For example, you may inform the agent on day one to by no means expose inside database IDs in API responses. Sixty messages later, you ask it to construct a brand new authentication circulate. That new request says nothing about IDs. For the reason that agent lacks a transparent motive to look again, it skips that step and ships an endpoint leaking the precise information you tried to guard.

This isn’t a made up situation. It’s the precise take a look at case I used for this text. Under, I’ll present you ways three totally different strategies deal with this precise downside.

Each outcome proven right here comes from precise take a look at runs utilizing Python 3.12 with no exterior dependencies. You possibly can clone the repo and run run_experiment.py to breed the numbers your self, except I particularly name out an remoted take a look at.

Full Code: https://github.com/Emmimal/intent-continuity/

What Intent Continuity Truly Means

Phrases get combined up right here fairly quick, so allow us to clear up the definitions.

Customary RAG, launched by Lewis et al. (2020) [1], connects a language mannequin to a retrieval system that finds related data from an exterior information supply. The essential query is easy: what data is related to this question?

Larger context home windows let fashions maintain extra textual content directly. However that measurement doesn’t make the mannequin examine an previous rule buried sixty turns again. Liu et al. (2023) [2] identified that fashions miss particulars caught in the course of lengthy prompts, even inside their acknowledged limits.

But that misses the true level. Even with complete recall, a mannequin nonetheless has to attach an previous rule about database IDs to a brand new login job. Reminiscence failure just isn’t the issue. Deciding what issues is.

Intent continuity is totally different. It means carrying an previous requirement into a brand new job with out the consumer repeating it, whereas dropping that rule if one thing newer overrides it.

Right here is the precise cut up this text focuses on:

Retrieval asks:"What historic data may be related?"Verification asks:"Is that data nonetheless legitimate?"Intent continuity asks:"What historic intent ought to affect this job, proper now?"

Proper now, most discuss agent reminiscence focuses purely on that first query.

Who This Is For

Construct this if you happen to run coding brokers on long-running tasks the place guidelines get acknowledged as soon as and forgotten. Consider multi-week refactors, codebases full of previous design decisions, or groups the place whoever set a constraint three weeks in the past just isn’t the individual prompting the agent at this time.

Skip it for fast, single-session duties that carry no historical past. Skip it in case your undertaking is sufficiently small which you can simply paste your full necessities doc into each immediate. Skip it in case you are already manually repeating each rule to the agent on each flip. If a human is continually reminding the agent what to do, the system by no means must lookup previous selections.

In case your agent classes keep quick and your guidelines by no means shift, commonplace search or zero reminiscence works nice. Lengthy tasks simply don’t work that means.

Full Pipeline Structure

This diagram maps how an AI coding agent recovers and verifies undertaking necessities from earlier conversations as a substitute of counting on an extended context window or plain vector search. Interplay historical past strikes left to proper via rule-based intent extraction, then drops down and continues proper to left via candidate retrieval and verification, the place outdated or out-of-scope selections get dropped earlier than something reaches the agent. The pipeline ends with a deterministic agent and a requirement checker, so each recovered requirement is graded the identical means it was verified. The entire system runs in pure Python, with no embedding mannequin or vector database anyplace within the chain.

I set one strict rule earlier than writing a single line of code: 100% pure Python.

No API keys, no exterior LLM, no embedding fashions, and no vector databases.

A part of the rationale was comfort. I needed anybody to clone the repo and run it in beneath a second with zero setup friction. However the larger motive was management. If I relied on an embedding mannequin, the benchmark outcomes would simply get tousled in how good or dangerous that particular mannequin occurred to be.

The verification logic is what truly does the heavy lifting right here, and I needed to show it may possibly stand fully by itself two ft.

The pipeline begins by turning uncooked chat logs into structured requirement data.

The extractor is simply easy. It scans every message for sentences that appear like necessities, identifies the a part of the system the requirement seems to focus on, and extracts particular values when they’re current.

# extractor.py (abridged)TRIGGER_PHRASES = [    "must", "never", "required", "require", "prefer", "should",    "always", "migrating", "let's use", "default", "for now",]def _looks_like_requirement(textual content: str) -> bool:    decrease = textual content.decrease()    return any(phrase in decrease for phrase in TRIGGER_PHRASES)

Set off-Phrase Flagging vs. Floor Reality — 70 Interactions

The important thing takeaway right here is 100% recall. The extractor caught each single planted requirement within the take a look at set.

Precision landed at 0.80 on goal. Three on a regular basis sentences tripped the filter as a result of they used set off phrases like “should”—as an example, telling somebody you should run to a dentist appointment.

The following step cleans these up. The part classifier checks if a candidate maps to an actual system half. If it doesn’t discover a match, it drops the merchandise.

I saved these false positives within the benchmark intentionally. A key phrase filter claiming flawless precision on a tuned take a look at set often simply hides its weaknesses. I needed predictable, measurable habits as a substitute.

Put merely, the extractor is written to catch an excessive amount of reasonably than miss a rule in silence.

Is that this code fancy? By no means. It’s only a light-weight extractor that feeds structured information to the subsequent steps. An actual manufacturing app would want one thing a lot heavier.

Element 2: Candidate Retrieval and the Area Schema

Candidate retrieval figures out which previous data to examine earlier than operating any verification. It depends on two easy indicators. First, does the document share the identical system part as the present job? Second, does it belong to a linked part listed in a website schema?

# domain_schema.pyCOMPONENT_RELATIONSHIPS = {    "auth": ["security", "api"],    "api": ["security", "testing"],    "safety": ["auth", "api"],    "database": ["deployment"],    "deployment": ["database"],    "testing": ["api"],    "ui": [],    "efficiency": [],}

I wrote this schema as soon as primarily based on normal backend design guidelines. For example, auth work impacts safety and API habits, whereas API work requires testing and safety evaluations.

The system applies this precise schema to each single job with out modification.

That mounted method issues. An earlier model let me outline customized relationships for every particular person job, which felt like dishonest. I’ll clarify why that skewed issues shortly.

Element 3: Verification

That is the a part of the pipeline that really handles intent continuity. As soon as the system pulls up candidate guidelines, verification runs two fast checks on each: has a more recent rule changed it, and does it apply to the present job context?

The system figures out if a rule is outdated utilizing a easy rule as a substitute of guide labels. If two data share the identical part, scope, and goal key, however have totally different values, the newer one replaces the older one.

Identical Rule, Two Totally different Outcomes

Take R2 and R7. They discuss the identical key in the identical scope, so the later one wins and drops the previous one.

Then take a look at R4 and R5. They disagree on the database engine too, however they aim totally different scopes: manufacturing versus prototype. The verification step retains each lively as a result of they apply to separate environments.

Dealing with that distinction mechanically issues loads. If the system handled them as conflicting, it might break a core use case. Writing code to compute that distinction as a substitute of hardcoding labels turned out to be probably the most crucial design alternative in the entire undertaking.

Element 4: The Compiler and the Deterministic Agent

No matter data survive the verification stage get flattened right into a clear key-value context. From there, they go straight right into a template that mimics a coding agent.

# compiler.py — decision rule, utilized identically for each situationdef compile_context(context_records):    provenance = {}    for document in sorted(context_records, key=lambda r: r.index):        provenance[record.effect_key] = document  # later overwrites earlier    fields = {key: r.effect_value for key, r in provenance.objects()}    token_estimate = sum(len(r.textual content.cut up()) for r in context_records)    return fields, provenance, token_estimate

The simulated agent itself is deliberately easy. It begins with a set set of defaults, will get up to date by no matter fields it truly receives, and has zero capacity to guess a rule it was by no means handed.

That’s the entire level of the setup. Each outcome you see beneath comes down fully to what every search technique managed to recuperate. It has nothing to do with a language mannequin having a great or dangerous day, as a result of there isn’t any mannequin within the pipeline in any respect.

Element 5: The Checker

A single perform grades each run utilizing the very same ground-truth discipline listing each time.

No method will get particular therapy or a unique rubric. The required fields for every job are locked in earlier than any search technique runs, and each methodology is measured in opposition to that very same mounted commonplace.

What Occurs on Process T1

Right here is how the pipeline performs in opposition to one of many benchmark duties:

“Implement the brand new authentication circulate.”

Floor fact: This job depends on three unspoken constraints: the OAuth2 migration from flip 39, a backward-compatibility rule from flip 4, and an internal-ID hiding rule from flip 15. The immediate doesn’t point out any of them.

Three Situations, One Process

This diagram walks via an actual supersession case from the intent-continuity experiment: an early resolution to make use of JWT-based authentication, later changed by a choice emigrate to OAuth2. Naive lexical retrieval, the form of habits you’d get from plain key phrase or vector similarity search, finds each data and lets the newer one win purely as a result of it was talked about extra just lately, a coincidence of ordering reasonably than an precise validity examine. Intent-aware retrieval finds the identical two data however runs an specific verification step that determines the older document is outdated earlier than both one reaches the coding agent. Each approaches land on the proper auth methodology on this explicit case, which is strictly the purpose: one bought there by luck, and the opposite by design, and that distinction is invisible till you take a look at a case the place the ordering does not occur to save lots of you.

Customary key phrase search will get the auth methodology proper, however solely as a result of R7 occurs to overwrite R2 throughout context meeting. The compiler simply retains the final document it sees, reasonably than determining that the older JWT resolution was truly invalid. It utterly misses backward compatibility and ID publicity as a result of these sentences don’t share a single phrase with “implement the brand new authentication circulate.”

Intent-aware retrieval grabs these hidden necessities via the area schema as a substitute of counting on key phrase matches. The verification step explicitly determines that R2 is outdated earlier than something reaches the agent, which is a deliberate validation step reasonably than a random ordering quirk.

It additionally picks up R10, a rate-limiting constraint, proper alongside the three graded necessities. R10 is a sound historic constraint that sits exterior the guidelines for this particular job, proving the system captures context with out over-specializing.

Said as plainly as attainable throughout the three circumstances:

• Baseline: “I have no idea the previous.”

• Customary key phrase search: “I discovered one thing with matching key phrases.”

• Intent-aware: “I discovered a number of associated objects, checked which of them have been nonetheless legitimate, and reconstructed what truly issues.”

The Experiment: 70 Interactions, 12 Necessities, 8 Duties

I constructed an artificial undertaking historical past as a substitute of utilizing actual chat logs for a similar motive the pipeline has zero exterior dependencies. I needed a floor fact I might totally confirm, reasonably than a dataset the place determining what the agent ought to have recognized turns into a subjective judgment name.

The setup accommodates seventy chronologically ordered interactions: twelve real planted necessities, three lure sentences designed to journey up a key phrase extractor with out being actual guidelines, and fifty-five strains of extraordinary noise like standup reminders, pull request feedback, and informal chat.

The 12 Planted Necessities

The 8 Later Duties (None Restate Their Dependencies)

Measuring What It Truly Recovers

All numbers beneath come from actual runs of run_experiment.py. Nothing here’s a projection or an estimate.

Mixture Outcomes Throughout All 8 Duties

This bar chart exhibits the ultimate task-completion outcomes from the intent-continuity experiment throughout the identical 8 coding duties. A coding agent with no entry to historical past handed 0 of 8 duties, breaking a requirement it was by no means informed nonetheless utilized. Naive lexical retrieval, the form of outcome you’d anticipate from primary key phrase or vector-similarity search with no validity checking, handed 4 of 8. Intent-aware retrieval, which provides an specific verification step to drop outdated or out-of-scope necessities earlier than they attain the agent, handed all 8. The hole between naive retrieval and intent-aware retrieval is the precise discovering of this experiment: retrieving associated historical past is not the identical as retrieving necessities the agent can at the moment belief.

Intent continuity, as carried out right here, just isn’t a compression approach. I need that acknowledged explicitly reasonably than left for a reader to deduce from the desk.

Intent-aware retrieval makes use of extra tokens than commonplace key phrase retrieval—199 versus 155, or roughly 28 p.c extra—as a result of it accurately recovers necessities that commonplace retrieval misses outright, and that trade-off prices one thing. The core declare right here is about correctness, not compression. The system used extra tokens and produced higher job accuracy, reasonably than a smaller footprint.

“Irrelevant Retrieved” Is Not a Single Quantity

Eleven of ordinary retrieval’s seventeen irrelevant hits are pure chatter matched by lexical accident, which is real junk with zero relation to the duty.

Intent-aware retrieval’s ten extras cut up evenly: 5 are true noise accurately filtered out throughout verification, and 5 are data that attain the agent with out being on that particular job’s graded guidelines. These 5 further data are actual, at the moment legitimate context, not errors.

That distinction issues. Customary retrieval’s extras will not be assured to be proper; they’re simply retrieved. For Process T3, commonplace retrieval pulls in each the proper manufacturing database resolution and the stale prototype resolution, and the stale one wins the sphere as a result of there isn’t any verification layer to cease it.

Per-Process Move/Fail

The Time I Nearly Shipped a Rigged Experiment

The area schema described in Element 2 went via an earlier model that was a lot narrower and far worse. I had declared these part relationships per job as a substitute of worldwide. Process T1 was individually informed prematurely, “you additionally depend upon api and safety.” Process T4 was individually informed, “you additionally depend upon safety and testing.” These two hand-picked declarations occurred to be precisely the elements these two duties’ appropriate solutions wanted, and nothing extra.

That isn’t a discovery mechanism. That’s a solution key dressed up as a retrieval rule, and it straight undercut all the premise of this undertaking, which is meant to work with out being informed the place to look.

I caught it the one means that really works: I deleted the per-task trace and reran the experiment with nothing put instead.

Model With vs. With out the Per-Process Trace (Ablation)

The drop was not refined. It failed precisely the 2 duties that had been individually hand-fed their solutions—proof that the trace had been doing actual, load-bearing work all the time. I had almost missed it just because the ultimate combination rating seemed good.

The repair, proven in full in Element 2, was changing the per-task trace with one normal schema, authored a single time, and utilized uniformly to each job, together with the six that by no means wanted the additional assist.

Making use of it uniformly reasonably than choosing it price one thing actual. Irrelevant data retrieved rose from 4 to 10, and tokens provided rose from 161 to 199, as a result of the schema now additionally fires harmlessly for duties that by no means wanted it. The outcome stayed at 8 out of 8, however this time, it earned it.

An artificial benchmark you constructed your self is the best factor on this planet to unconsciously rig, since you already know the solutions earlier than you write the take a look at. Delete the half you believe you studied is doing an excessive amount of heavy lifting and see what breaks. It’s the solely sanity examine that really labored right here.

Trustworthy Design Choices

The part and key phrase dictionaries are hand-authored for this area reasonably than realized. It is a managed demonstration of the verification mechanism, not a general-purpose extraction system you can level at an arbitrary codebase tomorrow.

Retrieval on this experiment makes use of plain lexical phrase overlap as a substitute of an embedding mannequin or vector database. This alternative retains retrieval high quality from changing into a confounding variable. If I had used a selected embedding mannequin, a skeptical reader might fairly argue the entire comparability trusted which mannequin I occurred to choose. Swapping in an actual embedding mannequin would seemingly change the recall numbers for normal retrieval, however it might not change the underlying argument. The verification step is what does the fascinating work, and it stays agnostic to how candidates have been discovered.

Eight duties and twelve necessities make up an indication reasonably than a statistically powered examine. The scope is sized to be totally inspectable and reproducible, to not generalize with confidence to arbitrary manufacturing codebases.

The simulated agent is a deterministic template on goal, not an actual coding language mannequin. This ensures each result’s attributable strictly to what every retrieval technique recovered, reasonably than to mannequin habits on a given day.

The checker solely checks fields explicitly declared in every job’s floor fact. That could be a slim rubric by design, not a normal measure of code high quality.

Commerce-offs and What’s Lacking

• Actual extraction: The rule-based extractor works right here as a result of I management the dataset. A manufacturing model would want a genuinely sturdy extraction entrance finish, seemingly a small classifier reasonably than a easy trigger-phrase listing.

• Embedding-based candidate retrieval: The retrieval step is a clear swap level. Drop in an embedding mannequin for candidate technology and the verification layer downstream doesn’t want to alter in any respect.

• An actual coding LLM: The deterministic agent template exists particularly to isolate what every retrieval technique recovers. Changing it with an precise mannequin would take a look at a unique speculation: whether or not the mannequin accurately makes use of the recovered context, reasonably than simply whether or not the context was efficiently discovered.

• Cross-session persistence: All the things right here runs utterly in-process. A light-weight persistent retailer sharing the identical document interface would enable intent continuity to outlive throughout restarts.

Closing

Retrieval will get you what is associated. Verification will get you what is legitimate. Intent continuity will get you what nonetheless issues, proper now, for the duty in entrance of you.

Most techniques optimize the primary and skip the opposite two. That is why they maintain transport code that quietly breaks a choice somebody made weeks in the past.

Full code: https://github.com/Emmimal/intent-continuity/

References

[1] Lewis, P., Perez, E., Piktus, A., et al. (2020). Retrieval-Augmented Era for Data-Intensive NLP Duties. NeurIPS 33, 9459–9474. https://arxiv.org/abs/2005.11401

[2] Liu, N. F., Lin, Okay., Hewitt, J., et al. (2023). Misplaced within the Center: How Language Fashions Use Lengthy Contexts. arXiv:2307.03172. https://arxiv.org/abs/2307.03172

Disclosure

All code on this article was written by me. It’s authentic work, developed and examined on Python 3.12. All benchmark numbers come from precise runs of the system and are reproducible by cloning the repository and operating run_experiment.py. Not one of the outcomes have been calculated or simulated after the actual fact.

The system makes use of zero exterior dependencies and runs fully on the Python commonplace library. It doesn’t use an embedding mannequin, vector database, or LLM API. I’ve no monetary relationship with any software, library, or firm talked about on this article.

All diagrams on this article, together with the featured picture, have been created by the writer. The featured picture was generated with ChatGPT (DALL·E).