Vector RAG Isn’t Sufficient — I Constructed a Context Graph Layer for Multi-Agent Reminiscence

• I wasn’t making an attempt to construct a brand new reminiscence structure. I used to be making an attempt to know why one agent stored forgetting selections made by one other. The benchmark got here later.
• Multi-agent programs lose cross-agent selections as a result of flat transcripts and vector search each have a structural blind spot — not only a noise drawback.
• A context graph shops information as entities and relationships as a substitute of textual content chunks, so it might reply questions that want two information mixed.
• This isn’t an idea. Three reminiscence architectures, 5 scripted eventualities, 18 graded queries, absolutely deterministic, zero LLM calls.
• Context graph: 88.9% accuracy at 26.9 tokens/question. Uncooked historical past dump: 61.1% accuracy at 490.9 tokens/question. Vector-only RAG: 50.0% accuracy at 75.9 tokens/question.
• I discovered two actual bugs constructing this — stale-fact retrieval and an entity-matching hole. Each are within the article.

The Downside That Made Me Construct This

I constructed a three-agent pipeline that labored nice for brief duties. However the second the dialog dragged on and an agent wanted to recall a previous resolution, the entire thing fell aside.

Right here is precisely the way it broke: Agent_Planner would resolve the venture ought to use PostgreSQL. Then, twenty turns of “sounds good” and “I’ll get to it” would go. Ultimately, Agent_Reviewer would pipe up and ask what storage expertise we had been utilizing. Even with your entire uncooked transcript sitting proper there within the context window, the agent couldn’t reply reliably.

I used to be working this pipeline domestically as a facet venture for EmiTechLogic simply to see how far I may push multi-agent coordination earlier than it hit a wall. Seems, it didn’t take very lengthy.

Initially, I assumed this was only a mannequin limitation. It isn’t. It’s a reminiscence structure drawback that often triggers one in all two large complications relying on the way you attempt to repair it.

The Different Repair: Vector Search and the Relational Entice

In the event you swap to vector search, you repair the noise drawback however instantly create a distinct one. A vector retailer retrieves chunks that look just like your question; it doesn’t retrieve relationships between information.

If a key resolution lives in a single chunk and a vital dependency notice about that call lives in one other, a similarity search has no approach to mix them—irrespective of how good your embedding mannequin is.

Each approaches hit completely different structural ceilings. As a substitute of guessing which compromise was “ok,” I made a decision to measure them each.

What This Downside Truly Is

To be clear about what this text is not: this isn’t a token-compression drawback, and it’s not a staleness drawback. It’s a structural retrieval drawback. Some questions can solely be answered by combining two separately-stated information, and neither a rising context window nor a vector index has a mechanism to do this. That may be a fully completely different failure mode than those I’ve written about earlier than, and it wanted a distinct benchmark.

The Check Setup

To check this, I constructed 5 deterministic eventualities containing 18 graded queries and ran all three reminiscence architectures towards the very same conversations.

All the outcomes beneath come from actual runs of that benchmark utilizing a localized setup:

• Surroundings: Python 3.12, CPU-only (no GPU wanted)
• API Calls: Zero
• Consistency: Reproduced identically throughout two separate machines

Code Repo: You could find the whole implementation and run the assessments your self right here: https://github.com/Emmimal/context-graph-benchmark/

What “Context Graph” Means Right here

A flat reminiscence retailer (whether or not it’s a uncooked chat transcript or a vector index) treats each single flip as an unbiased unit of textual content. To retrieve one thing, you simply discover the unit that greatest matches your question.

A context graph adjustments the underlying construction fully. It treats reminiscence as distinct entities with typed relationships connecting them:

• AuthModule —–> DEPENDS_ON —–> RateLimiter
• Agent_Implementer —–> ASSIGNED_TO —–> AuthModule

Retrieval on this mannequin means traversing these relationships as a substitute of simply matching key phrases or semantic vectors.

That structural distinction solely issues for one particular class of questions: something that requires you to mix two separately-stated information.

Think about a query like: “Which staff owns the element that relies on the service that X selected?”

There isn’t a single reply chunk sitting anyplace within the uncooked dialog historical past. The reply doesn’t exist as a block of textual content. It solely exists as a path by way of a number of information. A flat retailer can’t assemble that path on the fly. A graph walks proper by way of it.

Who This Is For

This strategy is price constructing in case you run multi-agent pipelines the place one agent’s resolution should be accurately retrieved by a distinct agent many turns later. It’s constructed for programs the place questions routinely require combining two or extra separately-stated information, or any long-running agent dialog the place the token value of re-sending historical past is turning into an actual line merchandise.

You must skip it for single-agent, single-turn duties as a result of there isn’t a cross-agent state to lose. Skip it in case your queries are at all times single-fact lookups with no joins. Vector RAG will get you a lot of the accuracy there at a fraction of the engineering value. Lastly, skip it in case your staff has no tolerance for an additional transferring half. A graph wants an extraction step (which is rule-based on this benchmark, however requires an LLM name in manufacturing) {that a} flat retailer avoids.

In case your multi-agent system finishes its work in a single alternate, plain context passing works fantastic. This drawback exhibits up particularly when conversations run lengthy and selections must survive previous the flip they had been made in.

The Three Architectures

Why There Are No LLM Calls within the Benchmark

I purposely ignored LLM calls from each stage of this benchmark: no LLMs for extraction, none for question answering, and none for grading.

If an actual LLM dealt with the extraction, the benchmark would measure LLM variance as a lot as precise architectural variations. Utilizing deterministic, rule-based stand-ins ensures that each single run produces the very same numbers.

I ran this take a look at independently on two completely different machines whereas scripting this piece. The output matched byte-for-byte, sustaining accuracy to 4 decimal locations and token counts all the way down to the precise integer.

Constructing a Benchmark That Doesn’t Secretly Favor the Graph

The simplest approach to make a graph win a benchmark is to solely ask it clear, single-fact questions. That proves nothing. To maintain the testing honest, each situation follows 4 strict guidelines:

• Distractors outnumber information: Each situation comprises much more “sounds good,” “I’ll verify that,” and “no blockers on my finish” turns than precise concrete selections.
• Queries span bodily distance: Some queries are requested proper after a reality is said (direct), some are requested many turns later (distant), and a few require stitching two separate information collectively (be a part of). An instance of a be a part of question is: “Which element does the module owned by Agent_Implementer rely upon?”
• Some queries are simple on function: Direct, single-fact lookups are included particularly to present the flat architectures a good shot.
• Grading is totally deterministic: The benchmark makes use of substring matching towards a hand-written floor reality slightly than counting on an LLM choose.

@dataclass
class Flip:
    turn_id: int
    turn_type: TurnType          # FACT, DISTRACTOR, or QUERY
    speaker: str
    textual content: str
    topic: str | None = None    # structured triple, FACT turns solely
    predicate: str | None = None
    object: str | None = None
    fact_id: str | None = None
    query_type: str | None = None # "direct", "distant", "be a part of"
    required_fact_ids: tuple = ()
    ground_truth: str | None = None

The benchmark covers 5 distinct eventualities throughout completely different domains: software program planning, a analysis pipeline, incident response, buyer assist escalation, and a knowledge pipeline.

Throughout these 5 setups, there are 18 whole queries cut up into three particular classes:

• 6 Direct queries: Lookups requested instantly after the very fact is said.
• 7 Distant queries: Lookups requested many turns after the very fact is said.
• 5 Be part of queries: Questions that require combining two separately-stated information to get the reply.

Structure 1: Uncooked Historical past Dump

Each single flip will get appended to a flat transcript, and your entire transcript will get resent on each question. That is precisely what you get by default when you don’t design a reminiscence system on function.

I constructed this to function a genuinely honest baseline. It will get the total, excellent transcript with nothing hidden from it. The reply extraction makes use of key phrase overlap with gentle stemming, searched from the newest flip backward. This setup intently mirrors how a context-stuffed immediate tends to weight recency anyway.

class RawHistoryDump:
    def ingest(self, flip: Flip) -> None:
        self.transcript.append(f"{flip.speaker}: {flip.textual content}")

    def answer_query(self, query_turn: Flip) -> tuple[str, int]:
        immediate = self._build_prompt(query_turn)   # the ENTIRE transcript
        tokens = count_tokens(immediate)
        reply = self._extract_answer(query_turn)
        return reply, tokens

The associated fee mannequin matches precisely what you see in manufacturing: each question resends your entire rising dialog historical past.

Structure 2: Vector-Solely RAG

Each flip, reality and distractor alike, will get embedded and saved as a bit. An actual vector retailer doesn’t know upfront which turns will matter later. On a question, the top-Okay most related chunks are retrieved.

I used TF-IDF as a substitute of a neural embedding API for a similar purpose I averted LLM calls elsewhere. TfidfVectorizer has no random state, making it deterministic by development. It is usually not a toy stand-in. TF-IDF is an actual sparse-retrieval methodology utilized in manufacturing RAG, usually paired with dense embeddings in a hybrid setup.

class VectorOnlyRAG:
    def _retrieve(self, query_text: str) -> checklist[str]:
        if not self.chunks:
            return []
        corpus = self.chunks + [query_text]
        vectorizer = TfidfVectorizer()
        matrix = vectorizer.fit_transform(corpus)
        sims = cosine_similarity(matrix[-1], matrix[:-1]).flatten()
        top_idx = sims.argsort()[::-1][:self.top_k]
        return [self.chunks[i] for i in top_idx if sims[i] > 0]

(The precise implementation wraps fit_transform in a strive/besides block to deal with the uncommon edge case of a question containing solely cease phrases. I skipped that right here for area, however it’s within the repository.)

The structural ceiling stays clear: a be a part of question requires combining two distinct information. When these information are said throughout two completely different turns, no single chunk comprises each items of data. No embedding mannequin can repair that limitation by itself.

Structure 3: The Context Graph

Info get written as (topic, predicate, object) triples right into a NetworkX directed multigraph. Distractor turns by no means get written in any respect. That is the one place this structure will get a bonus the opposite two don’t: filtering information earlier than it ever hits storage.

In manufacturing, that filtering step is an LLM name performing entity extraction. On this benchmark, it’s deterministic as a result of the situation setup already tags which turns are information. I’m isolating precisely what the storage and retrieval structure does by itself, with extraction held fixed as a said assumption. I’m not claiming to have solved extraction totally free.

class ContextGraph:
    def ingest(self, flip: Flip) -> None:
        if flip.topic is None:
            return  # distractors carry no structured triple; not saved
        self.graph.add_node(flip.topic)
        self.graph.add_node(flip.object)
        self.graph.add_edge(flip.topic, flip.object,
                             predicate=flip.predicate, fact_id=flip.fact_id)

The join-query traversal is the half doing the true work. It performs a two-hop stroll throughout the graph nodes as a substitute of trying to find a single textual content chunk that occurs to comprise each information.

def _answer_join(self, query_turn, talked about):
    for entity in talked about:
        out_edges, in_edges = self._edges_touching(entity)
        intermediates = [v for _, v, _ in out_edges] + [u for u, _, _ in in_edges]
        for intermediate in intermediates:
            further_out, _ = self._edges_touching(intermediate)
            for _, goal, information in further_out:
                if goal != entity:
                    # rating candidates by predicate relevance
                    ...

Right here’s the distinction in search area throughout all three:

What Truly Occurred Once I First Ran It

The primary full run, with all three architectures constructed, scored the context graph at 0% accuracy.

I’m together with this as a result of it’s the half most “I constructed X” posts skip. I may have rewritten the eventualities to be friendlier as a substitute of debugging the code. That may have given me a faux outcome. I traced it as a substitute.

Bug 1: Entity Vocabulary Mismatch

Graph nodes had been named issues like Project_Alpha or AuthModule. The queries, written the best way an agent would truly phrase them, stated “this venture” or “the authentication module.” A literal substring match between the question textual content and the node title discovered completely nothing.

That is the very same vocabulary-mismatch drawback individuals criticize vector seek for. It simply hits the graph at write time as a substitute of question time.

The repair was a small alias desk standing in for an actual entity-linking step, which might often be dealt with by an LLM name in manufacturing. Utilizing a graph doesn’t get you out of this drawback. It merely strikes the issue from query-time retrieval to write-time decision. That’s an ongoing engineering value, not a one-time repair.

Bug 2: Returning Stale Info With Full Confidence

That is the precise subject I might flag first to anybody transport this sample in a manufacturing surroundings.

One situation includes a assist ticket that begins at a precedence stage of “excessive” and will get reclassified to “vital” mid-conversation. When querying “what’s the present precedence?”, the graph returned “excessive”—the stale worth, with the very same confidence it will have given the present one.

The trigger was easy: my first ingest() implementation simply added each new edge and by no means eliminated the previous one. The graph held two HAS_PRIORITY edges originating from the identical node. Whichever edge occurred to be visited first within the iteration order received the lookup, fully ignoring which reality was truly present.

# the bug
Ticket_4471 --HAS_PRIORITY--> "excessive"      # said first
Ticket_4471 --HAS_PRIORITY--> "vital"  # said later, supersedes the primary
# each edges exist without delay; nothing tells the graph which one is "now"

A flat chat dump searched with recency bias tends to floor the newer point out simply by scanning backward. In distinction, a graph with no time mannequin arms again both reality with equal structural confidence as a result of graphs don’t natively know a relationship has been changed except you explicitly inform them.

That failure mode is worse than a fuzzy search returning a stale chunk. The graph seems to be fully authoritative even when it’s fully fallacious.

The repair: when a brand new reality restates an current (topic, predicate) pair, the previous edge will get dropped earlier than the brand new one is written.

def ingest(self, flip: Flip) -> None:
    if flip.topic is None:
        return
    self.graph.add_node(flip.topic)
    self.graph.add_node(flip.object)

    stale_edges = [
        (u, v, k) for u, v, k, data in self.graph.edges(keys=True, data=True)
        if u == turn.subject and data.get("predicate") == turn.predicate
    ]
    for u, v, ok in stale_edges:
        self.graph.remove_edge(u, v, key=ok)

    self.graph.add_edge(flip.topic, flip.object,
                         predicate=flip.predicate, fact_id=flip.fact_id)

In case you are transport something like this, dealing with reality supersession will not be non-compulsory. It’s the precise line between constructing a dependable reminiscence layer and constructing a serious legal responsibility.

Remaining Benchmark Outcomes

5 eventualities, 18 queries, absolutely deterministic, reproduced identically on two separate machines.

The context graph wins on accuracy and makes use of about 18x fewer tokens per question than the uncooked dump. That isn’t a tradeoff—it’s a win on each axes.

Vector RAG’s token value can also be low and isn’t the graph’s predominant differentiator. Each architectures retrieve a bounded variety of gadgets, so each keep low cost no matter dialog size. What separates the graph from vector RAG is the be a part of column: 80% versus 20%. That hole is the structural argument for a graph—vector similarity has no native approach to mix two separately-stated information.

The uncooked dump’s accuracy got here in greater than I anticipated at 61.1%, and it earns that. An ideal, lossless transcript with respectable key phrase matching does fantastic on single-fact lookups. It falls aside particularly on joins (40%) for a similar structural purpose as vector RAG, simply with a a lot greater token invoice.

One limitation was left in on function: two queries within the data-pipeline situation fail as a result of they consult with an entity by description slightly than title—”the dataset that at present has an anomaly” as a substitute of naming Upstream_Orders instantly. Fixing that requires actual semantic understanding of a descriptive clause, not easy alias matching. Extending the alias desk to cowl my very own take a look at queries would imply overfitting the benchmark slightly than representing an actual limitation, so it stays damaged. In case your manufacturing queries lean towards descriptive references, finances for an LLM-based decision step as a substitute of an ever-growing static alias desk.

How Token Price Scales With Dialog Size

My working assumption getting in was that raw-dump token value scales O(N^2) as conversations develop. I measured it as a substitute of assuming it, as a result of transport an imprecise complexity declare to an viewers that checks it’s a quick approach to lose credibility.

The setup: one reality said as soon as, adopted by a rising variety of filler turns (starting from 10 as much as 800), adopted by a single question asking for that reality. This isolates per-query token value as a pure perform of dialog size, with info content material held fully mounted.

When the dialog size grew 80x (from 10 to 800 turns), the uncooked dump’s token rely grew 64.15x. In the meantime, vector RAG and the context graph each grew 1.00x—fully flat.

The uncooked dump’s tokens-per-query is O(N), which is linear in dialog size, converging to about 12.6 tokens per filler flip. It’s not quadratic. The O(N^2) story solely turns into correct in case you sum the fee throughout a complete multi-query dialog: Q queries, every run towards a transcript that has grown linearly, lands round O(N.Q) whole value. That’s the actual quantity, only a extra exact one than “every question prices O(N^2).”

Vector RAG and the context graph each maintain flat at O(1) per question as a result of each architectures solely ever pull a bounded variety of gadgets no matter how lengthy the dialog will get.

What I’d Flag Earlier than Taking This to Manufacturing

A number of issues are price being direct about earlier than anybody copies this sample into an actual software.

On latency: Vector RAG is definitely the slowest structure right here, not the graph. It refits TF-IDF over your entire corpus on each question name slightly than sustaining an incremental index. Averaged throughout all 5 eventualities, context graph question answering got here in at 0.050ms versus Vector RAG’s 1.764ms.

That hole closes in an actual deployment the place you’d cache the vectorizer as a substitute of refitting from scratch—the benchmark measured default conduct, not best-case engineered variations. The graph’s occasional spike to 1.9ms comes fully from be a part of queries strolling a number of candidate paths earlier than scoring.

On what the alias desk is definitely doing: The entity alias desk that lets “the authentication module” resolve to AuthModule is a hardcoded stand-in for actual entity linking. In manufacturing, that step is an LLM name. The benchmark is deterministic as a result of I hardcoded the aliases I anticipated—it doesn’t imply the vocabulary-mismatch drawback is solved for arbitrary question phrasing. It’s a actual ongoing value that I’m flagging, not hiding.

On token estimation: I used a ~4-characters-per-token heuristic as a substitute of tiktoken, as a result of tiktoken downloads its BPE rank file from a distant URL on first use—a hidden community dependency in a benchmark constructed to have none. The heuristic is utilized identically throughout all three architectures, so it can’t bias the comparability between them, however the absolute token numbers are approximations.

On what this benchmark didn’t take a look at: Distractor turns listed below are generic chatter—”no blockers on my finish,” “sounds good.” Actual manufacturing noise is topically near precise information. I might anticipate all three architectures to drop in accuracy beneath adversarial noise, and I’ve not measured that, so I received’t declare the lead holds.

On what’s lacking for manufacturing use: actual entity extraction (the ingest() interface already accepts a structured triple, so swapping in an LLM-based extractor is a contained change), incremental vector indexing, graph pruning for long-running conversations that accumulate entities indefinitely, and chronic storage. The repo features a NetworkX-to-Neo4j export path for anybody who wants sturdiness and concurrent multi-agent writes—however that’s an non-compulsory step, not a efficiency improve. The explanations to make that bounce are transactional ensures and concurrency, not uncooked question pace.

What the Numbers Truly Say

None of this wanted an even bigger mannequin or an extended context window. Each single outcome got here from altering how info is represented, not how a lot information will get crammed right into a immediate.

In the event you take just one quantity from this text, take the join-query hole: 80% versus 20–40%. That’s the actual argument for structured reminiscence, not the token financial savings.

Whereas the token financial savings are actual and measurable, they’re secondary. On this benchmark, questions requiring two information from fully completely different components of the dialog had been the place the graph structure confirmed its largest benefit. That hole held constantly throughout all 5 eventualities, not simply those that occurred to be simple for a graph.

The total venture—5 eventualities, three architectures, the take a look at suite that locks these numbers in as regression assessments, and the Neo4j export path—is offered on the repository beneath.

Full supply code: https://github.com/Emmimal/context-graph-benchmark/

References

[1] Liu, N. F., Lin, Okay., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2024). Misplaced within the Center: How Language Fashions Use Lengthy Contexts. Transactions of the Affiliation for Computational Linguistics, 12, 157–173. https://doi.org/10.1162/tacl_a_00638

[2] Zhang, W., Zhou, Y., Qu, H., & Li, H. (2026). Loosely-Structured Software program: Engineering Context, Construction, and Evolution Entropy in Runtime-Rewired Multi-Agent Methods (arXiv:2603.15690). arXiv. https://arxiv.org/abs/2603.15690

[3] A. Kollegger, “Context Graphs & Agentic Choices,” Neo4j Developer Weblog, Jan. 31, 2026. [Online]. Out there: https://medium.com/neo4j/context-graphs-agentic-decisions-9a125f22f411

[4] W. Lyon, “When Your Brokers Share a Mind: Constructing Multi-Agent Reminiscence with Neo4j,” Neo4j Developer Weblog, Apr. 13, 2026. [Online]. Out there: https://medium.com/neo4j/when-your-agents-share-a-brain-building-multi-agent-memory-with-neo4j-bac609f17b23

[5] Macklin, N., Zaim, Z., & Erdl, A. (2026). Context Graphs and AI Reminiscence Throughout the Globe. Neo4j Developer Weblog. https://medium.com/neo4j/context-graphs-and-ai-memory-across-the-globe-bb17e293df32

[6] NetworkX documentation. https://networkx.org/

[7] Scikit-learn Builders, “TfidfVectorizer,” Scikit-learn Documentation. [Online]. Out there: https://scikit-learn.org/steady/modules/generated/sklearn.feature_extraction.textual content.TfidfVectorizer.html

[8] OpenAI. Counting tokens with tiktoken. https://github.com/openai/tiktoken

[9] Neo4j Python Driver documentation. https://neo4j.com/docs/api/python-driver/present/

Disclosure

All code on this article was written by me and is authentic work, developed and examined on Python 3.12 (Home windows, PyCharm). Benchmark numbers are from precise runs of the code within the linked repository and are reproducible by cloning it and working benchmark.py and measure_scaling.py, besides the place the article explicitly notes a quantity is a heuristic or estimate slightly than a measured outcome. I’ve no monetary relationship with any instrument, library, or firm talked about on this article.