days, it’s all about brokers, which I’m all for, anbeyond fundamental vector search by giving LLMs entry to a variety of instruments:
- Internet search
- Numerous API calls
- Querying completely different databases
Whereas there’s a surge in new MCP servers being developed, there’s surprisingly little analysis occurring. Positive, you possibly can hook an LLM with varied completely different instruments, however do you actually know the way it’s going to behave? That’s why I’m planning a collection of weblog posts centered on evaluating each off-the-shelf and customized graph MCP servers, particularly those who retrieve data from Neo4j.
Mannequin Context Protocol (MCP) is Anthropic’s open normal that features like “a USB-C port for AI functions,” standardizing how AI techniques hook up with exterior information sources via light-weight servers that expose particular capabilities to purchasers. The important thing perception is reusability. As a substitute of customized integrations for each information supply, builders construct reusable MCP servers as soon as and share them throughout a number of AI functions.
An MCP server implements the Mannequin Context Protocol, exposing instruments and information to an AI consumer by way of structured JSON-RPC calls. It handles requests from the consumer and executes them towards native or distant APIs, returning outcomes to counterpoint the AI’s context.
To judge MCP servers and their retrieval strategies, step one is to generate an analysis dataset, one thing we’ll use an LLM to assist with. Within the second stage, we’ll take an off-the-shelf mcp-neo4j-cypher server and check it towards the benchmark dataset we created.
The objective for now’s to determine a stable dataset and framework so we are able to constantly examine completely different retrievers all through the collection.
Code is offered on GitHub.
Analysis dataset
Final 12 months, Neo4j launched the Text2Cypher (2024) Dataset, which was designed round a single-step strategy to Cypher era. In single-step Cypher era, the system receives a pure language query and should produce one full Cypher question that immediately solutions that query, primarily a one-shot translation from textual content to database question.
Nonetheless, this strategy doesn’t mirror how brokers really work with graph databases in observe. Brokers function via multi-step reasoning: they’ll execute a number of instruments iteratively, generate a number of Cypher statements in sequence, analyze intermediate outcomes, and mix findings from completely different queries to construct as much as a last reply. This iterative, exploratory strategy represents a essentially completely different paradigm from the prescribed single-step mannequin.
The present benchmark dataset fails to seize this distinction of how MCP servers really get utilized in agentic workflows. The benchmark wants updating to judge multi-step reasoning capabilities reasonably than simply single-shot text2cypher translation. This may higher mirror how brokers navigate advanced data retrieval duties that require breaking down issues, exploring information relationships, and synthesizing outcomes throughout a number of database interactions.
Analysis metrics
An important shift when transferring from single-step text2cypher analysis to an agentic strategy lies in how we measure accuracy.
In conventional text2query duties like text2cypher, analysis usually entails evaluating the database response on to a predefined floor reality, usually checking for precise matches or equivalence.
Nonetheless, agentic approaches introduce a key change. The agent might carry out a number of retrieval steps, select completely different question paths, and even rephrase the unique intent alongside the way in which. Consequently, there could also be no single appropriate question. As a substitute, we shift our focus to evaluating the ultimate reply generated by the agent, whatever the intermediate queries it used to reach there.
To evaluate this, we use an LLM-as-a-judge setup, evaluating the agent’s last reply towards the anticipated reply. This lets us consider the semantic high quality and usefulness of the output reasonably than the inner mechanics or particular question outcomes.
Consequence Granularity and Agent Conduct
One other necessary consideration in agentic analysis is the quantity of information returned from the database. In conventional text2cypher duties, it’s widespread to permit and even anticipate giant question outcomes, because the objective is to check whether or not the right information is retrieved. Nonetheless, this strategy doesn’t translate nicely to evaluating agentic workflows.
In an agentic setting, we’re not simply testing whether or not the agent can entry the right information, however whether or not it may possibly generate a concise, correct last reply. If the database returns an excessive amount of data, the analysis turns into entangled with different variables, such because the agent’s capability to summarize or navigate giant outputs, reasonably than specializing in whether or not it understood the consumer’s intent and retrieved the right data.
Introducing Actual-World Noise
To additional align the benchmark with real-world agentic utilization, we additionally introduce managed noise into the analysis prompts.
This consists of components reminiscent of:
- Typographical errors in named entities (e.g., “Andrwe Carnegie” as an alternative of “Andrew Carnegie”),
- Colloquial phrasing or casual language (e.g., “present me what’s up with Tesla’s board” as an alternative of “record members of Tesla’s board of administrators”),
- Overly broad or under-specified intents that require follow-up reasoning or clarification.
These variations mirror how customers really work together with brokers in observe. In actual deployments, brokers should deal with messy inputs, incomplete formulations, and conversational shorthand, that are circumstances not often captured by clear, canonical benchmarks.
To raised mirror these insights round evaluating agentic approaches, I’ve created a brand new benchmark utilizing Claude 4.0. Not like conventional benchmarks that concentrate on Cypher question correctness, this one is designed to evaluate the standard of the last solutions produced by multi-step brokers
Databases
To make sure quite a lot of evaluations, we use a few completely different databases which are accessible on the Neo4j demo server. Examples embody:
MCP-Neo4j-Cypher server
mcp-neo4j-cypher is a ready-to-use MCP device interface that enables brokers to work together with Neo4j via pure language. It helps three core features: viewing the graph schema, operating Cypher queries to learn information, and executing write operations to replace the database. Outcomes are returned in a clear, structured format that brokers can simply perceive and use.
It really works out of the field with any framework that helps MCP servers, making it easy to plug into present agent setups with out further integration work. Whether or not you’re constructing a chatbot, information assistant, or customized workflow, this device lets your agent safely and intelligently work with graph information.
Benchmark
Lastly, let’s run the benchmark analysis.
We used LangChain to host the agent and join it to the mcp-neo4j-cypher server, which is the one device offered to the agent. This setup makes the analysis easy and practical: the agent should rely fully on pure language interplay with the MCP interface to retrieve and manipulate graph information.
For the analysis, we used Claude 3.7 Sonnet because the agent and GPT-4o Mini because the choose.
The benchmark dataset consists of roughly 200 pure language question-answer pairs, categorized by variety of hops (1-hop, 2-hop, and so forth.) and whether or not the queries comprise distracting or noisy data. This construction helps assess the agent’s reasoning accuracy and robustness in each clear and noisy contexts. The analysis code is offered on GitHub.
Let’s look at the outcomes collectively.
The analysis exhibits that an agent utilizing solely the mcp-neo4j-cypher interface can successfully reply advanced pure language questions over graph information. Throughout a benchmark of round 200 questions, the agent achieved a median rating of 0.71, with efficiency dropping as query complexity elevated. The presence of noise within the enter considerably decreased accuracy, revealing the agent’s sensitivity to typos in named entities and such.
On the device utilization aspect, the agent averaged 3.6 device calls per query. That is in line with the present requirement to make not less than one name to fetch the schema and one other to execute the principle Cypher question. Most queries fell inside a 2–4 name vary, exhibiting the agent’s capability to motive and act effectively. Notably, a small variety of questions have been answered with only one and even zero device calls, anomalies which will recommend early stopping, incorrect planning, or agent bugs, and are value additional evaluation. Trying forward, device rely could possibly be decreased additional if schema entry is embedded immediately by way of MCP assets, eliminating the necessity for an specific schema fetch step.
The true worth of getting a benchmark is that it opens the door to systematic iteration. As soon as baseline efficiency is established, you can begin tweaking parameters, observing their affect, and making focused enhancements. As an example, if agent execution is expensive, you may need to check whether or not capping the variety of allowed steps to 10 utilizing a LangGraph recursion restrict has a measurable impact on accuracy. With the benchmark in place, these trade-offs between efficiency and effectivity might be explored quantitatively reasonably than guessed.
With a 10-step restrict in place, efficiency dropped noticeably. The imply analysis rating fell to 0.535. Accuracy decreased sharply on extra advanced (3-hop+) questions, suggesting the step restrict minimize off deeper reasoning chains. Noise continued to degrade efficiency, with noisy questions averaging decrease scores than clear ones.
Abstract
We’re dwelling in an thrilling second for AI, with the rise of autonomous brokers and rising requirements like MCP dramatically increasing what LLMs can do, particularly on the subject of structured, multi-step duties. However whereas the capabilities are rising quick, strong analysis continues to be lagging behind. That’s the place this GRAPE mission is available in.
The objective is to construct a sensible, evolving benchmark for graph-based query answering utilizing the MCP interface. Over time, I plan to refine the dataset, experiment with completely different retrieval methods, and discover the right way to lengthen or adapt the Cypher MCP for higher accuracy. There’s nonetheless plenty of work forward from cleansing information, bettering retrieval to tightening analysis. Nonetheless, having a transparent benchmark means we are able to monitor progress meaningfully, check concepts systematically, and push the boundaries of what these brokers can reliably do.
