, authorized contracts are foundational paperwork that outline the relationships, obligations, and obligations between events. Whether or not it’s a partnership settlement, an NDA, or a provider contract, these paperwork usually include crucial data that drives decision-making, threat administration, and compliance. Nevertheless, navigating and extracting insights from these contracts could be a advanced and time-consuming course of.
On this submit, we’ll discover how we are able to streamline the method of understanding and dealing with authorized contracts by implementing an end-to-end resolution utilizing Agentic Graphrag. I see GraphRAG as an umbrella time period for any technique that retrieves or causes over data saved in a data graph, enabling extra structured and context-aware responses.
By structuring authorized contracts right into a data graph in Neo4j, we are able to create a strong repository of data that’s simple to question and analyze. From there, we’ll construct a LangGraph agent that permits customers to ask particular questions concerning the contracts, making it potential to quickly uncover new insights.
The code is accessible on this GitHub repository.
Why structuring information issues
Some domains work effectively with naive RAG, however authorized contracts current distinctive challenges.
As proven within the picture, relying solely on a vector index to retrieve related chunks can introduce dangers, akin to pulling data from irrelevant contracts. It is because authorized language is extremely structured, and comparable wording throughout totally different agreements can result in incorrect or deceptive retrieval. These limitations spotlight the necessity for a extra structured method, akin to GraphRAG, to make sure exact and context-aware retrieval.
To implement GraphRAG, we first must assemble a data graph.
To construct a data graph for authorized contracts, we’d like a method to extract structured data from paperwork and retailer it alongside the uncooked textual content. An LLM can assist by studying by way of contracts and figuring out key particulars akin to events, dates, contract varieties, and vital clauses. As an alternative of treating the contract as only a block of textual content, we break it down into structured parts that mirror its underlying authorized which means. For instance, an LLM can acknowledge that “ACME Inc. agrees to pay $10,000 per thirty days beginning January 1, 2024” comprises each a cost obligation and a begin date, which we are able to then retailer in a structured format.
As soon as we’ve got this structured information, we retailer it in a data graph, the place entities like corporations, agreements, and clauses are represented as represented together with their relationships. The unstructured textual content stays accessible, however now we are able to use the structured layer to refine our searches and make retrieval much more exact. As an alternative of simply fetching essentially the most related textual content chunks, we are able to filter contracts primarily based on their attributes. This implies we are able to reply questions that naive RAG would wrestle with, akin to what number of contracts had been signed final month or whether or not we’ve got any energetic agreements with a particular firm. These questions require aggregation and filtering, which isn’t potential with customary vector-based retrieval alone.
By combining structured and unstructured information, we additionally make retrieval extra context-aware. If a person asks a few contract’s cost phrases, we be certain that the search is constrained to the proper settlement slightly than counting on textual content similarity, which could pull in phrases from unrelated contracts. This hybrid method overcomes the restrictions of naive RAG and permits for a a lot deeper and extra dependable evaluation of authorized paperwork.
Graph building
We’ll leverage an LLM to extract structured data from authorized paperwork, utilizing the CUAD (Contract Understanding Atticus Dataset), a broadly used benchmark dataset for contract evaluation licensed below CC BY 4.0. CUAD dataset comprises over 500 contracts, making it a really perfect dataset for evaluating our structured extraction pipeline.
The token depend distribution for the contracts is visualized beneath.
Most contracts on this dataset are comparatively brief, with token counts beneath 10,000. Nevertheless, there are some for much longer contracts, with just a few reaching as much as 80,000 tokens. These lengthy contracts are uncommon, whereas shorter ones make up the bulk. The distribution exhibits a steep drop-off, which means lengthy contracts are the exception slightly than the rule.
We’re utilizing Gemini-2.0-Flash for extraction, which has a 1 million token enter restrict, so dealing with these contracts isn’t an issue. Even the longest contracts in our dataset (round 80,000 tokens) match effectively inside the mannequin’s capability. Since most contracts are a lot shorter, we don’t have to fret about truncation or breaking paperwork into smaller chunks for processing.
Structured information extraction
Most business LLMs have the choice to make use of Pydantic objects to outline the schema of the output. An instance for location:
class Location(BaseModel):
"""
Represents a bodily location together with deal with, metropolis, state, and nation.
"""
deal with: Non-obligatory[str] = Discipline(
..., description="The road deal with of the placement.Use None if not supplied"
)
metropolis: Non-obligatory[str] = Discipline(
..., description="Town of the placement.Use None if not supplied"
)
state: Non-obligatory[str] = Discipline(
..., description="The state or area of the placement.Use None if not supplied"
)
nation: str = Discipline(
...,
description="The nation of the placement. Use the two-letter ISO customary.",
)When utilizing LLMs for structured output, Pydantic helps outline a transparent schema by specifying the forms of attributes and offering descriptions that information the mannequin’s responses. Every area has a kind, akin to str or Non-obligatory[str], and an outline that tells the LLM precisely learn how to format the output.
For instance, in a Location mannequin, we outline key attributes like deal with, metropolis, state, and nation, specifying what information is predicted and the way it ought to be structured. The nation area, for example, follows two-letter nation code customary like "US", "FR", or "JP", as an alternative of inconsistent variations like “United States” or “USA.” This precept applies to different structured information as effectively, ISO 8601 retains dates in a typical format (YYYY-MM-DD), and so forth.
By defining structured output with Pydantic, we make LLM responses extra dependable, machine-readable, and simpler to combine into databases or APIs. Clear area descriptions additional assist the mannequin generate accurately formatted information, decreasing the necessity for post-processing.
The Pydantic schema fashions could be extra refined just like the Contract mannequin beneath, which captures key particulars of a authorized settlement, guaranteeing the extracted information follows a standardized construction.
class Contract(BaseModel):
"""
Represents the important thing particulars of the contract.
"""
abstract: str = Discipline(
...,
description=("Excessive stage abstract of the contract with related details and particulars. Embrace all related data to offer full image."
"Do no use any pronouns"),
)
contract_type: str = Discipline(
...,
description="The kind of contract being entered into.",
enum=CONTRACT_TYPES,
)
events: Record[Organization] = Discipline(
...,
description="Record of events concerned within the contract, with particulars of every occasion's position.",
)
effective_date: str = Discipline(
...,
description=(
"Enter the date when the contract turns into efficient in yyyy-MM-dd format."
"If solely the yr (e.g., 2015) is understood, use 2015-01-01 because the default date."
"All the time fill in full date"
),
)
contract_scope: str = Discipline(
...,
description="Description of the scope of the contract, together with rights, duties, and any limitations.",
)
period: Non-obligatory[str] = Discipline(
None,
description=(
"The period of the settlement, together with provisions for renewal or termination."
"Use ISO 8601 durations customary"
),
)
end_date: Non-obligatory[str] = Discipline(
None,
description=(
"The date when the contract expires. Use yyyy-MM-dd format."
"If solely the yr (e.g., 2015) is understood, use 2015-01-01 because the default date."
"All the time fill in full date"
),
)
total_amount: Non-obligatory[float] = Discipline(
None, description="Complete worth of the contract."
)
governing_law: Non-obligatory[Location] = Discipline(
None, description="The jurisdiction's legal guidelines governing the contract."
)
clauses: Non-obligatory[List[Clause]] = Discipline(
None, description=f"""Related summaries of clause varieties. Allowed clause varieties are {CLAUSE_TYPES}"""
)This contract schema organizes key particulars of authorized agreements in a structured manner, making it simpler to research with LLMs. It consists of various kinds of clauses, akin to confidentiality or termination, every with a brief abstract. The events concerned are listed with their names, places, and roles, whereas contract particulars cowl issues like begin and finish dates, whole worth, and governing legislation. Some attributes, akin to governing legislation, could be outlined utilizing nested fashions, enabling extra detailed and sophisticated outputs.
The nested object method works effectively with some AI fashions that deal with advanced information relationships, whereas others might wrestle with deeply nested particulars.
We will take a look at our method utilizing the next instance. We’re utilizing the LangChain framework to orchestrate LLMs.
llm = ChatGoogleGenerativeAI(mannequin="gemini-2.0-flash")
llm.with_structured_output(Contract).invoke(
"Tomaz works with Neo4j since 2017 and can make a billion greenback till 2030."
"The contract was signed in Las Vegas"
)which outputs
Contract(
abstract="Tomaz works with Neo4j since 2017 and can make a billion greenback till 2030.",
contract_type="Service",
events=[
Organization(
name="Tomaz",
location=Location(
address=None,
city="Las Vegas",
state=None,
country="US"
),
role="employee"
),
Organization(
name="Neo4j",
location=Location(
address=None,
city=None,
state=None,
country="US"
),
role="employer"
)
],
effective_date="2017-01-01",
contract_scope="Tomaz will work with Neo4j",
period=None,
end_date="2030-01-01",
total_amount=1_000_000_000.0,
governing_law=None,
clauses=None
)Now that our contract information is in a structured format, we are able to outline the Cypher question wanted to import it into Neo4j, mapping entities, relationships, and key clauses right into a graph construction. This step transforms uncooked extracted information right into a queryable data graph, enabling environment friendly traversal and retrieval of contract insights.
UNWIND $information AS row
MERGE (c:Contract {file_id: row.file_id})
SET c.abstract = row.abstract,
c.contract_type = row.contract_type,
c.effective_date = date(row.effective_date),
c.contract_scope = row.contract_scope,
c.period = row.period,
c.end_date = CASE WHEN row.end_date IS NOT NULL THEN date(row.end_date) ELSE NULL END,
c.total_amount = row.total_amount
WITH c, row
CALL (c, row) {
WITH c, row
WHERE row.governing_law IS NOT NULL
MERGE (c)-[:HAS_GOVERNING_LAW]->(l:Location)
SET l += row.governing_law
}
FOREACH (occasion IN row.events |
MERGE (p:Social gathering {identify: occasion.identify})
MERGE (p)-[:HAS_LOCATION]->(pl:Location)
SET pl += occasion.location
MERGE (p)-[pr:PARTY_TO]->(c)
SET pr.position = occasion.position
)
FOREACH (clause IN row.clauses |
MERGE (c)-[:HAS_CLAUSE]->(cl:Clause {kind: clause.clause_type})
SET cl.abstract = clause.abstract
)This Cypher question imports structured contract information into Neo4j by creating Contract nodes with attributes akin to abstract, contract_type, effective_date, period, and total_amount. If a governing legislation is specified, it hyperlinks the contract to a Location node. Events concerned within the contract are saved as Social gathering nodes, with every occasion related to a Location and assigned a job in relation to the contract. The question additionally processes clauses, creating Clause nodes and linking them to the contract whereas storing their kind and abstract.
After processing and importing the contracts, the ensuing graph follows the next graph schema.
Let’s additionally check out a single contract.
This graph represents a contract construction the place a contract (orange node) connects to varied clauses (purple nodes), events (blue nodes), and places (violet nodes). The contract has three clauses: Renewal & Termination, Legal responsibility & Indemnification, and Confidentiality & Non-Disclosure. Two events, Modus Media Worldwide and Dragon Methods, Inc., are concerned, every linked to their respective places, Netherlands (NL) and United States (US). The contract is ruled by U.S. legislation. The contract node additionally comprises extra metadata, together with dates and different related particulars.
A public read-only occasion containing CUAD authorized contracts is accessible with the next credentials.
URI: neo4j+s://demo.neo4jlabs.com
username: legalcontracts
password: legalcontracts
database: legalcontractsEntity decision
Entity decision in authorized contracts is difficult as a consequence of variations in how corporations, people, and places are referenced. An organization may seem as “Acme Inc.” in a single contract and “Acme Company” in one other, requiring a course of to find out whether or not they discuss with the identical entity.
One method is to generate candidate matches utilizing textual content embeddings or string distance metrics like Levenshtein distance. Embeddings seize semantic similarity, whereas string distance measures character-level variations. As soon as candidates are recognized, extra analysis is required, evaluating metadata akin to addresses or tax IDs, analyzing shared relationships within the graph, or incorporating human assessment for crucial instances.
For resolving entities at scale, each open-source options like Dedupe and business instruments like Senzing supply automated strategies. Selecting the best method relies on information high quality, accuracy necessities, and whether or not guide oversight is possible.
With the authorized graph constructed, we are able to transfer onto the agentic GraphRAG implementation.
Agentic GraphRAG
Agentic architectures differ broadly in complexity, modularity, and reasoning capabilities. At their core, these architectures contain an LLM appearing as a central reasoning engine, usually supplemented with instruments, reminiscence, and orchestration mechanisms. The important thing differentiator is how a lot autonomy the LLM has in making choices and the way interactions with exterior techniques are structured.
One of many easiest and simplest designs, notably for chatbot-like implementations, is a direct LLM-with-tools method. On this setup, the LLM serves because the decision-maker, dynamically choosing which instruments to invoke (if any), retrying operations when mandatory, and executing a number of instruments in sequence to meet advanced requests.
The diagram represents a easy LangGraph agent workflow. It begins at __start__, shifting to the assistant node, the place the LLM processes person enter. From there, the assistant can both name instruments to fetch related data or transition on to __end__ to finish the interplay. If a instrument is used, the assistant processes the response earlier than deciding whether or not to name one other instrument or finish the session. This construction permits the agent to autonomously decide when exterior data is required earlier than responding.
This method is especially well-suited to stronger business fashions like Gemini or GPT-4o, which excel at reasoning and self-correction.
Instruments
LLMs are highly effective reasoning engines, however their effectiveness usually relies on how effectively they’re geared up with exterior instruments. These instruments , whether or not database queries, APIs, or search capabilities, prolong an LLM’s potential to retrieve details, carry out calculations, or work together with structured information.
Designing instruments which can be each basic sufficient to deal with various queries and exact sufficient to return significant outcomes is extra artwork than science. What we’re actually constructing is a semantic layer between the LLM and the underlying information. Moderately than requiring the LLM to know the precise construction of a Neo4j data graph or a database schema, we outline instruments that summary away these complexities.
With this method, the LLM doesn’t must know whether or not contract data is saved as graph nodes and relationships or as uncooked textual content in a doc retailer. It solely must invoke the proper instrument to fetch related information primarily based on a person’s query.
In our case, the contract retrieval instrument serves as this semantic interface. When a person asks about contract phrases, obligations, or events, the LLM calls a structured question instrument that interprets the request right into a database question, retrieves related data, and presents it in a format the LLM can interpret and summarize. This allows a versatile, model-agnostic system the place totally different LLMs can work together with contract information without having direct data of its storage or construction.
There’s no one-size-fits-all customary for designing an optimum toolset. What works effectively for one mannequin might fail for an additional. Some fashions deal with ambiguous instrument directions gracefully, whereas others wrestle with advanced parameters or require express prompting. The trade-off between generality and task-specific effectivity means instrument design requires iteration, testing, and fine-tuning for the LLM in use.
For contract evaluation, an efficient instrument ought to retrieve contracts and summarize key phrases with out requiring customers to phrase queries rigidly. Reaching this flexibility relies on considerate immediate engineering, strong schema design, and adaptation to totally different LLM capabilities. As fashions evolve, so do methods for making instruments extra intuitive and efficient.
On this part, we’ll discover totally different approaches to instrument implementation, evaluating their flexibility, effectiveness, and compatibility with numerous LLMs.
My most well-liked method is to dynamically and deterministically assemble a Cypher question and execute it in opposition to the database. This technique ensures constant and predictable question era whereas sustaining implementation flexibility. By structuring queries this manner, we reinforce the semantic layer, permitting person inputs to be seamlessly translated into database retrievals. This retains the LLM centered on retrieving related data slightly than understanding the underlying information mannequin.
Our instrument is meant to establish related contracts, so we have to present the LLM with choices to go looking contracts primarily based on numerous attributes. The enter description is once more supplied as a Pydantic object.
class ContractInput(BaseModel):
min_effective_date: Non-obligatory[str] = Discipline(
None, description="Earliest contract efficient date (YYYY-MM-DD)"
)
max_effective_date: Non-obligatory[str] = Discipline(
None, description="Newest contract efficient date (YYYY-MM-DD)"
)
min_end_date: Non-obligatory[str] = Discipline(
None, description="Earliest contract finish date (YYYY-MM-DD)"
)
max_end_date: Non-obligatory[str] = Discipline(
None, description="Newest contract finish date (YYYY-MM-DD)"
)
contract_type: Non-obligatory[str] = Discipline(
None, description=f"Contract kind; legitimate varieties: {CONTRACT_TYPES}"
)
events: Non-obligatory[List[str]] = Discipline(
None, description="Record of events concerned within the contract"
)
summary_search: Non-obligatory[str] = Discipline(
None, description="Examine abstract of the contract"
)
nation: Non-obligatory[str] = Discipline(
None, description="Nation the place the contract applies. Use the two-letter ISO customary."
)
energetic: Non-obligatory[bool] = Discipline(None, description="Whether or not the contract is energetic")
monetary_value: Non-obligatory[MonetaryValue] = Discipline(
None, description="The whole quantity or worth of a contract"
)With LLM instruments, attributes can take numerous varieties relying on their function. Some fields are easy strings, akin to contract_type and nation, which retailer single values. Others, like events, are lists of strings, permitting a number of entries (e.g., a number of entities concerned in a contract).
Past fundamental information varieties, attributes can even signify advanced objects. For instance, monetary_value makes use of a MonetaryValue object, which incorporates structured information akin to forex kind and the operator. Whereas attributes with nested objects supply a transparent and structured illustration of information, fashions are likely to wrestle to deal with them successfully, so we must always preserve them easy.
As a part of this venture, we’re experimenting with a further cypher_aggregation attribute, offering the LLM with higher flexibility for eventualities that require particular filtering or aggregation.
cypher_aggregation: Non-obligatory[str] = Discipline(
None,
description="""Customized Cypher assertion for superior aggregations and analytics.
This can be appended to the bottom question:
```
MATCH (c:Contract)
WITH c, abstract, contract_type, contract_scope, effective_date, end_date, events, energetic, monetary_value, contract_id, nations
```
Examples:
1. Depend contracts by kind:
```
RETURN contract_type, depend(*) AS depend ORDER BY depend DESC
```
2. Calculate common contract period by kind:
```
WITH contract_type, effective_date, end_date
WHERE effective_date IS NOT NULL AND end_date IS NOT NULL
WITH contract_type, period.between(effective_date, end_date).days AS period
RETURN contract_type, avg(period) AS avg_duration ORDER BY avg_duration DESC
```
3. Calculate contracts per efficient date yr:
```
RETURN effective_date.yr AS yr, depend(*) AS depend ORDER BY yr
```
4. Counts the occasion with the very best variety of energetic contracts:
```
UNWIND events AS occasion
WITH occasion.identify AS party_name, energetic, depend(*) AS contract_count
WHERE energetic = true
RETURN party_name, contract_count
ORDER BY contract_count DESC
LIMIT 1
```
""" The cypher_aggregation attribute permits LLMs to outline customized Cypher statements for superior aggregations and analytics. It extends the bottom question by appending question-specified aggregation logic, enabling versatile filtering and computation.
This characteristic helps use instances akin to counting contracts by kind, calculating common contract period, analyzing contract distributions over time, and figuring out key events primarily based on contract exercise. By leveraging this attribute, the LLM can dynamically generate insights tailor-made to particular analytical wants with out requiring predefined question constructions.
Whereas this flexibility is efficacious, it ought to be fastidiously evaluated, as elevated adaptability comes at the price of diminished consistency and robustness as a result of added complexity of the operation.
We should clearly outline the perform’s identify and outline when presenting it to the LLM. A well-structured description helps information the mannequin in utilizing the perform accurately, guaranteeing it understands its function, anticipated inputs, and outputs. This reduces ambiguity and improves the LLM’s potential to generate significant and dependable queries.
class ContractSearchTool(BaseTool):
identify: str = "ContractSearch"
description: str = (
"helpful for when it's good to reply questions associated to any contracts"
)
args_schema: Kind[BaseModel] = ContractInputLastly, we have to implement a perform that processes the given inputs, constructs the corresponding Cypher assertion, and executes it effectively.
The core logic of the perform facilities on developing the Cypher assertion. We start by matching the contract as the inspiration of the question.
cypher_statement = "MATCH (c:Contract) "Subsequent, we have to implement the perform that processes the enter parameters. On this instance, we primarily use attributes to filter contracts primarily based on the given standards.
Easy property filtering
For instance, the contract_type attribute is used to carry out easy node property filtering.
if contract_type:
filters.append("c.contract_type = $contract_type")
params["contract_type"] = contract_typeThis code provides a Cypher filter for contract_type whereas utilizing question parameters for values to forestall question injection safety situation.
For the reason that potential contract kind values are introduced within the attribute description
contract_type: Non-obligatory[str] = Discipline(
None, description=f"Contract kind; legitimate varieties: {CONTRACT_TYPES}"
)we don’t have to fret about mapping values from enter to legitimate contract varieties because the LLM will deal with that.
Inferred property filtering
We’re constructing instruments for an LLM to work together with a data graph, the place the instruments function an abstraction layer over structured queries. A key characteristic is the flexibility to make use of inferred properties at runtime, much like an ontology however dynamically computed.
if energetic will not be None:
operator = ">=" if energetic else "<"
filters.append(f"c.end_date {operator} date()")Right here, energetic acts as a runtime classification, figuring out whether or not a contract is ongoing (>= date()) or expired (< date()). This logic extends structured KG queries by computing properties solely when wanted, enabling extra versatile LLM reasoning. By dealing with logic like this inside instruments, we make sure the LLM interacts with simplified, intuitive operations, conserving it centered on reasoning slightly than question formulation.
Neighbor filtering
Generally filtering relies on neighboring nodes, akin to limiting outcomes to contracts involving particular events. The events attribute is an non-obligatory record, and when supplied, it ensures solely contracts linked to these entities are thought-about:
if events:
parties_filter = []
for i, occasion in enumerate(events):
party_param_name = f"party_{i}"
parties_filter.append(
f"""EXISTS {{
MATCH (c)<-[:PARTY_TO]-(occasion)
WHERE toLower(occasion.identify) CONTAINS ${party_param_name}
}}"""
)
params[party_param_name] = occasion.decrease()This code filters contracts primarily based on their related events, treating the logic as AND, which means all specified circumstances have to be met for a contract to be included. It iterates by way of the supplied events record and constructs a question the place every occasion situation should maintain.
For every occasion, a singular parameter identify is generated to keep away from conflicts. The EXISTS clause ensures that the contract has a PARTY_TO relationship to a celebration whose identify comprises the required worth. The identify is transformed to lowercase to permit case-insensitive matching. Every occasion situation is added individually, implementing an implicit AND between them.
If extra advanced logic had been wanted, akin to supporting OR circumstances or permitting totally different matching standards, the enter would want to alter. As an alternative of a easy record of occasion names, a structured enter format specifying operators can be required.
Moreover, we may implement a party-matching technique that tolerates minor typos, enhancing the person expertise by dealing with variations in spelling and formatting.
Customized operator filtering
So as to add extra flexibility, we are able to introduce an operator object as a nested attribute, permitting extra management over filtering logic. As an alternative of hardcoding comparisons, we outline an enumeration for operators and use it dynamically.
For instance, with financial values, a contract may have to be filtered primarily based on whether or not its whole quantity is bigger than, lower than, or precisely equal to a specified worth. As an alternative of assuming a set comparability logic, we outline an enum that represents the potential operators:
class NumberOperator(str, Enum):
EQUALS = "="
GREATER_THAN = ">"
LESS_THAN = "<"
class MonetaryValue(BaseModel):
"""The whole quantity or worth of a contract"""
worth: float
operator: NumberOperator
if monetary_value:
filters.append(f"c.total_amount {monetary_value.operator.worth} $total_value")
params["total_value"] = monetary_value.worthThis method makes the system extra expressive. As an alternative of inflexible filtering guidelines, the instrument interface permits the LLM to specify not only a worth however the way it ought to be in contrast, making it simpler to deal with a broader vary of queries whereas conserving the LLM’s interplay easy and declarative.
Some LLMs wrestle with nested objects as inputs, making it more durable to deal with structured operator-based filtering. Including a between operator introduces extra complexity because it requires two separate values, which might result in ambiguity in parsing and enter validation.
Min and Max attributes
To maintain issues less complicated, I are likely to gravitate towards utilizing min and max attributes for dates, as this naturally helps vary filtering and makes the between logic easy.
if min_effective_date:
filters.append("c.effective_date >= date($min_effective_date)")
params["min_effective_date"] = min_effective_date
if max_effective_date:
filters.append("c.effective_date <= date($max_effective_date)")
params["max_effective_date"] = max_effective_dateThis perform filters contracts primarily based on an efficient date vary by including an non-obligatory decrease and higher certain situation when min_effective_date and max_effective_date are supplied, guaranteeing that solely contracts inside the specified date vary are included.
Semantic search
An attribute can be used for semantic search, the place as an alternative of counting on a vector index upfront, we use a post-filtering method to metadata filtering. First, structured filters, like date ranges, financial values, or events, are utilized to slender down the candidate set. Then, vector search is carried out over this filtered subset to rank outcomes primarily based on semantic similarity.
if summary_search:
cypher_statement += (
"WITH c, vector.similarity.cosine(c.embedding, $embedding) "
"AS rating ORDER BY rating DESC WITH c, rating WHERE rating > 0.9 "
) # Outline a threshold restrict
params["embedding"] = embeddings.embed_query(summary_search)
else: # Else we type by newest
cypher_statement += "WITH c ORDER BY c.effective_date DESC "This code applies semantic search when summary_search is supplied by computing cosine similarity between the contract’s embedding and the question embedding, ordering outcomes by relevance, and filtering out low-scoring matches with a threshold of 0.9. In any other case, it defaults to sorting contracts by the latest effective_date.
Dynamic queries
The cypher aggregation attribute is an experiment I wished to check that provides the LLM a level of partial text2cypher functionality, permitting it to dynamically generate aggregations after the preliminary structured filtering. As an alternative of predefining each potential aggregation, this method lets the LLM specify calculations like counts, averages, or grouped summaries on demand, making queries extra versatile and expressive. Nevertheless, since this shifts extra question logic to the LLM, guaranteeing all generated queries work accurately turns into difficult, as malformed or incompatible Cypher statements can break execution. This trade-off between flexibility and reliability is a key consideration in designing the system.
if cypher_aggregation:
cypher_statement += """WITH c, c.abstract AS abstract, c.contract_type AS contract_type,
c.contract_scope AS contract_scope, c.effective_date AS effective_date, c.end_date AS end_date,
[(c)<-[r:PARTY_TO]-(occasion) | {occasion: occasion.identify, position: r.position}] AS events, c.end_date >= date() AS energetic, c.total_amount as monetary_value, c.file_id AS contract_id,
apoc.coll.toSet([(c)<-[:PARTY_TO]-(occasion)-[:LOCATED_IN]->(nation) | nation.identify]) AS nations """
cypher_statement += cypher_aggregationIf no cypher aggregation is supplied, we return the overall depend of recognized contracts together with solely 5 instance contracts to keep away from overwhelming the immediate. Dealing with extreme rows is essential, as an LLM scuffling with a large consequence set isn’t helpful. Moreover, LLM producing solutions with 100 contract titles isn’t a superb person expertise both.
cypher_statement += """WITH gather(c) AS nodes
RETURN {
total_count_of_contracts: measurement(nodes),
example_values: [
el in nodes[..5] |
{abstract:el.abstract, contract_type:el.contract_type,
contract_scope: el.contract_scope, file_id: el.file_id,
effective_date: el.effective_date, end_date: el.end_date,
monetary_value: el.total_amount, contract_id: el.file_id,
events: [(el)<-[r:PARTY_TO]-(occasion) | {identify: occasion.identify, position: r.position}],
nations: apoc.coll.toSet([(el)<-[:PARTY_TO]-()-[:LOCATED_IN]->(nation) | nation.identify])}
]
} AS output"""This cypher assertion collects all matching contracts into a listing, returning the overall depend and as much as 5 instance contracts with key attributes, together with abstract, kind, scope, dates, financial worth, related events with roles, and distinctive nation places.
Now that our contract search instrument is constructed, we hand it off to the LLM and similar to that, we’ve got agentic GraphRAG carried out.
Agent Benchmark
In the event you’re severe about implementing agentic GraphRAG, you want an analysis dataset, not simply as a benchmark however as a basis for your complete venture. A well-constructed dataset helps outline the scope of what the system ought to deal with, guaranteeing that preliminary growth aligns with real-world use instances. Past that, it turns into a useful instrument for evaluating efficiency, permitting you to measure how effectively the LLM interacts with the graph, retrieves data, and applies reasoning. It’s additionally important for immediate engineering optimizations, letting you iteratively refine queries, instrument use, and response formatting with clear suggestions slightly than guesswork. With no structured dataset, you’re flying blind, making enhancements more durable to quantify and inconsistencies harder to catch.
The code for the benchmark is accessible on GitHub.
I’ve compiled a listing of twenty-two questions which we’ll use to guage the system. Moreover, we’re going to introduce a brand new metric referred to as answer_satisfaction the place we can be present a customized immediate.
answer_satisfaction = AspectCritic(
identify="answer_satisfaction",
definition="""You'll consider an ANSWER to a authorized QUESTION primarily based on a supplied SOLUTION.
Price the reply on a scale from 0 to 1, the place:
- 0 = incorrect, considerably incomplete, or deceptive
- 1 = right and sufficiently full
Contemplate these analysis standards:
1. Factual correctness is paramount - the reply should not contradict the answer
2. The reply should deal with the core parts of the answer
3. Further related data past the answer is suitable and should improve the reply
4. Technical authorized terminology ought to be used appropriately if current within the resolution
5. For quantitative authorized analyses, correct figures have to be supplied
+ fewshots
"""Many questions can return a considerable amount of data. For instance, asking for contracts signed earlier than 2020 may yield lots of of outcomes. For the reason that LLM receives each the overall depend and some instance entries, our analysis ought to deal with the overall depend, slightly than which particular examples the LLM chooses to point out.
The supplied outcomes point out that every one evaluated fashions (Gemini 1.5 Professional, Gemini 2.0 Flash, and GPT-4o) carry out equally effectively for many instrument calls, with GPT-4o barely outperforming the Gemini fashions (0.82 vs. 0.77). The noticeable distinction emerges primarily when partial text2cypher is used, notably for numerous aggregation operations.
Notice that that is solely 22 pretty easy questions, so we didn’t actually discover reasoning capabilities of LLMs.
Moreover, I’ve seen tasks the place accuracy could be improved considerably by leveraging Python for aggregations, as LLMs sometimes deal with Python code era and execution higher than producing advanced Cypher queries immediately.
Net Software
We’ve additionally constructed a easy React net utility, powered by LangGraph hosted on FastAPI, which streams responses on to the frontend. Particular due to Anej Gorkic for creating the net app.
You possibly can launch your complete stack with the next command:
docker compose upAnd navigate to localhost:5173
Abstract
As LLMs acquire stronger reasoning capabilities, they, when paired with the proper instruments, can grow to be highly effective brokers for navigating advanced domains like authorized contracts. On this submit, we’ve solely scratched the floor, specializing in core contract attributes whereas barely touching the wealthy number of clauses present in real-world agreements. There’s vital room for progress, from increasing clause protection to refining instrument design and interplay methods.
The code is accessible on GitHub.
Photos
All pictures on this submit had been created by the writer.
