Graph RAG vs SQL RAG

I in each a graph database and a SQL database, then used numerous massive language fashions (LLMs) to reply questions in regards to the knowledge by way of a retrieval-augmented technology (RAG) strategy. By utilizing the identical dataset and questions throughout each techniques, I evaluated which database paradigm delivers extra correct and insightful outcomes.

Retrieval-Augmented Era (RAG) is an AI framework that enhances massive language fashions (LLMs) by letting them retrieve related exterior data earlier than producing a solution. As an alternative of relying solely on what the mannequin was skilled on, RAG dynamically queries a data supply (on this article a SQL or graph database) and integrates these outcomes into its response. An introduction to RAG will be discovered right here.

SQL databases set up knowledge into tables made up of rows and columns. Every row represents a document, and every column represents an attribute. Relationships between tables are outlined utilizing keys and joins, and all knowledge follows a hard and fast schema. SQL databases are perfect for structured, transactional knowledge the place consistency and precision are necessary — for instance, finance, stock, or affected person information.

Graph databases retailer knowledge as nodes (entities) and edges (relationships) with optionally available properties hooked up to each. As an alternative of becoming a member of tables, they straight symbolize relationships, permitting for quick traversal throughout related knowledge. Graph databases are perfect for modelling networks and relationships — similar to social graphs, data graphs, or molecular interplay maps — the place connections are as necessary because the entities themselves.

Knowledge

The dataset I used to check the efficiency of RAGs comprises Method 1 outcomes from 1950 to 2024. It contains detailed outcomes at races of drivers and constructors (groups) protecting qualifying, dash race, predominant race, and even lap occasions and pit cease occasions. The standings of the drivers and constructors’ championships after each race are additionally included.

SQL Schema

This dataset is already structured in tables with keys so {that a} SQL database will be simply arrange. The database’s schema is proven under:

Races is the central desk which is linked with all forms of outcomes in addition to further data like season and circuits. The outcomes tables are additionally linked with Drivers and Constructors tables to document their outcome at every race. The championship standings after every race are saved within the Driver_standings and Constructor_standings tables.

Graph Schema

The schema of the graph database is proven under:

As graph databases can retailer data in nodes and relationships it solely requires six nodes in comparison with 14 tables of the SQL database. The Automotive node is an intermediate node that’s used to mannequin {that a} driver drove a automobile of a constructor at a specific race. Since driver – constructor pairings are altering over time, this relationship must be outlined for every race. The race outcomes are saved within the relationships e.g. :RACED between Automotive and Race. Whereas the :STOOD_AFTER relationships include the motive force and constructor championship standings after every race.

Querying the Database

I used LangChain to construct a RAG chain for each database sorts that generates a question primarily based on a consumer query, runs the question, and converts the question outcome to a solution to the consumer. The code will be discovered on this repo. I outlined a generic system immediate that could possibly be used to generate queries of any SQL or graph database. The one knowledge particular data was included by inserting the auto-generated database schema into the immediate. The system prompts will be discovered right here.

Right here is an instance the best way to initialize the mannequin chain and ask the query: “What driver gained the 92 Grand Prix in Belgium?”

from langchain_community.utilities import SQLDatabase
from langchain_openai import ChatOpenAI
from qa_chain import GraphQAChain
from config import DATABASE_PATH

# hook up with database
connection_string = f"sqlite:///{DATABASE_PATH}"
db = SQLDatabase.from_uri(connection_string)

# initialize LLM
llm = ChatOpenAI(temperature=0, mannequin="gpt-5")

# initialize qa chain
chain = GraphQAChain(llm, db, db_type='SQL', verbose=True)

# ask a query
chain.invoke("What driver gained the 92 Grand Prix in Belgium?")

Which returns:

{'write_query': {'question': "SELECT d.forename, d.surname
FROM outcomes r
JOIN races ra ON ra.raceId = r.raceId
JOIN drivers d ON d.driverId = r.driverId
WHERE ra.12 months = 1992
AND ra.identify = 'Belgian Grand Prix'
AND r.positionOrder = 1
LIMIT 10;"}} 
{'execute_query': {'outcome': "[('Michael', 'Schumacher')]"}}
 {'generate_answer': {'reply': 'Michael Schumacher'}}

The SQL question joins the Outcomes, Races, and Drivers tables, selects the race on the 1992 Belgian Grand Prix and the motive force who completed first. The LLM transformed the 12 months 92 to 1992 and the race identify from “Grand Prix in Belgium” to “Belgian Grand Prix”. It derived these conversions from the database schema which included three pattern rows of every desk. The question result’s “Michael Schumacher” which the LLM returned as reply.

Analysis

Now the query I need to reply is that if an LLM is healthier in querying the SQL or the graph database. I outlined three issue ranges (straightforward, medium, and onerous) the place straightforward have been questions that could possibly be answered by querying knowledge from just one desk or node, medium have been questions which required one or two hyperlinks amongst tables or nodes and onerous questions required extra hyperlinks or subqueries. For every issue degree I outlined 5 questions. Moreover, I outlined 5 questions that might not be answered with knowledge from the database.

I answered every query with three LLM fashions (GPT-5, GPT-4, and GPT-3.5-turbo) to research if probably the most superior fashions are wanted or older and cheaper fashions may additionally create passable outcomes. If a mannequin gave the right reply, it acquired 1 level, if it replied that it couldn’t reply the query it acquired 0 factors, and in case it gave a improper reply it acquired -1 level. All questions and solutions are listed right here. Beneath are the scores of all fashions and database sorts:

It’s exceptional how extra superior fashions outperform less complicated fashions: GPT-3-turbo acquired about half the variety of questions improper, GPT-4 acquired 2 to three questions improper however couldn’t reply 6 to 7 questions, and GPT-5 acquired all besides one query appropriate. Easier fashions appear to carry out higher with a SQL than graph database whereas GPT-5 achieved the identical rating with both database.

The one query GPT-5 acquired improper utilizing the SQL database was “Which driver gained probably the most world championships?”. The reply “Lewis Hamilton, with 7 world championships” just isn’t appropriate as a result of Lewis Hamilton and Michael Schumacher gained 7 world championships. The generated SQL question aggregated the variety of championships by driver, sorted them in descending order and solely chosen the primary row whereas the motive force within the second row had the identical variety of championships.

Utilizing the graph database, the one query GPT-5 acquired improper was “Who gained the Method 2 championship in 2017?” which was answered with “Lewis Hamilton” (Lewis Hamilton gained the Method 1 however not Method 2 championship that 12 months). It is a difficult query as a result of the database solely comprises Method 1 however not Method 2 outcomes. The anticipated reply would have been to answer that this query couldn’t be answered primarily based on the offered knowledge. Nonetheless, contemplating that the system immediate didn’t include any particular details about the dataset it’s comprehensible that this query was not appropriately answered.

Apparently utilizing the SQL database GPT-5 gave the right reply “Charles Leclerc”. The generated SQL question solely searched the drivers desk for the identify “Charles Leclerc”. Right here the LLM will need to have acknowledged that the database doesn’t include Method 2 outcomes and answered this query from its widespread data. Though this led to the right reply on this case it may be harmful when the LLM just isn’t utilizing the offered knowledge to reply questions. One solution to cut back this threat could possibly be to explicitly state within the system immediate that the database should be the one supply to reply questions.

Conclusion

This comparability of RAG efficiency utilizing a Method 1 outcomes dataset reveals that the newest LLMs carry out exceptionally properly, producing extremely correct and contextually conscious solutions with none further immediate engineering. Whereas less complicated fashions wrestle, newer ones like GPT-5 deal with advanced queries with near-perfect precision. Importantly, there was no vital distinction in efficiency between the graph and SQL database approaches – customers can merely select the database paradigm that most closely fits the construction of their knowledge.

The dataset used right here serves solely as an illustrative instance; outcomes could differ when utilizing different datasets, particularly those who require specialised area data or entry to private knowledge sources. Total, these findings spotlight how far retrieval-augmented LLMs have superior in integrating structured knowledge with pure language reasoning.

If not said in any other case, all photos have been created by the writer.