Person: “What does the inexperienced highlighting imply on this doc?”
RAG system: “Inexperienced highlighted textual content is interpreted as configuration settings.”
the sorts of solutions we anticipate immediately from Retrieval-Augmented Technology (RAG) methods.
Over the previous few years, RAG has change into one of many central architectural constructing blocks for knowledge-based language fashions: As a substitute of relying completely on the information saved within the mannequin, RAG methods mix language fashions with exterior doc sources.
The time period was launched by Lewis et al. and describes an strategy that’s broadly used to cut back hallucinations, enhance the traceability of solutions, and allow language fashions to work with proprietary knowledge.
I wished to know why a system selects one particular reply as an alternative of a really comparable different. This resolution is usually made already on the retrieval stage, lengthy earlier than an LLM comes into play.
Because of this, I performed three experiments on this article to analyze how completely different chunk sizes (80, 220, 500) affect retrieval habits.
Desk of Contents
1 – Why Chunk Dimension Is Extra Than Only a Parameter
2 – How Does Chunk Dimension Affect the Stability of Retrieval Ends in Small RAG Methods?
3 – Minimal RAG System With out Output Technology
4 – Three Experiments: Chunk Dimension as a Variable
5 – Ultimate Ideas
1 – Why Chunk Dimension Is Extra Than Only a Parameter
In a typical RAG pipeline, paperwork are first break up into smaller textual content segments, embedded into vectors, and saved in an index. When a question is issued, semantically comparable textual content segments are retrieved after which processed into a solution. This last step is usually carried out together with a language mannequin.
Typical parts of a RAG system embrace:
- Doc preprocessing
- Chunking
- Embedding
- Vector index
- Retrieval logic
- Non-obligatory: Technology of the output
On this article, I deal with the retrieval step. This step is dependent upon a number of parameters:
- Alternative of the embedding mannequin:
The embedding mannequin determines how textual content is transformed into numerical vectors. Totally different fashions seize which means at completely different ranges of granularity and are educated on completely different aims. For instance, light-weight sentence-transformer fashions are sometimes enough for semantic search, whereas bigger fashions could seize extra nuance however include greater computational price. - Distance or similarity metric:
The space or similarity metric defines how the closeness between two vectors is measured. Frequent selections embrace cosine similarity, dot product or Euclidean distance. For normalized embeddings, cosine similarity is usually used - Variety of retrieved outcomes (Prime-k):
The variety of retrieved outcomes specifies what number of textual content segments are returned by the retrieval step. A small Prime-k can miss related context, whereas a big Prime-k will increase recall however could introduce noise. - Overlap between textual content segments:
Overlap defines how a lot textual content is shared between consecutive chunks. It’s usually used to keep away from shedding vital data at chunk boundaries. A small overlap reduces redundancy however dangers slicing explanations in half, whereas a bigger overlap will increase robustness at the price of storing and processing extra comparable chunks. - Chunk dimension:
Describes the dimensions of the textual content models which are extracted from a doc and saved as particular person vectors. Relying on the implementation, chunk dimension may be outlined primarily based on characters, phrases, or tokens. The dimensions determines how a lot context a single vector represents.
Small chunks comprise little or no context and are extremely particular. Giant chunks embrace extra surrounding data, however at a a lot coarser degree. Consequently, chunk dimension determines which components of the which means are literally in contrast when a question is matched towards a piece.
Chunk dimension implicitly displays assumptions about how a lot context is required to seize which means, how strongly data could also be fragmented, and the way clearly semantic similarity may be measured.
With this text, I wished to discover precisely this by a small RAG system experiment and requested myself:
How do completely different chunk sizes have an effect on retrieval habits?
The main focus is just not on a system supposed for manufacturing use. As a substitute, I wished to learn the way completely different chunk sizes have an effect on the retrieval outcomes.
2 – How Does Chunk Dimension Affect the Retrieval Ends in Small RAG Methods?
I subsequently requested myself the next questions:
- How does chunk dimension change retrieval ends in a small, managed RAG system?
- Which textual content segments make it to the highest of the rating when the queries are similar however the chunk sizes differ?
To analyze this, I intentionally outlined a easy setup through which all circumstances (besides chunk dimension) stay the identical:
- Three Markdown paperwork because the information base
- Three similar, fastened questions
- The identical embedding mannequin for vectorizing the texts
The textual content used within the three Markdown recordsdata relies on a documentation from an actual instrument known as OneLatex. To maintain the experiment targeted on retrieval habits, the content material was barely simplified and decreased to the core explanations related for the questions.
The three questions I used the place:
"Q1: What's the fundamental benefit of separating content material creation from formatting in OneLatex?"
"Q2: How does OneLatex interpret textual content highlighted in inexperienced in OneNote?"
"Q3: How does OneLatex interpret textual content highlighted in yellow in OneNote?"As well as, I intentionally omitted an LLM for output era.
The explanation for that is easy: I didn’t need an LLM to show incomplete or poorly-matched textual content segments right into a coherent reply. This makes it a lot clearer what truly occurs within the retrieval step, how the parameters of the retrieval work together, and what function the sentence transformer performs.
3 – Minimal RAG System With out Output Technology
For the experiments, I subsequently used a small RAG system with the next parts: Markdown paperwork because the information base, a easy chunking logic with overlap, a sentence transformer mannequin to generate embeddings, and a rating of textual content segments utilizing cosine similarity.
Because the embedding mannequin, I used all-MiniLM-L6-v2 from the Sentence-Transformers library. This mannequin is light-weight and subsequently well-suited for working regionally on a private laptop computer (I ran it regionally on my Lenovo laptop computer with 64 GB of RAM). The similarity between a question and a textual content phase is calculated utilizing cosine similarity. As a result of the vectors are normalized, the dot product may be in contrast instantly.
I intentionally saved the system small and subsequently didn’t embrace any chat historical past, reminiscence or agent logic, or LLM-based reply era.
As an “reply,” the system merely returns the highest-ranked textual content phase. This makes it a lot clearer which content material is definitely recognized as related by the retrieval step.
The complete code for the mini RAG system may be present in my GitHub repository:
→ 🤓 Discover the total code within the GitHub Repo 🤓 ←
4 – Three Experiments: Chunk Dimension as a Variable
For the analysis, I ran the three instructions beneath through the command line:
#Experiment 1 - Baseline
python fundamental.py --chunk-size 220 --overlap 40 --top-k 3
#Experiment 2 - Small Chunk-Dimension
python fundamental.py --chunk-size 80 --overlap 10 --top-k 3
#Experiment 3 - Huge Chunk-Dimension
python fundamental.py --chunk-size 500 --overlap 50 --top-k 3The setup from Part 3 stays precisely the identical: The identical three paperwork, the identical three questions, and the identical embedding mannequin.
Chunk dimension defines the variety of characters per textual content phase. As well as, I used an overlap in every experiment to cut back data loss at chunk boundaries. For every experiment, I computed the semantic similarity scores between the question and all chunks and ranked the highest-scoring segments.
Small Chunks (80 Characters) – Lack of Context
With very small chunks (chunk-size 80), a powerful fragmentation of the content material turns into obvious: Particular person textual content segments typically comprise solely sentence fragments or remoted statements with out enough context. Explanations are break up throughout a number of chunks, in order that particular person segments comprise solely components of the unique content material.
Formally, the retrieval nonetheless works accurately: Semantically comparable fragments are discovered and ranked extremely.
Nevertheless, once we take a look at the precise content material, we see that the outcomes are hardly usable:
The returned chunks are thematically associated, however they don’t present a self-contained reply. The system roughly acknowledges what the subject is about, but it surely breaks the content material down so strongly that the person outcomes don’t say a lot on their very own.
Medium Chunks (220 characters) – Obvious Stability
With the medium chunks (chunk-size 220), the outcomes already improved clearly. Many of the returned textual content segments contained full explanations and have been content-wise believable. At first look, the retrieval appeared secure and dependable: It normally returned precisely the data one would anticipate.
Nevertheless, a concrete drawback grew to become obvious when distinguishing between inexperienced and yellow highlighted textual content. No matter whether or not I requested in regards to the which means of the inexperienced or the yellow highlighting, the system returned the chunk in regards to the yellow highlighting as the highest end in each circumstances. The right chunk was current, but it surely was not chosen as Prime-1.
The explanation lies within the very comparable similarity scores of the 2 prime outcomes:
- Rating for Prime-1: 0.873
- Rating for Prime-2: 0.774
The system can hardly distinguish between the 2 candidates semantically and in the end selects the chunk with the marginally greater rating.
The issue? It doesn’t match the query content-wise and is just improper.
For us as people, that is very straightforward to acknowledge. For a sentence transformer like all-MiniLM-L6-v2, it appears to be a problem.
What issues right here is that this: If we solely take a look at the Prime-1 end result, this error stays invisible. Solely by evaluating the scores can we see that the system is unsure on this scenario. Since it’s pressured to make a transparent resolution in our setup, it returns the Prime-1 chunk as the reply.
Giant Chunks (500 characters) – Strong Contexts
With the bigger chunks (chunk-size 500), the textual content segments comprise far more coherent context. There’s additionally hardly any fragmentation anymore: Explanations are now not break up throughout a number of chunks.
And certainly, the error in distinguishing between inexperienced and yellow now not happens. The questions on inexperienced and yellow highlighting are actually accurately distinguished, and the respective matching chunk is clearly ranked as the highest end result. We are able to additionally see that the similarity scores of the related chunks are actually extra clearly separated.
This makes the rating extra secure and simpler to know. The draw back of this setting, nevertheless, is the coarser granularity: Particular person chunks comprise extra data and are much less finely tailor-made to particular elements.
In our setup with three Markdown recordsdata, the place the content material is already thematically properly separated, this draw back hardly performs a task. With otherwise structured documentation, corresponding to lengthy steady texts with a number of matters per part, an excessively giant chunk dimension may result in irrelevant data being retrieved along with related content material.
On my Substack Knowledge Science Espresso, I share sensible guides and bite-sized updates from the world of Knowledge Science, Python, AI, Machine Studying, and Tech — made for curious minds like yours.
Take a look and subscribe on Medium or on Substack if you wish to keep within the loop.
5 – Ultimate Ideas
The outcomes of the three quite simple experiments may be traced again to how retrieval works. Every chunk is represented as a vector, and its proximity to the question is calculated utilizing cosine similarity. The ensuing rating signifies how comparable the query and the textual content phase are within the semantic area.
What’s vital right here is that the rating is just not a measure of correctness. It’s a measure of relative comparability throughout the obtainable chunks for a given query in a single run.
When a number of segments are semantically very comparable, even minimal variations within the scores can decide which chunk is returned as Prime-1. One instance of this was the inaccurate distinction between inexperienced and yellow within the medium chunk dimension.
One attainable extension could be to permit the system to explicitly sign uncertainty. If the scores of the Prime-1 and Prime-2 chunks are very shut, the system may return an “I don’t know” or “I’m unsure” response as an alternative of forcing a choice.
Primarily based on this small RAG system experiment, it isn’t actually attainable to derive a “greatest chunk dimension” conclusion.
However what we will observe as an alternative is the next:
- Small chunks result in excessive variance: Retrieval reacts very exactly to particular person phrases however rapidly loses the general context.
- Medium-sized chunks: Seem secure at first look, however can create harmful ambiguities when a number of candidates are scored nearly equally.
- Giant chunks: Present extra sturdy context and clearer rankings, however they’re coarser and fewer exactly tailor-made.
Chunk dimension subsequently, determines how sharply retrieval can distinguish between comparable items of content material.
On this small setup, this didn’t play a serious function. Nevertheless, once we take into consideration bigger RAG methods in manufacturing environments, this sort of retrieval instability may change into an actual drawback: Because the variety of paperwork grows, the variety of semantically comparable chunks will increase as properly. Because of this many conditions with very small rating variations are more likely to happen. I may think about that such results are sometimes masked by downstream language fashions, when an LLM turns incomplete or solely partially matching textual content segments into believable solutions.
