. They remedy an actual downside, and in lots of instances, they’re the proper alternative for RAG methods. However right here’s the factor: simply since you’re utilizing embeddings doesn’t imply you want a vector database.
We’ve seen a rising pattern the place each RAG implementation begins by plugging in a vector DB. Which may make sense for large-scale, persistent data bases, nevertheless it’s not at all times essentially the most environment friendly path, particularly when your use case is extra dynamic or time-sensitive.
At Planck, we make the most of embeddings to boost LLM-based methods. Nevertheless, in certainly one of our real-world purposes, we opted to keep away from a vector database and as an alternative used a easy key-value retailer, which turned out to be a a lot better match.
Earlier than I dive into that, let’s discover a easy, generalized model of our situation to elucidate why.
Foo Instance
Let’s think about a easy RAG-style system. A consumer uploads a couple of textual content information, possibly some studies or assembly notes. We break up these information into chunks, generate embeddings for every chunk, and use these embeddings to reply questions. The consumer asks a handful of questions over the subsequent couple of minutes, then leaves. At that time, each the information and their embeddings are ineffective and might be safely discarded.
In different phrases, the info is ephemeral, the consumer will ask solely a few questions, and we wish to reply them as quick as attainable.
Now pause for a second and ask your self:
The place ought to I retailer these embeddings?
Most individuals’s intuition is: “I’ve embeddings, so I want a vector database”, however pause for a second and take into consideration what’s truly occurring behind that abstraction. If you ship embeddings to a vector DB, it doesn’t simply “retailer” them. It builds an index that hurries up similarity searches. That indexing work is the place a variety of the magic comes from, and in addition the place a variety of the price lives.
In a long-lived, large-scale data base, this trade-off makes good sense: you pay an indexing price as soon as (or incrementally as knowledge adjustments), after which unfold that price over thousands and thousands of queries. In our Foo instance, that’s not what’s occurring. We’re doing the other: always including small, one-off batches of embeddings, answering a tiny variety of queries per batch, after which throwing the whole lot away.
So the true query is just not “ought to I exploit a vector database?” however “is the indexing work price it?” To reply that, we will have a look at a easy benchmark.
Benchmarking: No-Index Retrieval vs. Listed Retrieval
This part is extra technical. We’ll have a look at Python code and clarify the underlying algorithms. If the precise implementation particulars aren’t related to you, be at liberty to skip forward to the Outcomes part.
We wish to examine two methods:
- No indexing in any respect, simply retains embeddings in reminiscence and scans them straight.
- A vector database, the place we pay an indexing price upfront to make every question quicker.
First, take into account the “no vector DB” strategy. When a question is available in, we compute similarities between the question embedding and all saved embeddings, then choose the top-k. That’s simply Okay-Nearest Neighbors with none index.
import numpy as np
def run_knn(embeddings: np.ndarray, query_embedding: np.ndarray, top_k: int) -> np.ndarray:
sims = embeddings @ query_embedding
return sims.argsort()[-top_k:][::-1]The code makes use of the dot product as a proxy for cosine similarity (assuming normalized vectors) and kinds the scores to seek out the very best matches. It actually simply scans all vectors and picks the closest ones.
Now, let’s have a look at what a vector DB sometimes does. Beneath the hood, most vector databases depend on an approximate nearest neighbor (ANN) index. ANN strategies commerce a little bit of accuracy for a big enhance in search velocity, and one of the broadly used algorithms for that is HNSW. We’ll use the hnswlib library to simulate the index conduct.
import numpy as np
import hnswlib
def create_hnsw_index(embeddings: np.ndarray, num_dims: int) -> hnswlib.Index:
index = hnswlib.Index(house='cosine', dim=num_dims)
index.init_index(max_elements=embeddings.form[0])
index.add_items(embeddings)
return index
def query_hnsw(index: hnswlib.Index, query_embedding: np.ndarray, top_k: int) -> np.ndarray:
labels, distances = index.knn_query(query_embedding, okay=top_k)
return labels[0]To see the place the trade-off lands, we will generate some random embeddings, normalize them, and measure how lengthy every step takes:
import time
import numpy as np
import hnswlib
from tqdm import tqdm
def run_benchmark(num_embeddings: int, num_dims: int, top_k: int, num_iterations: int) -> None:
print(f"Benchmarking with {num_embeddings} embeddings of dimension {num_dims}, retrieving top-{top_k} nearest neighbors.")
knn_times: listing[float] = []
index_times: listing[float] = []
hnsw_query_times: listing[float] = []
for _ in tqdm(vary(num_iterations), desc="Operating benchmark"):
embeddings = np.random.rand(num_embeddings, num_dims).astype('float32')
embeddings = embeddings / np.linalg.norm(embeddings, axis=1, keepdims=True)
query_embedding = np.random.rand(num_dims).astype('float32')
query_embedding = query_embedding / np.linalg.norm(query_embedding)
start_time = time.time()
run_knn(embeddings, query_embedding, top_k)
knn_times.append((time.time() - start_time) * 1e3)
start_time = time.time()
vector_db_index = create_hnsw_index(embeddings, num_dims)
index_times.append((time.time() - start_time) * 1e3)
start_time = time.time()
query_hnsw(vector_db_index, query_embedding, top_k)
hnsw_query_times.append((time.time() - start_time) * 1e3)
print(f"BENCHMARK RESULTS (averaged over {num_iterations} iterations)")
print(f"[Naive KNN] Common search time with out indexing: {np.imply(knn_times):.2f} ms")
print(f"[HNSW Index] Common index building time: {np.imply(index_times):.2f} ms")
print(f"[HNSW Index] Common question time with indexing: {np.imply(hnsw_query_times):.2f} ms")
run_benchmark(num_embeddings=50000, num_dims=1536, top_k=5, num_iterations=20)Outcomes
On this instance, we use 50,000 embeddings with 1,536 dimensions (matching OpenAI’s text-embedding-3-small) and retrieve the top-5 neighbors. The precise outcomes will fluctuate with totally different configs, however the sample we care about is identical.
I encourage you to run the benchmark with your personal numbers, it’s the easiest way to see how the trade-offs play out in your particular use case.
On common, the naive KNN search takes 24.54 milliseconds per question. Constructing the HNSW index for a similar embeddings takes round 277 seconds. As soon as the index is constructed, every question takes about 0.47 milliseconds.
From this, we will estimate the break-even level. The distinction between naive KNN and listed queries is 24.07 ms per question. That means you want 11,510 queries earlier than the time saved on every question compensates for the time spent constructing the index.
Moreover, even with totally different values for the variety of embeddings and top-k, the break-even level stays within the 1000’s of queries and stays inside a reasonably slim vary. You don’t get a situation the place indexing begins to repay after only a few dozen queries.
Now examine that to the Foo instance. A consumer uploads a small set of information and asks a couple of questions, not 1000’s. The system by no means reaches the purpose the place the index pays off. As an alternative, the indexing step merely delays the second when the system can reply the primary query and provides operational complexity.
For this form of short-lived, per-user context, the easy in-memory KNN strategy is just not solely simpler to implement and function, however additionally it is quicker end-to-end.
If in-memory storage is just not an possibility, both as a result of the system is distributed or as a result of we have to protect the consumer’s state for a couple of minutes, we will use a key-value retailer like Redis. We will retailer a singular identifier for the consumer’s request as the important thing and retailer all of the embeddings as the worth.
This offers us a light-weight, low-complexity resolution that’s well-suited to our use case of short-lived, low-query contexts.
Actual-World Instance: Why We Selected a Key-Worth Retailer
At Planck, we reply insurance-related questions on companies. A typical request begins with a enterprise title and handle, after which we retrieve real-time knowledge about that particular enterprise, together with its on-line presence, registrations, and different public data. This knowledge turns into our context, and we use LLMs and algorithms to reply questions based mostly on it.
The vital bit is that each time we get a request, we generate a contemporary context. We’re not reusing present knowledge, it’s fetched on demand and stays related for a couple of minutes at most.
If you happen to suppose again to the sooner benchmark, this sample ought to already be triggering your “this isn’t a vector DB use case” sensor.
Each time we obtain a request, we generate contemporary embeddings for short-lived knowledge that we’ll doubtless question only some hundred occasions. Indexing these embeddings in a vector DB provides pointless latency. In distinction, with Redis, we will instantly retailer the embeddings and run a fast similarity search within the software code with virtually no indexing delay.
That’s why we selected Redis as an alternative of a vector database. Whereas vector DBs are glorious at dealing with giant volumes of embeddings and supporting quick nearest-neighbor queries, they introduce indexing overhead, and in our case, that overhead is just not price it.
In Conclusion
If it is advisable to retailer thousands and thousands of embeddings and help high-query workloads throughout a shared corpus, a vector DB can be a greater match. And sure, there are positively use instances on the market that actually want and profit from a vector DB.
However simply since you’re utilizing embeddings or constructing a RAG system doesn’t imply it is best to default to a vector DB.
Every database expertise has its strengths and trade-offs. Your best option begins with a deep understanding of your knowledge and use case, reasonably than mindlessly following the pattern.
So, the subsequent time it is advisable to select a database, pause for a second and ask: am I choosing the proper one based mostly on goal trade-offs, or am I simply going with the trendiest, shiniest alternative?
