Lesson 10.1 · 27 min

Vector Databases: Storing and Searching Embeddings at Scale

How does a database find "I forgot my password" when the help article is titled "Reset your login credentials" and the two share no words at all?

In short: A vector database stores embeddings (lists of numbers that capture meaning) next to the original data and answers one question very fast: which stored vectors are closest to this query vector? It measures closeness with cosine similarity, dot product or Euclidean distance, and uses approximate indexes such as HNSW, IVF and PQ so it does not have to compare the query with every vector.

What is a vector database?

Imagine we run a support chatbot for an online shop. We have 200,000 help articles, past tickets and product pages. A customer types: "I forgot my password". The best article is titled "Reset your login credentials". A normal text search looks for shared words and finds nothing useful, because the two sentences share no important word. Yet any human sees at once that they mean the same thing.

A vector database solves exactly this. It is a database built to store vectors (lists of numbers) and to answer one kind of question very quickly: "Which stored vectors are most similar to this new vector?" If the vectors capture the meaning of text, images or audio, then "most similar vector" means "most similar meaning". That is the engine behind semantic search, recommendation and Retrieval-Augmented Generation (RAG), where an LLM is given relevant documents before it answers.

Think of it like a library arranged by topic, not by title A normal database is like a library sorted alphabetically by title: great if you know the exact title, useless if you only know what the book is about. A vector database is like a library where every book sits on a giant map, and books about similar things sit near each other. To find something, you walk to the spot on the map that matches your question and look at the books around you.

Popular examples include dedicated systems such as Pinecone, Weaviate, Milvus, Qdrant and Chroma, and vector features added to existing databases, such as the pgvector extension for PostgreSQL and vector search in Elasticsearch, OpenSearch and Redis. Libraries such as FAISS (from Meta) provide the core search algorithms without the database parts. Feature sets change quickly, so always check the current docs of the one you pick.

A quick recap of embeddings

An embedding is a list of numbers that an embedding model (a neural network trained for this job) produces from a piece of data. Typical text embeddings have a few hundred to a few thousand numbers; 384, 768, 1024 and 1536 are common sizes. Each list is a point in a high-dimensional space. The model is trained so that inputs with similar meaning land close together and unrelated inputs land far apart.

We cannot draw 768 dimensions, but the idea is the same in 2-D. In the map below, click a word and see that its nearest neighbours are words with related meaning. A vector database does this "find the neighbours" step for millions of points.

• "I forgot my password" might become [0.85, 0.20, 0.05, …].
• "Reset your login credentials" might become [0.80, 0.30, 0.10, …]: close, because the meaning is close.
• "Our refund policy" might become [0.10, 0.90, 0.20, …]: far away, because the meaning is different.

One rule to remember Vectors are only comparable if the same embedding model made them. A vector from model A and a vector from model B live in different spaces, so their distance means nothing. If we change models, we must re-embed the whole collection.

Why normal databases fall short

A relational database such as PostgreSQL or MySQL is brilliant at exact questions: WHERE user_id = 42, WHERE price < 20, WHERE title LIKE '%password%'. Its indexes (B-trees and hash indexes) work because values can be sorted or matched exactly. Sorting lets the database jump straight to the right place instead of reading every row.

Vectors break this. There is no useful way to sort 768-number lists in one line so that "similar" vectors end up next to each other. The question we ask is also different: not "equal to X" but "closest to X", and closeness depends on all 768 numbers at once. A B-tree cannot answer that. Without a special index, the only option is to compute the distance from the query to every stored vector, which we will see is too slow at scale.

What a vector database actually stores

Each record in a vector database usually has three parts:

• An ID, such as doc-1832#chunk-4, so we can find, update or delete the record.
• The vector, e.g. 768 floating-point numbers from the embedding model.
• Metadata (the payload): the original text or a link to it, plus fields such as language, source, date, customer tier or access rights.

Metadata matters more than beginners expect. It lets us filter: "find the nearest vectors, but only English articles updated in 2026 that this user may see". Vector databases support this as filtered search. Some apply the filter before the search (pre-filtering), some after (post-filtering, which can return fewer than k results if most neighbours are filtered out), and good engines blend the filter into the index walk itself. The details differ between products.

Around this sit normal database features: inserting and deleting records, persistence to disk, replication, sharding across machines, and access control. These are what separate a vector database from a vector library like FAISS, which gives us fast search but leaves storage and updates to us.

How do we measure similarity?

To say "closest", we need a number for how close two vectors are. Three measures dominate. Let us use two small vectors: a = [3, 4] and b = [4, 3].

Which should we use? Use the measure the embedding model was trained with; its documentation says. A handy fact: if every vector is normalised to length 1, then cosine similarity equals the dot product, and Euclidean distance ranks results in exactly the same order (because ‖a − b‖² = 2 − 2·cos(a, b) for unit vectors). That is why many systems normalise once at insert time and then use the cheap dot product.

Pause and think: Two vectors point in exactly the same direction, but one is twice as long. What is their cosine similarity, and is their Euclidean distance zero?

Cosine similarity is exactly 1, because cosine only looks at direction. Euclidean distance is not zero: the points are at different places along the same line. That is why the choice of metric matters for vectors that are not normalised.

The nearest neighbour problem and why brute force is too slow

The core task has a name: k-nearest neighbour (k-NN) search. Given a query vector q and a collection of N vectors, return the k vectors closest to q. The obvious way is brute force (also called flat or exhaustive search): compute the distance from q to every vector, then keep the best k. It is always exactly right.

Now count the work. One distance between 768-number vectors needs about 768 multiplications and 768 additions. With N = 10 million vectors that is about 7.7 billion multiply-adds for one query, plus reading about 30 GB of float32 numbers from memory (10M × 768 × 4 bytes). With hundreds of queries per second, that cost does not fit a fast, cheap service. Brute force grows linearly: 10× more data means 10× more work per query.

Brute force is not always wrong For a few thousand to maybe a hundred thousand vectors, brute force with a fast matrix multiply is often quick enough, gives perfect results, and needs no index tuning. Many libraries call this a "flat" index. Reach for ANN when data or traffic grows.

Approximate Nearest Neighbour (ANN) and indexing

Approximate Nearest Neighbour (ANN) search gives up a tiny bit of accuracy for a huge gain in speed. Instead of guaranteeing the true top-k, it returns results that are almost always the true top-k, while looking at only a small fraction of the data. Quality is measured with recall@k: of the true k nearest neighbours, what fraction did the index return? A recall@10 of 0.95 means on average 9.5 of the true top 10 were found.

To do this, the database builds an index: an extra data structure, built when data is inserted, that lets a query skip most vectors. Three ideas cover most production systems: HNSW (a graph), IVF (clusters) and PQ (compression). They are often combined, for example IVF-PQ.

HNSW, IVF and PQ explained simply

HNSW (Hierarchical Navigable Small World) builds a graph: each vector is a node linked to some of its near neighbours. It stacks several layers. The top layer has few nodes with long links (like motorways); lower layers have more nodes with shorter links (like local streets); the bottom layer has every node. A search starts at the top, greedily hops to whichever neighbour is closest to the query, drops a layer when it cannot improve, and repeats until the bottom, where it explores a small candidate list. HNSW was described by Malkov and Yashunin (2016) and is the default index in many vector databases because it is fast with high recall. Its costs: extra memory for the links, and slower inserts.

IVF (Inverted File index) first groups all vectors into, say, 1,000 clusters with k-means. Each cluster has a centre (centroid) and a list of its members (the "inverted list"). At query time we compare the query with the 1,000 centroids, pick the closest few clusters (the nprobe setting), and search only inside them. With nprobe = 10 we search about 1% of the data. The risk: a true neighbour sitting just across a cluster border is missed, which is why raising nprobe raises recall.

PQ (Product Quantization) attacks memory rather than search order. It cuts each vector into, say, 96 sub-vectors of 8 numbers, and for each slot learns 256 typical patterns (a codebook). Each sub-vector is replaced by the 1-byte ID of its closest pattern. A 768-float vector (3,072 bytes) becomes 96 bytes, a 32× saving. Distances are then estimated from small lookup tables. PQ was introduced by Jégou, Douze and Schmid (2011). The price is some precision, so systems often re-check the top candidates with the full vectors.

What happens when a query arrives

1. Embed the query: The same embedding model that built the collection turns "I forgot my password" into a vector q.
2. Apply filters: Metadata conditions (language = en, tenant = acme) narrow which records may be returned.
3. Walk the index: HNSW hops through the graph, or IVF picks the nearest clusters, so only a small set of candidates is examined.
4. Score candidates: Each candidate gets a similarity score (cosine, dot or L2), possibly estimated from PQ codes.
5. Return top-k: The k best IDs, scores and payloads go back to the app, e.g. to feed an LLM in a RAG pipeline.

Pause and think: An IVF index has 1,000 clusters of roughly equal size, and we set nprobe = 20. Roughly what fraction of the stored vectors does each query compare against in detail?

About 20 / 1,000 = 2% of the vectors (plus 1,000 cheap comparisons against the centroids). That is the source of the speed-up, and also why some true neighbours in unsearched clusters can be missed.

A small code example

Here is a toy vector store in about 35 lines of numpy. It keeps IDs, text, metadata and vectors; scores with any of the three metrics; and supports a metadata filter. It uses brute force, which is exactly what a "flat" index does. The 3-number vectors are hand-made stand-ins for real embeddings.

tiny_vector_db.py

import numpy as np
# A tiny "vector database": id -> (text, metadata, embedding)
# The 3-number embeddings are hand-made stand-ins for real 768-number ones.
store = {
"d1": ("How to reset your password", {"lang": "en"}, [0.9, 0.1, 0.0]),
"d2": ("Change your login credentials", {"lang": "en"}, [0.8, 0.3, 0.1]),
"d3": ("Our refund policy for orders", {"lang": "en"}, [0.1, 0.9, 0.2]),
"d4": ("Restablecer la contrasena", {"lang": "es"}, [0.9, 0.2, 0.1]),
"d5": ("Track the shipping of an order", {"lang": "en"}, [0.0, 0.6, 0.8]),
}
ids = list(store)
M = np.array([store[i][2] for i in ids], dtype=float)   # one row per vector
def search(q, k=2, metric="cosine", where=None):
q = np.array(q, dtype=float)
if metric == "cosine":
s = (M @ q) / (np.linalg.norm(M, axis=1) * np.linalg.norm(q))
elif metric == "dot":
s = M @ q
else:                                   # euclidean: smaller = closer
s = -np.linalg.norm(M - q, axis=1)  # negate so bigger = better
hits = []
for i in np.argsort(-s):                # brute force: score every row
meta = store[ids[i]][1]
if where and any(meta.get(f) != v for f, v in where.items()):
continue                        # metadata filter
score = -s[i] if metric == "euclidean" else s[i]   # show real distance
hits.append((ids[i], store[ids[i]][0], round(float(score), 3)))
if len(hits) == k:
break
return hits
query = [0.85, 0.2, 0.05]   # pretend embedding of "I forgot my password"
for metric in ["cosine", "dot", "euclidean"]:
print(f"{metric:9}", search(query, metric=metric))
print("en only  ", search(query, where={"lang": "en"}))

Output:

cosine    [('d4', 'Restablecer la contrasena', 0.999), ('d1', 'How to reset your password', 0.991)]
dot       [('d4', 'Restablecer la contrasena', 0.81), ('d1', 'How to reset your password', 0.785)]
euclidean [('d4', 'Restablecer la contrasena', 0.071), ('d2', 'Change your login credentials', 0.122)]
en only   [('d1', 'How to reset your password', 0.991), ('d2', 'Change your login credentials', 0.99)]

Read the output carefully. Cosine and dot product agree on the top two. Euclidean distance puts d2 second instead of d1: these vectors are not normalised, so length changes the answer. The Spanish article d4 wins on meaning, which is great for a multilingual model, but when the filter lang = en is applied it disappears and the English articles take its place. Real systems do the same thing, just with an ANN index instead of a full sort.

Real-world applications and common mistakes

Where vector databases are used RAG chatbots fetch the most relevant document chunks for an LLM. Semantic search on sites and in apps finds results by meaning. Recommendations find products, songs or articles similar to what a user liked. Image and audio search find similar photos or sounds from embeddings of the media. Deduplication and clustering find near-duplicate tickets or records. Anomaly detection flags items far from all their neighbours.

Common mistakes Mixing vectors from different embedding models (or versions) in one index. Using a metric the model was not trained for. Forgetting that ANN results are approximate and never measuring recall. Relying only on vector search for exact terms like order numbers or error codes, where keyword search is better. Ignoring metadata filters and access rights, so users can retrieve documents they should not see.

When might we not need a vector database? If the data is small (a few thousand items), a numpy array or a flat index is enough. If users search for exact identifiers, a keyword index is better. If we already run PostgreSQL and the scale is moderate, an extension like pgvector may save us running a separate system. Choose the simplest tool that meets recall and latency targets.

Worked example, step by step

Before we pick an index, it helps to do the memory sums on paper. Say our shop splits its 200,000 articles and tickets into 2,000,000 chunks, and each chunk becomes 768 float32 numbers. The link and payload sizes below are illustrative; real engines differ in the details.

Sizing the collection

1. Raw vectors: 2,000,000 × 768 × 4 bytes = 6,144,000,000 bytes, about 6.1 GB. A flat (brute-force) index needs exactly this.
2. Add an HNSW graph: Links are stored as IDs. If each node keeps about 32 links on the bottom layer and an ID takes 4 bytes, that is about 128 bytes per vector, or roughly 0.26 GB more. The vectors, not the links, dominate.
3. Try PQ instead: 96 one-byte codes per vector: 2,000,000 × 96 bytes = 192 MB. That is 32× smaller than the raw vectors, paid for with less exact distances.
4. Do not forget the payload: If each chunk carries about 1 KB of text and metadata, that is another 2 GB. Many systems keep the payload on disk and only the index in memory.
5. Decide: About 6.4 GB for HNSW fits in the memory of one modest server. If the collection grows 10×, 64 GB may not, and PQ, smaller vectors or several machines (shards) come into play.

Two lessons from the sums. First, the dimension is the big lever: an embedding model with 384 numbers instead of 768 halves every row of the table. Second, memory grows in a straight line with the number of vectors, so we can size next year from a guess of how much data we will have.

Practice: try it yourself

We will build a tiny store with a language filter and compare the two simple ways to apply it: post-filtering (search, then throw away records that fail the filter) and pre-filtering (throw away first, then search). The vectors are normalised once at insert time, so a plain dot product is the cosine similarity.

practice_filtered_search.py

import numpy as np
# Records: id -> (language, embedding). Hand-made, illustrative vectors.
records = {
"en-reset":  ("en", [0.90, 0.10, 0.10]),
"es-reset":  ("es", [0.90, 0.20, 0.00]),
"de-reset":  ("de", [0.80, 0.10, 0.20]),
"fr-reset":  ("fr", [0.85, 0.15, 0.10]),
"en-login":  ("en", [0.70, 0.40, 0.10]),
"en-refund": ("en", [0.10, 0.90, 0.20]),
"en-ship":   ("en", [0.00, 0.50, 0.90]),
}
ids = list(records)
V = np.array([records[i][1] for i in ids], dtype=float)
V /= np.linalg.norm(V, axis=1, keepdims=True)   # normalise once, at insert time
def ranked(q):
q = np.array(q, dtype=float)
q /= np.linalg.norm(q)
scores = V @ q                               # dot product = cosine now
return [(ids[i], round(float(scores[i]), 3)) for i in np.argsort(-scores)]
def post_filter(q, lang, k):                     # search first, filter after
return [(i, s) for i, s in ranked(q)[:k] if records[i][0] == lang]
def pre_filter(q, lang, k):                      # filter first, then keep k
return [(i, s) for i, s in ranked(q) if records[i][0] == lang][:k]
query = [0.88, 0.15, 0.05]                       # "I forgot my password"
print("no filter  :", ranked(query)[:3])
print("post-filter:", post_filter(query, "en", 3))
print("pre-filter :", pre_filter(query, "en", 3))

Output:

no filter  : [('fr-reset', 0.998), ('es-reset', 0.997), ('en-reset', 0.997)]
post-filter: [('en-reset', 0.997)]
pre-filter : [('en-reset', 0.997), ('en-login', 0.938), ('en-refund', 0.281)]

Now change it:

• In the post-filter print line, change 3 to 5 so the search fetches more candidates before filtering. Predict how many English results survive, then run it.
• Change the filter language to "es" in both calls. Predict whether pre-filtering can still return 3 results, and why not.
• Delete the normalising line (line 15) and change the en-ship vector to [0.0, 5.0, 9.0]. Predict which record jumps to the top of the unfiltered list, and what that says about the dot product on vectors of different lengths.

Pause and think: We asked the post-filter for 3 results and got 1. Nothing crashed. Why did this happen, and what would a real engine do about it?

The 3 nearest records overall were French, Spanish and English. Filtering after the search removed two of them, and nothing refilled the list. Real engines fix this by fetching more candidates than k before filtering, or by applying the filter before or during the index walk so that every candidate already passes it.

Pause and think: Only 1% of our records are in Spanish, and a query filters on lang = es. Which is likely to work better: an ANN search followed by a post-filter, or a pre-filter followed by brute force on what is left?

Pre-filter then brute force. The filter leaves a small set, so scoring all of it is cheap and exact. A post-filter would have to pull about 100 candidates for every Spanish result it keeps, and could still return fewer than k.

Key takeaways

• A vector database stores embeddings plus IDs and metadata, and finds the nearest vectors to a query fast.
• Similarity is measured with cosine similarity, dot product or Euclidean distance; for unit vectors they rank results the same way.
• Brute force is exact but costs one distance per stored vector, so it does not scale to millions of vectors and high traffic.
• ANN indexes trade a little recall for big speed: HNSW (graph), IVF (clusters) and PQ (compression).
• Always use one embedding model per index, use its intended metric, and combine vector search with metadata filters.

Key terms

• Vector database: A database that stores vectors with metadata and answers nearest-neighbour queries quickly.
• Embedding: A list of numbers produced by a model so that similar inputs get nearby vectors.
• Cosine similarity: The cosine of the angle between two vectors; 1 means same direction, length is ignored.
• k-NN search: Finding the k stored vectors closest to a query vector.
• ANN: Approximate nearest neighbour search: very fast search that returns almost always the true nearest neighbours.
• Recall@k: The fraction of the true top-k neighbours that the search actually returned.
• HNSW: A layered graph index that finds neighbours by greedy hops from coarse to fine layers.
• Product Quantization: A compression method that replaces sub-vectors with short codebook IDs to save memory.

← 9.5 Context Compaction: Fitting More Into a Finite Window · 10.2 ANN Search: Finding Similar Vectors Without Brute Force →