Embeddings, Vector Databases and Advanced RAG

How embeddings are trained, similarity metrics, HNSW and IVF indexes, scaling to billions of vectors, multi-tenancy, drift, GraphRAG and agentic retrieval.

• Advanced
• 17 min read
• 11 questions

Before this lessonLesson 4: RAG: Retrieval-Augmented Generation

Once you can describe a basic RAG pipeline, senior interviews go one level down: how do vector databases find neighbours so fast, what happens when you switch embedding models, how do you keep one tenant from seeing another's documents, and what do you do when questions need several hops of reasoning?

These questions reward numbers and trade-offs. Practise the memory arithmetic in this lesson until you can do it aloud.

The 11 questions in this lesson

45. What are embeddings, and how are embedding models trained?
46. Compare cosine similarity, dot product and Euclidean distance. Which do you use and why?
47. What is a vector database and how does it differ from a traditional database? Compare flat, IVF, HNSW and PQ indexes.
48. How does HNSW work, and which parameters trade recall against latency?
49. How do you choose an embedding model, and how do dimensionality and Matryoshka embeddings affect cost?
50. You deployed a new embedding model and search quality crashed, or the new model has different dimensions from the existing vectors. What do you do?
51. How do you scale vector search to hundreds of millions or billions of vectors?
52. How do you handle multi-tenant data and per-user access control in a vector database?
53. What are GraphRAG, agentic RAG and Self-RAG, and when do you use each over plain RAG?
54. How do you keep a RAG system fresh and consistent: document updates, deletions, versioning and conflicting sources?
55. How do you fine-tune an embedding model for your domain, and when is it worth it?

45. What are embeddings, and how are embedding models trained?

Warm-up

An embedding is a list of numbers (a vector, commonly 384 to 3,072 dimensions) that represents the meaning of a piece of text, an image or another object so that similar things sit close together in the vector space. That geometry lets a computer search by meaning: "how do I get my money back" lands near "refund policy" even with no shared words.

Modern text embedders are transformer encoders (or decoders with pooling) trained with contrastive learning: given a query and a passage that answers it (a positive pair), the model is rewarded for making their vectors close and for pushing them away from other passages in the same batch (in-batch negatives) and from deliberately tricky hard negatives. The loss most often used is InfoNCE. Training data comes from search logs, question-answer pairs, duplicated questions, titles-and-bodies, and synthetic pairs generated by LLMs.

Two practical details worth knowing: many models are asymmetric and expect different prefixes for queries and documents (such as "query:" and "passage:"), and getting that wrong quietly hurts quality; and embeddings from different models are not comparable, so a query and its index must always use the same model and version.

Follow-up: Why can you not compare a vector from model A with a vector from model B, even if both have 768 dimensions?

46. Compare cosine similarity, dot product and Euclidean distance. Which do you use and why?

Core

• Dot product: sum of element-wise products. Sensitive to both angle and vector length.
• Cosine similarity: the dot product of the vectors after scaling each to length 1, so it measures only the angle (direction). Range -1 to 1.
• Euclidean (L2) distance: straight-line distance between points. Sensitive to length as well as direction.

The key fact: if all vectors are normalised to unit length, cosine similarity, dot product and Euclidean distance produce the same ranking (because for unit vectors, squared distance = 2 - 2 x cosine). Most embedding models output normalised vectors or are meant to be used with cosine, so the practical rule is: use whatever metric the model was trained with, normalise once at ingestion, and then use the dot product because it is the fastest to compute. Use raw dot product deliberately when magnitude carries meaning (for example some recommendation models).

import math

def dot(a, b): return sum(x * y for x, y in zip(a, b))
def norm(a): return math.sqrt(dot(a, a))
def cosine(a, b): return dot(a, b) / (norm(a) * norm(b))
def l2(a, b): return math.sqrt(sum((x - y) ** 2 for x, y in zip(a, b)))
def unit(a): n = norm(a); return [x / n for x in a]

q  = [1.0, 2.0, 3.0]
d1 = [2.0, 4.0, 6.0]       # same direction as q, twice as long
d2 = [3.0, 2.0, 1.0]       # different direction, similar length

print("raw   dot   d1=%.1f d2=%.1f" % (dot(q, d1), dot(q, d2)))       # length inflates d1
print("raw   L2    d1=%.2f d2=%.2f" % (l2(q, d1), l2(q, d2)))          # d1 looks farther away
print("cosine      d1=%.2f d2=%.2f" % (cosine(q, d1), cosine(q, d2)))  # d1 is identical in meaning
qu, u1, u2 = unit(q), unit(d1), unit(d2)
print("unit  dot   d1=%.2f d2=%.2f" % (dot(qu, u1), dot(qu, u2)))      # equals cosine
print("unit  L2^2  d1=%.2f d2=%.2f" % (l2(qu, u1) ** 2, l2(qu, u2) ** 2))   # = 2 - 2*cosine

Follow-up: Why does normalising vectors at ingestion time speed up search?

47. What is a vector database and how does it differ from a traditional database? Compare flat, IVF, HNSW and PQ indexes.

Core

A vector database stores vectors plus metadata and answers nearest-neighbour queries. A relational database finds rows by exact matching and ranges on indexed columns; a vector database finds the most similar items by geometry. Exact nearest-neighbour search is a brute-force scan (cost proportional to the number of vectors), which is fine for thousands of vectors and hopeless for hundreds of millions, so vector databases use Approximate Nearest Neighbour (ANN) indexes that trade a little recall for orders of magnitude of speed.

Beyond the index, real vector databases add metadata filtering, hybrid (keyword plus vector) search, replication and sharding, and durability. Common choices: pgvector inside Postgres (best when you already run Postgres and the scale is moderate), managed services such as Pinecone, and dedicated engines such as Qdrant, Weaviate, Milvus and OpenSearch/Elasticsearch. Pick by scale, filtering needs, ops burden and cost, not by benchmark headlines.

Follow-up: When would you keep vectors in Postgres with pgvector rather than adopt a dedicated vector database?

48. How does HNSW work, and which parameters trade recall against latency?

Deep dive

HNSW (Hierarchical Navigable Small World) builds a multi-layer proximity graph. Every vector is a node connected to some nearby nodes. The bottom layer contains all nodes; each higher layer contains a random, exponentially smaller subset, forming express lanes. Search starts at an entry point in the sparsest top layer, greedily hops to whichever neighbour is closer to the query until no neighbour improves, drops to the next layer from that position, and repeats. On the bottom layer it keeps a candidate list of size efSearch and returns the best k found.

flowchart TB subgraph L2["Layer 2 - few nodes, long jumps"] A2((entry)) --> B2((n1)) end subgraph L1["Layer 1 - more nodes"] B1((n1)) --> C1((n5)) --> D1((n9)) end subgraph L0["Layer 0 - every vector, short hops"] D0((n9)) --> E0((n11)) --> F0((n14 nearest)) end B2 -. descend .-> B1 D1 -. descend .-> D0

• M: maximum neighbours per node. Higher gives better recall and connectivity but more memory and slower build.
• efConstruction: candidate list size while building. Higher gives a better graph and slower indexing.
• efSearch: candidate list size at query time. The main dial: raise it for higher recall, lower it for speed. Latency grows roughly with it.

Operational caveats: the whole graph and usually the vectors live in RAM, so memory is the constraint; deleting and updating nodes leaves tombstones that degrade quality until the index is rebuilt or compacted; and filtered search can hurt recall if the filter removes most graph neighbours (good engines handle filtering inside the traversal).

Follow-up: Recall@10 is 0.88 and product wants 0.97. Which knobs do you turn and what do you pay?

49. How do you choose an embedding model, and how do dimensionality and Matryoshka embeddings affect cost?

Core

Choose on your data and task, not a leaderboard rank. Shortlist three to five candidates and evaluate them on a labelled sample of your own queries and documents, measuring recall@k and MRR. Criteria: retrieval quality on your domain (legal, code, medical text behave differently), language coverage (multilingual needs are a hard filter), maximum input length versus your chunk size, latency and throughput, cost (API price or GPU hosting), licence and data-privacy constraints (can text leave your network?), and vector dimension.

def index_gb(n_vectors, dims, bytes_per_value=4):
    return n_vectors * dims * bytes_per_value / 1e9

n = 10_000_000
for dims, label, b in [(1536, "float32", 4), (768, "float32", 4), (384, "float32", 4), (1536, "int8", 1), (1536, "binary", 1 / 8)]:
    print("%-8s %5d dims -> %7.1f GB" % (label, dims, index_gb(n, dims, b)))
print("1 billion x 768 float32: %.1f TB" % (index_gb(1_000_000_000, 768) / 1000))

Follow-up: A stakeholder wants the highest-ranked model on a public benchmark. What do you say?

50. You deployed a new embedding model and search quality crashed, or the new model has different dimensions from the existing vectors. What do you do?

Deep dive

Vectors from different models live in different spaces, so you cannot mix them in one index: a query embedded with the new model against old-model vectors is meaningless, and a dimension mismatch is simply rejected. There is no shortcut except a re-embed.

flowchart LR subgraph BLUE["Live: index v1 (old model)"] Q1[Queries] --> I1[(Index v1)] end subgraph GREEN["Build: index v2 (new model)"] DOCS[Source documents] --> RE[Re-embed in batches] RE --> I2[(Index v2)] end I2 --> EV[Evaluate on golden set and shadow traffic] EV --> SW{Better and stable?} SW -- yes --> CUT[Cut over gradually, keep v1 for rollback] SW -- no --> FIX[Fix and rebuild]

Playbook: build a second index with the new model while the old one keeps serving; write new and updated documents to both during the transition; run your golden set and replay real queries (shadow traffic) against v2; cut over gradually with a feature flag; keep v1 until confident, then delete it. Store the model name and version as metadata on every vector so this never becomes a guessing game, and keep the original text so re-embedding never depends on the old vectors.

If it crashed overnight without a deliberate migration, suspect an unpinned embedding-model alias that the provider updated, a changed preprocessing step, or a missing query/document prefix. Pin model versions and add an embedding-similarity canary test (a fixed set of query-document pairs whose scores must stay in range) to your monitoring.

Follow-up: Re-embedding 500 million chunks costs a lot. How do you reduce the cost and risk?

51. How do you scale vector search to hundreds of millions or billions of vectors?

Deep dive

Work down the levers in order:

1. Shrink each vector: lower dimensionality (Matryoshka truncation), scalar or product quantisation, binary codes for a first pass. Rerank a small shortlist with full-precision vectors.
2. Partition the index: shard by a natural key (tenant, date, language, collection) so a query touches one shard; hash-shard for uniform load; replicate for throughput and availability.
3. Choose the right index for the hardware: IVF-PQ for memory-tight, high-recall-tolerant workloads; HNSW when RAM is plentiful and latency matters; DiskANN-style indexes to keep the bulk on SSD.
4. Use tiered storage: hot recent or popular data in memory, cold data on disk or object storage, with a query router.
5. Filter early: metadata pre-filters cut candidates dramatically; design the partitioning around your most common filter.
6. Cache frequent queries and their results, and cache embeddings of repeated query text.
7. Batch and update smartly: ingest in batches, build indexes off the serving path, compact deletions on a schedule.

Always confirm with measurements: recall@k against an exact baseline on a sample, p50/p99 latency and cost per million queries, because each memory-saving step costs recall you must be able to justify.

import random, math
random.seed(5)

def dist(a, b): return sum((x - y) ** 2 for x, y in zip(a, b))
DIM, N = 8, 3000
vecs = [[random.gauss(0, 1) for _ in range(DIM)] for _ in range(N)]

# IVF-style index: pick centroids, put each vector in its nearest cell
cents = random.sample(vecs, 40)
cells = {i: [] for i in range(len(cents))}
for idx, v in enumerate(vecs):
    cells[min(range(len(cents)), key=lambda c: dist(v, cents[c]))].append(idx)

def ann(q, nprobe, k=10):
    order = sorted(range(len(cents)), key=lambda c: dist(q, cents[c]))[:nprobe]
    cand = [i for c in order for i in cells[c]]
    return sorted(cand, key=lambda i: dist(q, vecs[i]))[:k], len(cand)

def exact(q, k=10):
    return sorted(range(N), key=lambda i: dist(q, vecs[i]))[:k]

queries = [[random.gauss(0, 1) for _ in range(DIM)] for _ in range(60)]
for nprobe in (1, 3, 8, 20):
    rec, scanned = 0.0, 0
    for q in queries:
        got, n = ann(q, nprobe)
        rec += len(set(got) & set(exact(q))) / 10
        scanned += n
    print("nprobe=%-2d recall@10=%.2f  scanned %.0f%% of vectors" % (nprobe, rec / len(queries), 100 * scanned / len(queries) / N))

Follow-up: Your p99 latency doubled after the index grew 3x, with the same hardware. What do you check?

52. How do you handle multi-tenant data and per-user access control in a vector database?

Core

Security must be enforced in the retrieval layer, before any text reaches the model. If a forbidden chunk is in the prompt, no instruction can reliably stop the model from revealing it. Options, from strongest isolation to most efficient:

For per-user document permissions in an enterprise: sync ACLs from the source systems (SharePoint, Drive, Confluence) into chunk metadata as group and user ids, resolve the caller's groups at query time, and pass them as a mandatory filter built server-side from the authenticated identity, never from anything the client or the model supplies. Re-check permissions at answer time for high-risk sources, because ACLs change. Add tests that try cross-tenant queries and log every retrieval with the identity and the returned document ids for audit.

Follow-up: A document's permissions change from public to restricted. How fast must that propagate, and how do you make it so?

53. What are GraphRAG, agentic RAG and Self-RAG, and when do you use each over plain RAG?

Deep dive

Plain RAG retrieves a few similar chunks once and answers. It struggles when the answer needs relationships across many documents or several retrieval steps.

• GraphRAG: an LLM extracts entities and relations from the corpus to build a knowledge graph (often with community detection and pre-written summaries of each community). Queries can traverse relationships ("which suppliers are connected to this incident?") and answer global questions such as "what are the main themes across these reports?" that no single chunk contains. Cost: expensive, slower indexing, and the graph must be maintained.
• Agentic RAG: an agent decides whether, what and where to retrieve, can rewrite queries, call several indexes and tools, inspect the results and retrieve again until it has enough evidence. Best for complex, multi-hop or ambiguous questions; costs more latency and tokens and needs step limits and evaluation of the whole trajectory.
• Self-RAG: the model is trained to emit special reflection tokens that decide when retrieval is needed and to critique whether retrieved passages are relevant and its own output is supported. It builds the retrieve-and-verify loop into the model instead of the orchestration code.

My rule: start with well-tuned hybrid RAG plus reranking; add query decomposition for multi-hop questions; go agentic only for the query types that measurably fail; use GraphRAG when the domain is relationship-heavy or users ask corpus-level questions.

Follow-up: Give an example question that plain RAG will fail on but GraphRAG or an agentic loop will answer.

54. How do you keep a RAG system fresh and consistent: document updates, deletions, versioning and conflicting sources?

Core

Treat the index as a derived view of the source of truth that must be kept in sync, not a one-time import.

• Change detection: use source webhooks or periodic crawls with content hashes. When a document changes, re-parse and re-embed only the changed chunks; assign deterministic chunk ids (document id plus position or content hash) so updates overwrite instead of duplicating.
• Deletes and permissions: propagate deletions and access changes promptly (tombstone, then compact). Stale, deleted or restricted content in the index is both a quality and a security bug.
• Versioning: store version, effective date and status (draft, current, superseded) as metadata; filter to current by default but allow point-in-time queries ("what was the policy in 2024?"). Blue-green re-indexing lets you validate a big change before switching.
• Conflicts: deduplicate near-identical chunks; prefer newer or higher-authority sources via ranking boosts; include dates in the prompt and instruct the model to flag disagreements instead of blending them; and fix contradictions at the source when you find them.
• Freshness monitoring: report index lag (source updated versus indexed), track answers citing superseded documents, and alert on ingestion failures.

Follow-up: Users report answers quoting last year's price list. Walk through how you would find the cause.

55. How do you fine-tune an embedding model for your domain, and when is it worth it?

Deep dive

Consider it when off-the-shelf models miss your jargon (legal citations, part numbers, medical abbreviations) and you have already fixed chunking, hybrid search and reranking. It is often a cheaper win than a bigger LLM.

1. Build training pairs: real (query, relevant passage) pairs from search logs and click data, expert-labelled examples, or synthetic queries generated by an LLM for each of your chunks (filtered to remove bad ones).
2. Mine hard negatives: passages that look relevant but are not, taken from your current retriever's top results. They teach the model the distinctions that matter in your domain.
3. Train contrastively (often a few epochs with a small learning rate, sometimes with LoRA) and keep a held-out set of queries and documents that never appears in training.
4. Evaluate honestly against the base model on that held-out set (recall@k, MRR) and on general tasks to catch regressions.
5. Deploy safely: a new embedding model means a full re-embed and index migration, so treat it as a release.

Risks: overfitting to synthetic data, leakage between train and evaluation queries, and a model that is now excellent on your domain and worse on out-of-domain queries. Sometimes a reranker fine-tuned on the same data gives most of the gain without re-embedding the corpus.

Follow-up: You have 200 labelled queries and 2 million chunks. How do you generate enough training data?

Sources and further reading

• Malkov and Yashunin, HNSW
• Jegou et al., Product Quantization for Nearest Neighbor Search
• Kusupati et al., Matryoshka Representation Learning
• Edge et al., From Local to Global: A Graph RAG Approach
• Asai et al., Self-RAG
• AI Engineering interview questions (Outcome School, Apache-2.0)