Chunking, Vector Databases and ANN Search

How to split documents, store vectors, and search millions of them fast with approximate nearest neighbour indexes.

• Intermediate
• 26 min read
• 3 objectives

Before this lessonLesson 6: Vectors and Embeddings

You now know that text becomes vectors and that similar meanings sit close together. But a real company has millions of documents. Two practical questions follow: how do we cut documents into pieces a model can use (chunking), and how do we find the nearest vectors among millions in milliseconds (vector databases and approximate search)? This lesson answers both.

Step 1: Chunking, because one vector per document is a blurry vector

Embed a whole 30-page handbook as a single vector and you get a smoothie of every topic in it. A question about the refund policy will match that smoothie only weakly. So we split documents into chunks and embed each chunk. Retrieval then returns the specific passages that matter, and only those go into the prompt (saving tokens and money, lesson 2).

Chunking is where many RAG projects quietly succeed or fail. The trade-off:

• Too small (one sentence): precise, but loses context. "It must be returned within 30 days", what is "it"?
• Too big (five pages): full context, but the vector is blurry, retrieval is imprecise, and you waste prompt tokens on irrelevant text.
• Sweet spot: usually 200–800 tokens, with a small overlap (10–20%) so sentences that straddle a boundary are not lost.

Chunking strategies, from simple to smart

• Fixed size: every N characters/tokens. Easy, but cuts mid-sentence.
• Recursive / by structure: split on paragraphs first, then sentences, then words, only as needed. The sensible default.
• Semantic chunking: split where the topic changes (embedding similarity between neighbouring sentences drops).
• Document-aware: use headings, Markdown sections, code functions, table rows or slide boundaries as natural chunks.
• Parent-child: embed small chunks for precise matching, but hand the model the larger parent section for context (more in the RAG evaluation lesson).

Two chunkers side by side, on the same text, so you can see the difference:

import re

text = ("Refunds. Laptops can be returned within 30 days for a full refund. Opened software is not refundable. "
        "Shipping. Shipping is free on orders over 50 dollars. Express shipping takes two days. "
        "Warranty. Every laptop has a one year warranty. Batteries are covered for six months.")

def fixed_chunks(text, size, overlap=0):
    step = size - overlap
    return [text[i:i + size] for i in range(0, len(text), step)]

def sentence_chunks(text, max_chars):
    sentences = re.split(r"(?<=[.!?])\s+", text)
    chunks, current = [], ""
    for s in sentences:
        if current and len(current) + len(s) + 1 > max_chars:
            chunks.append(current)
            current = s
        else:
            current = f"{current} {s}".strip()
    if current:
        chunks.append(current)
    return chunks

print("FIXED (cuts words in half):")
for c in fixed_chunks(text, 90):
    print("  |", c)

print("\nSENTENCE-AWARE:")
for c in sentence_chunks(text, 110):
    print("  |", c)

FIXED (cuts words in half):
  | Refunds. Laptops can be returned within 30 days for a full refund. Opened software is not 
  | refundable. Shipping. Shipping is free on orders over 50 dollars. Express shipping takes t
  | wo days. Warranty. Every laptop has a one year warranty. Batteries are covered for six mon
  | ths.

SENTENCE-AWARE:
  | Refunds. Laptops can be returned within 30 days for a full refund. Opened software is not refundable.
  | Shipping. Shipping is free on orders over 50 dollars. Express shipping takes two days. Warranty.
  | Every laptop has a one year warranty. Batteries are covered for six months.

The fixed chunker slices "refundable" in the middle. The sentence-aware one keeps ideas whole. Always add metadata to each chunk (source file, page, section title, date, access permissions); it powers filtering, citations and debugging later.

Step 2: Searching millions of vectors

Suppose you have 10 million chunk vectors, each with 1,536 numbers. The honest way to find the nearest to a query is brute force: compute the similarity to every single one and keep the best. That is called exact or flat search. It is perfectly accurate, and fine up to maybe a hundred thousand vectors. At ten million it means about 15 billion multiplications per query, too slow for a chatbot.

The solution is Approximate Nearest Neighbour (ANN) search: build an index that finds very probably the best matches while looking at only a tiny fraction of the data. You give up a sliver of accuracy (say, 98% recall) for a 100× speedup. Let us feel the idea with the simplest ANN method: clustering ("IVF"). Group vectors into buckets around a few centres. At query time, look only in the bucket(s) whose centre is closest.

import math, random

rng = random.Random(42)

# 3,000 fake 2-D "vectors" scattered around 6 hidden topic centres
centres = [(0, 0), (10, 0), (0, 10), (10, 10), (5, 5), (-5, 8)]
points = [(cx + rng.gauss(0, 1.2), cy + rng.gauss(0, 1.2)) for cx, cy in centres for _ in range(500)]

def dist(a, b):
    return math.dist(a, b)

# --- brute force: compare against every point ---
def brute(query, k=3):
    scored = sorted(points, key=lambda p: dist(query, p))
    return scored[:k], len(points)

# --- IVF: assign each point to its nearest centre once, up front ---
buckets = {c: [] for c in centres}
for p in points:
    buckets[min(centres, key=lambda c: dist(p, c))].append(p)

def ivf(query, k=3, probe=1):
    nearest_centres = sorted(centres, key=lambda c: dist(query, c))[:probe]
    candidates = [p for c in nearest_centres for p in buckets[c]]
    return sorted(candidates, key=lambda p: dist(query, p))[:k], len(candidates)

query = (9.0, 9.5)
exact, looked_exact = brute(query)
approx, looked_ivf = ivf(query)

print(f"brute force looked at {looked_exact} points")
print(f"IVF (1 bucket) looked at {looked_ivf} points  ({looked_exact // looked_ivf}x fewer)")
print("same top-3 result:", exact == approx)

brute force looked at 3000 points
IVF (1 bucket) looked at 502 points  (5x fewer)
same top-3 result: True

About five times fewer comparisons and the same answer here. Real indexes are cleverer, and they do occasionally miss the true best match, which is why it is called approximate. The popular ones:

• HNSW (Hierarchical Navigable Small World): builds a multi-layer graph, like a road network with highways on top and local streets below. Search starts on the highway layer, hops toward the query, then drops to finer layers. Fast and accurate, the most common default. Costs extra memory.
• IVF (inverted file): the clustering idea you just ran. Often combined with compression.
• PQ (product quantization): compress each vector into a few bytes so billions fit in RAM, at some accuracy cost.
• DiskANN and friends: keep most of the index on SSD for huge collections.

flowchart TB subgraph L2["Layer 2 - highways, few nodes, big jumps"] direction LR A2[A] --- F2[F] end subgraph L1["Layer 1 - main roads"] direction LR A1[A] --- C1[C] --- F1[F] --- H1[H] end subgraph L0["Layer 0 - every home, tiny hops"] direction LR A0[A] --- B0[B] --- C0[C] --- D0[D] --- E0[E] --- F0[F] end L2 --> L1 --> L0

Start at the top, hop toward the query, drop a layer, repeat until you land next door.

The ANN dials

Every ANN index has knobs that trade speed against recall: for HNSW, M (connections per node), efConstruction (build effort) and efSearch (search effort); for IVF, the number of clusters and how many to probe. Higher effort means better recall and slower queries. You tune them by measuring recall@k against exact search on a sample of your own data.

Choosing a vector database

A vector database stores vectors alongside their text and metadata, builds ANN indexes, and answers "give me the nearest k, optionally filtered by metadata". The landscape:

• pgvector (a PostgreSQL extension): use your existing Postgres. Great when you already run it, want SQL joins and transactions, and have up to a few million vectors.
• Chroma, LanceDB: embedded/local, ideal for prototypes and small apps.
• FAISS: a library (not a server) from Meta; a fast building block.
• Qdrant, Weaviate, Milvus: dedicated open-source engines with filtering, hybrid search and horizontal scaling.
• Pinecone: fully managed cloud service; least ops work.
• Elasticsearch / OpenSearch / MongoDB Atlas: existing search/database products that added vector support, handy if you are already on them.

How to pick: start with the simplest thing you already run (often pgvector). Move to a dedicated engine when you hit scale, latency or feature limits (advanced filtering, multi-tenant isolation, very large collections).

Metadata filtering: the feature you will need on day two

Real questions are rarely "nearest anywhere". They are "nearest among documents this user is allowed to see, from the last year, in English". Filter on metadata, either pre-filtering (restrict candidates first) or post-filtering (search then discard, which can leave you with too few results). Good engines filter during the graph traversal. This is also how you enforce access control in RAG: never rely on the model to hide documents from users; never retrieve them in the first place.

# pgvector: nearest chunks for a tenant, in plain SQL
SELECT id, text, 1 - (embedding <=> %(query_vec)s) AS similarity
FROM   chunks
WHERE  tenant_id = %(tenant)s
  AND  published_at > now() - interval '1 year'
ORDER  BY embedding <=> %(query_vec)s      -- cosine distance
LIMIT  5;

Operating a vector store

• Keep text and vectors together with a stable chunk ID so you can cite and debug.
• Version your embeddings (model name + date) so you know when to re-embed.
• Re-index incrementally when documents change; deleting stale chunks matters as much as adding new ones.
• Watch memory: 10 million 1,536-dim float32 vectors is about 60 GB before the index. Compression or lower dimensions help.

# Write your solution here