Vectors and Embeddings

What a vector is, how text becomes numbers that capture meaning, and how cosine similarity finds related text.

• Intermediate
• 26 min read
• 3 objectives

Before this lessonLesson 5: APIs, SSE, WebSockets and Streaming

How does a computer know that "puppy" and "dog" are related, while "puppy" and "invoice" are not? Not from a dictionary, and not by matching letters ("puppy" and "puppet" share letters and mean nothing alike). The answer is embeddings: a way of turning text into a list of numbers so that meaning becomes geometry. This one idea powers semantic search, RAG, recommendations, clustering and much of modern AI. Let us build the intuition from zero.

What is a vector?

A vector is just a list of numbers. That is all. [3, 4] is a vector. So is [0.12, -0.53, 0.91]. The reason vectors are useful is that a vector with two numbers is a point on a map, three numbers is a point in 3D space, and 1,536 numbers is a point in a space we cannot picture but can still do maths in.

And once things are points, we can ask: how far apart are they? Points that are close together are similar. That is the whole trick.

A map of meaning (with two numbers)

Imagine a tiny map where the horizontal axis says "how much like an animal is it?" and the vertical axis says "how much like food is it?". We can place words on it by hand:

import math

# (animal-ness, food-ness) - hand-made for illustration
words = {
    "dog":       (0.95, 0.05),
    "puppy":     (0.90, 0.05),
    "cat":       (0.93, 0.07),
    "pizza":     (0.00, 0.98),
    "sandwich":  (0.02, 0.95),
    "chicken":   (0.60, 0.70),   # both an animal and a food!
}

def distance(a, b):
    return math.dist(words[a], words[b])

for a, b in [("dog", "puppy"), ("dog", "cat"), ("dog", "pizza"), ("pizza", "sandwich"), ("chicken", "dog"), ("chicken", "pizza")]:
    print(f"{a:>8} vs {b:<9} distance = {distance(a, b):.2f}")

     dog vs puppy     distance = 0.05
     dog vs cat       distance = 0.03
     dog vs pizza     distance = 1.33
   pizza vs sandwich  distance = 0.04
 chicken vs dog       distance = 0.74
 chicken vs pizza     distance = 0.66

"dog" and "puppy" are close (0.05). "dog" and "pizza" are far apart (about 1.3). And "chicken" sits in the middle, near both animals and foods, which is exactly right. An embedding model does this, but it invents its own axes, learned from reading enormous amounts of text, and it uses hundreds or thousands of them instead of two. Nobody names the axes; they capture subtle things like formality, topic, sentiment, tense and countless others.

What is an embedding, then?

An embedding is the vector an embedding model produces for a piece of text (a word, a sentence, a paragraph, a whole document). Similar meanings get nearby vectors. Typical sizes: 384, 768, 1,024, 1,536 or 3,072 numbers. You send text to an embedding API or model; you get a list of floats back.

embed("How do I reset my password?")   ->  [ 0.021, -0.113,  0.087, ..., 0.045 ]   (1,536 numbers)
embed("I forgot my login credentials")  ->  [ 0.019, -0.109,  0.091, ..., 0.041 ]   <- nearly identical!
embed("Best pizza in Naples")           ->  [-0.204,  0.310, -0.012, ..., 0.152 ]   <- somewhere else entirely

Notice the first two share almost no words yet land right next to each other. That is the magic keyword search cannot do: matching by meaning, not by spelling.

Measuring similarity: cosine, dot product, distance

There are three common ways to compare two vectors:

• Euclidean distance: the straight-line distance between the points. Smaller = more similar.
• Dot product: multiply matching numbers and add them up. Larger = more similar (but it also grows with vector length).
• Cosine similarity: the cosine of the angle between the vectors. It ignores length and only cares about direction. Ranges from -1 (opposite) to 1 (same direction); 0 means unrelated. This is the default for text, because a long document and a short one about the same topic point the same way.

Cosine similarity is dot(a, b) / (|a| × |b|). Here it is from scratch, so no library hides the idea:

import math

def dot(a, b):
    return sum(x * y for x, y in zip(a, b))

def norm(a):
    return math.sqrt(dot(a, a))

def cosine(a, b):
    return dot(a, b) / (norm(a) * norm(b))

cat  = [0.9, 0.1]
dog  = [0.8, 0.2]
pizza = [0.05, 0.95]

print("cat vs dog  :", round(cosine(cat, dog), 3))
print("cat vs pizza:", round(cosine(cat, pizza), 3))

# Length does not matter to cosine: a doubled vector points the same way
print("cat vs 2*cat:", round(cosine(cat, [x * 2 for x in cat]), 3))

cat vs dog  : 0.991
cat vs pizza: 0.163
cat vs 2*cat: 1.0

A working semantic search, with a toy embedder

Real embedding models are huge neural networks, but we can simulate one honestly: give every word a hand-made vector over a few "concept" dimensions, and embed a sentence as the average of its word vectors. It is crude, but it shows exactly how embedding-based search behaves, including finding matches that share no words with the query.

import math

# dimensions: [animals, food, technology, money, travel]
VEC = {
    "dog": [1,0,0,0,0], "puppy": [1,0,0,0,0], "cat": [1,0,0,0,0], "pet": [1,0,0,0,0], "adopt": [.6,0,0,.1,0],
    "pizza": [0,1,0,0,0], "recipe": [0,1,0,0,0], "cook": [0,1,0,0,0], "dinner": [0,1,0,0,.1], "restaurant": [0,.9,0,.2,.2],
    "laptop": [0,0,1,.1,0], "software": [0,0,1,0,0], "bug": [0,0,1,0,0], "code": [0,0,1,0,0], "computer": [0,0,1,.1,0],
    "invoice": [0,0,.1,1,0], "payment": [0,0,.1,1,0], "refund": [0,0,0,1,0], "price": [0,0,0,1,0], "bank": [0,0,0,1,0],
    "eat": [0,1,0,0,0], "flight": [0,0,0,.2,1], "hotel": [0,0,0,.3,1], "trip": [0,0,0,.1,1], "airport": [0,0,0,0,1], "visa": [0,0,0,.3,.9],
}

def embed(text):
    vecs = [VEC[w] for w in text.lower().split() if w in VEC]
    n = len(vecs) or 1
    return [sum(col) / n for col in zip(*vecs)] if vecs else [0]*5

def cosine(a, b):
    d = sum(x*y for x, y in zip(a, b))
    na, nb = math.sqrt(sum(x*x for x in a)), math.sqrt(sum(y*y for y in b))
    return d / (na * nb) if na and nb else 0.0

docs = [
    "adopt a puppy",
    "cook a pizza recipe",
    "fix a software bug",
    "request a refund for an invoice",
    "book a flight and hotel",
    "dinner at a restaurant near the airport",
]

for query in ["my dog needs a new pet", "where can I eat before my flight"]:
    q = embed(query)
    print("Q:", query)
    for d in sorted(docs, key=lambda d: cosine(q, embed(d)), reverse=True)[:3]:
        print(f"   {cosine(q, embed(d)):.2f}  {d}")

Q: my dog needs a new pet
   1.00  adopt a puppy
   0.00  cook a pizza recipe
   0.00  fix a software bug
Q: where can I eat before my flight
   0.98  dinner at a restaurant near the airport
   0.71  book a flight and hotel
   0.70  cook a pizza recipe

The first query shares zero words with "adopt a puppy", yet that document wins. The second mixes two ideas (food and travel) and the best match is the document that mixes the same two, the restaurant near the airport, with the pure-travel document behind it. A keyword engine would have returned nothing for either. That is semantic search.

Where do real embeddings come from?

Embedding models are neural networks trained with a clever objective: pull texts that mean the same thing together, push unrelated ones apart (contrastive learning). Show the model millions of pairs (question and its answer, a title and its article, a sentence and its paraphrase) and it learns a space where meaning is distance. Popular options: OpenAI text-embedding-3, Cohere Embed, Google's embeddings, Voyage, and open models like BGE, E5 and GTE that you can run yourself.

# Typical real-world usage (shown for reading; needs a key or a model download)
from openai import OpenAI
client = OpenAI()

resp = client.embeddings.create(
    model="text-embedding-3-small",
    input=["How do I reset my password?", "Best pizza in Naples"],
)
vectors = [item.embedding for item in resp.data]   # two lists of 1,536 floats
print(len(vectors), len(vectors[0]))

Choosing an embedding model

• Quality: check the MTEB leaderboard, but always test on your data.
• Dimensions: more dimensions capture more nuance but cost more storage and search time. Some models let you truncate ("Matryoshka" embeddings).
• Language: use a multilingual model if you serve multiple languages.
• Domain: code, legal and medical text often benefit from specialised models.
• Cost and privacy: an API is easy; a self-hosted open model keeps data in-house.
• Never mix models: vectors from different models live in different spaces. If you change the model, you must re-embed everything.

What embeddings are bad at

• Exact matches: product codes, error numbers, names, IDs. "ERR-4471" and "ERR-4472" look nearly identical to an embedding. (Keyword search and hybrid search fix this; see the next lessons.)
• Negation: "I like this" and "I do not like this" can embed close together.
• Numbers and dates: "before 2020" is poorly captured.
• Domain jargon the model never saw.
• Long text: one vector for a 20-page document blurs everything together. That is why we chunk (next lesson).

Beyond text

Anything can be embedded: images, audio, code, products, users, songs. Multimodal models even put images and text into the same space, so the text "a red bicycle" lands near photos of red bicycles. That is how image search by description works. Recommendation systems embed both users and items and recommend the nearest items.

# Write your solution here