Hybrid Search and Reranking

Combine keyword and vector search with reciprocal rank fusion, then rerank with a cross-encoder for precision.

• Intermediate
• 24 min read
• 3 objectives

Before this lessonLesson 8: Keyword Search and BM25

You now own two search tools that fail in opposite ways. Vector search understands meaning but blurs exact tokens. BM25 nails exact tokens but is blind to paraphrase. The obvious question: can we use both? Yes, and the result, hybrid search, is the single biggest upgrade most RAG systems can make. Then a second stage called reranking squeezes out extra precision.

The two-stage idea: cast a wide net, then look closely

flowchart TB Q[Query] Q --> BM25["BM25 top 50"] Q --> VEC["Vector top 50"] BM25 --> FUSE["Fuse with RRF"] VEC --> FUSE FUSE --> CAND[Top 50 candidates] CAND --> RR[Reranker] RR --> TOP[Top 5 to the LLM prompt]

Stage 1 is cheap recall over millions of docs. Stage 2 is slower precision over about 50.

Fusing two rankings: Reciprocal Rank Fusion

The tricky part of hybrid search: BM25 scores (like 12.7) and cosine similarities (like 0.83) are on completely different scales, so you cannot just add them. Reciprocal Rank Fusion (RRF) sidesteps that by ignoring scores entirely and using only positions. A document scores 1 / (k + rank) in each list (with k around 60), and the scores are summed. Documents ranked high in either list rise; documents high in both win.

from collections import defaultdict

# Ranked results (best first) from two different retrievers
bm25_ranking   = ["doc_err4471", "doc_refund", "doc_shipping", "doc_warranty"]
vector_ranking = ["doc_refund", "doc_returns", "doc_warranty", "doc_err4471"]

def rrf(rankings, k=60):
    scores = defaultdict(float)
    for ranking in rankings:
        for rank, doc in enumerate(ranking, start=1):
            scores[doc] += 1 / (k + rank)
    return sorted(scores.items(), key=lambda kv: -kv[1])

for doc, score in rrf([bm25_ranking, vector_ranking]):
    print(f"{score:.4f}  {doc}")

0.0325  doc_refund
0.0320  doc_err4471
0.0315  doc_warranty
0.0161  doc_returns
0.0159  doc_shipping

"doc_refund" is near the top of both lists, so it wins. "doc_err4471" is #1 for keywords but last for vectors, and still stays high in the fused list. "doc_returns" only the vector search found, and it still makes it in. Best of both, no score calibration needed. Most vector databases (Qdrant, Weaviate, Elasticsearch, Pinecone hybrid) offer RRF or a weighted variant out of the box.

An alternative is weighted score fusion: normalise both score sets to 0–1 and take alpha * vector + (1 - alpha) * keyword. You must tune alpha, which is why many teams start with RRF.

A complete hybrid retriever

Let us assemble everything from the last three lessons: a toy embedder, BM25 and RRF, over a small support knowledge base. Try the queries: one needs keywords, one needs meaning.

import math, re
from collections import Counter, defaultdict

DOCS = {
    "refund":   "Laptops can be returned within 30 days for a full refund.",
    "software": "Opened software is not refundable.",
    "err4471":  "Error ERR-4471 means the payment gateway timed out. Retry in five minutes.",
    "shipping": "Shipping is free on orders over 50 dollars. Express shipping takes two days.",
    "warranty": "Every laptop includes a one year warranty covering defects.",
    "password": "To reset your password use the forgot password link on the login page.",
}
SYNONYMS = {"credentials": "password", "login": "password", "forgot": "password", "money": "refund",
            "back": "refund", "send": "shipping", "deliver": "shipping", "broken": "warranty", "defect": "warranty"}

tok = lambda t: re.findall(r"[a-z0-9-]+", t.lower())

# --- keyword side: BM25 ---
ids = list(DOCS)
T = {i: tok(DOCS[i]) for i in ids}
N, avg = len(ids), sum(len(t) for t in T.values()) / len(ids)
df = Counter(w for t in T.values() for w in set(t))
def bm25(q, i, k1=1.5, b=0.75):
    tf = Counter(T[i]); s = 0.0
    for w in tok(q):
        if w in tf:
            idf = math.log((N - df[w] + .5) / (df[w] + .5) + 1)
            s += idf * tf[w] * (k1 + 1) / (tf[w] + k1 * (1 - b + b * len(T[i]) / avg))
    return s

# --- "semantic" side: words mapped to shared concept tokens (toy embedder) ---
def concepts(text):
    return Counter(SYNONYMS.get(w, w) for w in tok(text))
def cosine(a, b):
    d = sum(a[k] * b[k] for k in a)
    return d / (math.sqrt(sum(v*v for v in a.values())) * math.sqrt(sum(v*v for v in b.values()))) if a and b else 0
C = {i: concepts(DOCS[i]) for i in ids}

def rank_bm25(q):  return [i for i in sorted(ids, key=lambda i: -bm25(q, i)) if bm25(q, i) > 0]
def rank_vec(q):   return [i for i in sorted(ids, key=lambda i: -cosine(concepts(q), C[i])) if cosine(concepts(q), C[i]) > 0]

def hybrid(q, k=3, rrf_k=60):
    score = defaultdict(float)
    for ranking in (rank_bm25(q), rank_vec(q)):
        for r, i in enumerate(ranking, 1):
            score[i] += 1 / (rrf_k + r)
    return [i for i, _ in sorted(score.items(), key=lambda kv: -kv[1])][:k]

for q in ["ERR-4471", "I forgot my login credentials", "get my money back"]:
    print(f"Q: {q}")
    print("   keyword :", rank_bm25(q)[:3])
    print("   semantic:", rank_vec(q)[:3])
    print("   HYBRID  :", hybrid(q))

Q: ERR-4471
   keyword : ['err4471']
   semantic: ['err4471']
   HYBRID  : ['err4471']
Q: I forgot my login credentials
   keyword : ['password']
   semantic: ['password']
   HYBRID  : ['password']
Q: get my money back
   keyword : []
   semantic: ['refund']
   HYBRID  : ['refund']

Query 1 is a keyword job. Query 2 only matches on the single word "login" ("credentials" never appears in the document). Query 3 shares no word with its answer at all, so keyword search returns nothing and only the meaning side finds it. The hybrid list is correct in all three cases because each retriever covers the other's blind spot. This is the pattern in production systems.

Stage two: reranking

Retrieval must be fast, so it scores each document independently against the query using precomputed vectors. That is a bi-encoder: the query and the document are encoded separately and compared. Quick, but the query and document never actually "see" each other.

A cross-encoder takes the query and a candidate document together in one pass and outputs a relevance score. Because the model reads both at once, it catches subtle things: negation, whether the passage answers the question or merely mentions the topic. It is far more accurate and far slower (one model run per candidate), so you only run it on the top 20–100 candidates from stage one.

• Bi-encoder (retrieval): encode once, compare with a dot product. Millions of documents in milliseconds. Less precise.
• Cross-encoder (reranker): joint reading of query + document. Hundreds of candidates in a fraction of a second. Most precise.
• Popular rerankers: Cohere Rerank, Voyage rerank, Jina reranker, and open models such as BGE-reranker. You can also use an LLM itself to rank candidates, at higher cost.

# Rerank the top 50 candidates down to the best 5 (shown for reading)
import cohere
co = cohere.Client()

results = co.rerank(
    model="rerank-english-v3.0",
    query="Can I return a laptop after 30 days?",
    documents=[c.text for c in candidates],   # the ~50 from hybrid search
    top_n=5,
)
best = [candidates[r.index] for r in results.results]

Fixing the query itself

Sometimes the retrieval is fine and the question is the problem. Users type vague, short or conversational queries ("what about that one?"). Cheap fixes that use an LLM before retrieval:

• Query rewriting: turn a follow-up like "and for laptops?" into a standalone question ("What is the return policy for laptops?") using the conversation history.
• Multi-query: ask the LLM for three paraphrases of the question, retrieve for each, and merge results (with RRF). Catches phrasing mismatches.
• HyDE (hypothetical document embeddings): have the LLM write a fake answer, then search with that text. Answers look more like documents than questions do.
• Query decomposition: split "compare the warranty and return policies" into two sub-queries.
• Metadata extraction: pull filters out of the question ("invoices from March" becomes a date filter).

Rules of thumb

• Retrieve a lot (30–100), rerank, then keep a few (3–8) for the prompt.
• Hybrid + rerank usually beats either alone, often dramatically on messy real-world data.
• Measure! Compare vector-only, hybrid, and hybrid+rerank on your own question set (next lessons) instead of trusting anyone's benchmark.

# Write your solution here