RAG: Retrieval-Augmented Generation

What RAG is and when to use it, chunking, embeddings, hybrid search, reranking, query rewriting, grounding, evaluation and debugging a RAG bot that gets things wrong.

• Intermediate
• 15 min read
• 11 questions

Before this lessonLesson 3: Prompt Engineering and Structured Output

RAG is the most-asked applied topic in AI interviews right now, because almost every company that ships an LLM feature ships some form of it. The basic definition is a warm-up. The real interview is a scenario: "your RAG bot is wrong 8% of the time; what do you do?"

Answer those with a method: measure retrieval and generation separately, find which one fails, fix the stage that is actually broken, and re-run the same test set.

The 11 questions in this lesson

34. What is Retrieval-Augmented Generation, why is it needed, and what does a basic RAG architecture look like?
35. When would you choose RAG over fine-tuning, or over just using a very long context window?
36. What are chunking strategies, and how do you choose chunk size and overlap?
37. Explain hybrid search. Why is it better than pure vector search, and what is Reciprocal Rank Fusion?
38. What is reranking? Compare bi-encoders, cross-encoders and ColBERT-style late interaction.
39. What is query transformation in RAG (rewriting, multi-query, HyDE, decomposition, step-back)?
40. How do you build the prompt, force grounded answers, add citations and make the bot say 'I don't know'?
41. How do you evaluate a RAG system? Explain recall@k, MRR, nDCG, faithfulness, answer relevance and context precision.
42. Your RAG bot gives wrong answers. How do you find out whether retrieval or generation is at fault?
43. Your RAG system hallucinates even though the right context is retrieved. Why, and how do you fix it?
44. How do you handle structured data, tables and PDFs in a RAG pipeline?

34. What is Retrieval-Augmented Generation, why is it needed, and what does a basic RAG architecture look like?

Warm-up

RAG gives an LLM fresh, private or domain-specific knowledge at query time by retrieving relevant passages from an external store and placing them in the prompt. It exists because a model's knowledge is frozen at its training cut-off, it has never seen your private documents, and it hallucinates when it does not know. Retrieval turns the task from "recall from memory" into "read and summarise this evidence", which is far more reliable and can cite sources.

flowchart TB subgraph OFF["Offline - build the knowledge base"] D[Documents] --> P[Parse and clean] P --> C[Chunk] C --> E[Embed chunks] E --> S[(Vector index plus keyword index plus metadata)] end subgraph ON["Online - answer a question"] Q[User question] --> R[Rewrite query] R --> H[Hybrid retrieve top-k] S --> H H --> RR[Rerank top-n] RR --> PR[Build prompt with sources] PR --> L[LLM generates grounded answer] L --> CK[Check faithfulness and cite] CK --> A[Answer with citations] end

Key components: ingestion (parsing, cleaning, chunking, embedding, indexing with metadata), retrieval (vector, keyword or hybrid search, filters, reranking), generation (prompt assembly, the model, citation and refusal behaviour) and evaluation and monitoring across all of it. Say explicitly that most production quality problems live in ingestion and retrieval, not in the LLM.

Follow-up: Where in this pipeline do access controls belong, and why not in the prompt?

35. When would you choose RAG over fine-tuning, or over just using a very long context window?

Core

Use the rule: RAG for knowledge, fine-tuning for behaviour, long context for small, stable, one-off inputs.

Long contexts keep getting cheaper, but they do not replace retrieval: cost and latency scale with tokens on every call, accuracy drops for facts buried in long prompts, and you still cannot put an enterprise's millions of documents into a window or enforce per-user permissions inside it. The pragmatic answer is often both: retrieve a focused set of passages, then use a larger window to hold more of them or the full parent document.

Follow-up: Your legal team wants answers that always quote a source paragraph. Which approach and why?

36. What are chunking strategies, and how do you choose chunk size and overlap?

Core

Chunking splits documents into retrievable pieces. It matters because the chunk is the unit that gets embedded, retrieved and shown to the model: too large and the embedding blurs several topics and wastes prompt tokens; too small and a chunk loses the context needed to be understood.

Starting point: roughly 200 to 500 tokens per chunk with 10 to 20 percent overlap, then tune against retrieval metrics rather than guessing. Prefix each chunk with its document title and section heading (contextual chunking) so an isolated paragraph still says what it is about, and keep tables and code intact. Attach metadata (source, date, section, permissions) to every chunk for filtering and citations.

import re

def chunk(text, max_words=40, overlap=8):
    # sentence-aware chunker with word overlap; a simple, testable baseline
    sentences = [s.strip() for s in re.split(r"(?<=[.!?])\s+", text.replace("\n", " ")) if s.strip()]
    chunks, current = [], []
    for s in sentences:
        words = s.split()
        if current and len(current) + len(words) > max_words:
            chunks.append(" ".join(current))
            current = current[-overlap:]           # carry the tail forward for context
        current.extend(words)
    if current:
        chunks.append(" ".join(current))
    return chunks

doc = ("Refunds are issued within 5 business days. A refund requires the original receipt. "
       "Digital goods are refundable within 14 days. Shipping fees are not refundable. "
       "Gift cards cannot be refunded. Contact support for damaged items. "
       "Damaged items are replaced at no cost. Replacements ship within 2 days.")
for i, c in enumerate(chunk(doc, max_words=25, overlap=6)):
    print(i, "|", c)

Follow-up: Answers keep missing because the key sentence is split across two chunks. What do you change?

37. Explain hybrid search. Why is it better than pure vector search, and what is Reciprocal Rank Fusion?

Core

Scores from the two systems are on incompatible scales, so the standard merge is Reciprocal Rank Fusion (RRF): each document scores the sum of 1 / (k + rank) over every list it appears in (k is usually 60). It uses only ranks, needs no score normalisation, and rewards documents that both retrievers like.

def rrf(rankings, k=60):
    scores = {}
    for ranking in rankings:
        for rank, doc in enumerate(ranking, start=1):
            scores[doc] = scores.get(doc, 0.0) + 1.0 / (k + rank)
    return sorted(scores, key=scores.get, reverse=True)

vector_hits  = ["refund-policy", "returns-faq", "shipping", "gift-cards"]
keyword_hits = ["err-4021-doc", "refund-policy", "returns-faq", "warranty"]   # exact code match on top
print(rrf([vector_hits, keyword_hits]))
# refund-policy wins because both retrievers agree; the exact-match doc is kept in the list

Also mention metadata filtering (by tenant, date, product) applied before or during search, which shrinks the candidate set and enforces access control.

Follow-up: A user searches for the exact error code E-4021 and vector search returns unrelated pages. Explain why, and the fix.

38. What is reranking? Compare bi-encoders, cross-encoders and ColBERT-style late interaction.

Core

Retrieval is a two-stage funnel: a fast, cheap first stage pulls a wide candidate set (say the top 50 to 200) by approximate search; a slower, more accurate reranker re-scores just those candidates and keeps the best 5 to 10 for the prompt. Reranking is usually the biggest single precision gain after hybrid search.

flowchart LR Q[Query] --> S1["Stage 1: hybrid retrieval - top 100 candidates, milliseconds"] S1 --> S2["Stage 2: reranker scores each query and passage pair - top 5"] S2 --> P[Prompt context]

The trade-off to state: rerankers add latency (tens to hundreds of ms) and cost, so rerank only the shortlist, batch the pairs, cache where possible, and measure the gain in recall@k and answer quality to prove it pays for itself.

Follow-up: Why can't you simply use a cross-encoder over the whole corpus?

39. What is query transformation in RAG (rewriting, multi-query, HyDE, decomposition, step-back)?

Core

User questions are often short, ambiguous or conversational ("what about the other plan?"), which makes poor search queries. Query transformation rewrites them before retrieval:

• Rewrite / condense: turn a follow-up into a standalone question using the chat history.
• Multi-query: generate several paraphrases, retrieve for each, merge the results. Improves recall on vague queries.
• HyDE (hypothetical document embeddings): have the LLM write a plausible answer, then embed that to search. Works because a hypothetical answer resembles real passages more than a question does; risky if the model's imagined answer is wrong.
• Decomposition: split a multi-part or multi-hop question into sub-questions, retrieve for each, then combine. Essential for "compare X and Y" style questions.
• Step-back: ask a more general question first ("what are the principles of X?") to fetch background, then the specific one.
• Routing: decide which index, filter or tool to query, or that no retrieval is needed.

Each step adds an LLM call and latency, so use them selectively: measure whether recall improves on your evaluation set and apply them only to the query types that benefit.

Follow-up: A user asks 'and what about pricing for the enterprise one?' with no other context. How does your pipeline handle it?

40. How do you build the prompt, force grounded answers, add citations and make the bot say 'I don't know'?

Core

Give the model explicit rules and clearly labelled evidence: "Answer only from the sources below. If they do not contain the answer, reply exactly: I could not find this in the documentation. Cite the source id after each claim." Number the passages, include metadata (title, section, date), and put the question last.

• Citations: ask for source ids inline, then verify them in code: check the cited passage really exists and, ideally, that it supports the claim (an NLI model or LLM check). Models can fabricate plausible citations.
• Refusals: a refusal must be an allowed, easy path. Add unanswerable questions to your test set and measure refusal accuracy, otherwise the model will always try to answer. A retrieval-score threshold ("nothing relevant found") can short-circuit before the LLM is even called.
• Conflicts and freshness: include dates and ask the model to prefer the newest source and to flag disagreement between documents instead of blending them.
• Context order and size: put the strongest passages first and last, drop weak ones; more context is not better.
• Post-checks: a faithfulness check that every claim is supported, and PII or policy filters on the output.

Follow-up: Your bot cites document IDs that exist but do not support the claim. How do you catch it automatically?

41. How do you evaluate a RAG system? Explain recall@k, MRR, nDCG, faithfulness, answer relevance and context precision.

Core

Evaluate the two halves separately and then end to end, on a golden set of real questions with known relevant documents and reference answers (include unanswerable ones).

Faithfulness and relevance are usually scored by an LLM-as-judge or an NLI model, calibrated against a sample of human labels. Tools such as RAGAS, TruLens and DeepEval package these metrics.

def recall_at_k(retrieved, relevant, k):
    return len(set(retrieved[:k]) & set(relevant)) / len(relevant)

def mrr(retrieved, relevant):
    for rank, doc in enumerate(retrieved, start=1):
        if doc in relevant:
            return 1.0 / rank
    return 0.0

cases = [
    (["a", "x", "b", "y"], {"a", "b"}),
    (["x", "y", "z", "b"], {"b"}),
    (["x", "y", "z", "w"], {"q"}),      # retrieval miss
]
print("recall@3:", sum(recall_at_k(r, rel, 3) for r, rel in cases) / len(cases))
print("MRR     :", round(sum(mrr(r, rel) for r, rel in cases) / len(cases), 3))

Follow-up: You have no labelled data and domain experts are expensive. How do you build the first evaluation set?

42. Your RAG bot gives wrong answers. How do you find out whether retrieval or generation is at fault?

Core

Do not tune blindly. Collect failing questions and inspect, for each, what was retrieved and what was generated. Then classify the failure:

flowchart TB W[Wrong answer] --> Q1{Was the needed passage retrieved at all?} Q1 -- no --> R1{Does the document exist in the index?} R1 -- no --> F1[Ingestion bug: parsing, missing file, stale index, permissions filter] R1 -- yes --> F2[Retrieval bug: chunking, embedding model, no hybrid search, bad query, filters, k too small] Q1 -- yes --> Q2{Was it ranked high and kept in the prompt?} Q2 -- no --> F3[Ranking bug: add a reranker, reduce noise, reorder context] Q2 -- yes --> Q3{Does the answer contradict or ignore the passage?} Q3 -- yes --> F4[Generation bug: stronger grounding prompt, lower temperature, better model, faithfulness check] Q3 -- no --> F5[Source itself is wrong, outdated or conflicting: fix content or add dates and precedence]

Typical order of fixes by payoff: fix parsing and chunking, add hybrid search and metadata filters, add a reranker, improve query rewriting, tighten the prompt and refusal rules, add faithfulness checks, and only then consider a bigger model or fine-tuning the embedder. After each change re-run the same evaluation set so you know it helped and did not break other cases.

Follow-up: Retrieval recall@10 is 95% but answer accuracy is 70%. Where do you look?

43. Your RAG system hallucinates even though the right context is retrieved. Why, and how do you fix it?

Core

When the evidence is present but the answer is still wrong, the fault is in generation or context assembly. Common causes and fixes:

• Model prior overrides context: the model 'knows' a different answer. Instruct it to use only the sources, quote them, and test with counter-factual passages.
• Too much or noisy context: the right passage is buried (lost in the middle) among distractors. Retrieve fewer, rerank, put the best evidence first or last.
• Conflicting passages: blend two versions of a policy. Attach dates and instruct precedence, or resolve conflicts in the index.
• Multi-hop reasoning failure: the answer needs facts from two passages. Decompose the question, or use an agentic loop that retrieves iteratively.
• Answer synthesis errors: numbers, units and dates get mangled. Ask for exact quotes, use structured extraction, or compute with tools.
• Over-eagerness to answer: no refusal path. Add unanswerable examples and an explicit 'not found' response.

Then add a safety net: an automatic faithfulness check (each claim must be supported by a retrieved span) and route low-confidence answers to a fallback or a human. Track the hallucination rate on the golden set as a release gate.

Follow-up: How would you implement the faithfulness check cheaply enough to run on every answer?

44. How do you handle structured data, tables and PDFs in a RAG pipeline?

Core

The pipeline is only as good as the parsing. PDFs are layout, not text: use a layout-aware parser or OCR/vision model to recover reading order, headings, multi-column text, headers and footers (strip them), and figures. Tables are the classic failure: flattening them into text destroys the row and column relationships. Options: convert each table to Markdown or JSON and keep it as a single chunk with its caption; embed a natural-language summary of the table for retrieval while returning the full table to the LLM; or use a vision-language model on the page image.

For truly structured data (SQL databases, spreadsheets), do not embed rows. Use text-to-SQL or a query tool: the LLM writes a query against a described schema (with sample values and column meanings), a validator checks it is read-only and safe, the database executes it, and the result goes back into the answer. Combine sources through a router that picks documents, SQL or an API per question.

Whatever the source, keep provenance (file, page, cell range) as metadata so answers can cite the exact place, and test with documents from your own corpus: parsing quality varies enormously between file types.

Follow-up: Your bot answers questions about a pricing table incorrectly. Where do you look first?

Sources and further reading

• Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• Khattab and Zaharia, ColBERT: Late Interaction
• Gao et al., HyDE: Precise Zero-Shot Dense Retrieval
• Es et al., RAGAS: Automated Evaluation of RAG
• Cormack et al., Reciprocal Rank Fusion (SIGIR 2009)
• AI Engineering interview questions (Outcome School, Apache-2.0)