Interview Special I: LLM, Embeddings, RAG and Search Questions

Forty-plus real interview questions with model answers on LLMs, prompting, embeddings, vector search, BM25, hybrid search and RAG.

• Intermediate
• 40 min read
• 3 objectives

Before this lessonLesson 17: Shipping AI to Production

Welcome to the interview special. Everything you learned in this course is now converted into the questions interviewers actually ask, with the answer shape that impresses them. A general rule before we start: interviewers rarely want definitions, they want judgement. "What is RAG?" is a warm-up; "your RAG bot is wrong 8% of the time, what do you do?" is the real interview. Answer every question in three moves: define it in one sentence, give a concrete example, then state the trade-off or when it fails.

Part A: LLM fundamentals

1. What is an LLM and how does it work?

A large language model is a neural network (almost always a transformer) trained on huge amounts of text to predict the next token. At inference it repeatedly produces a probability distribution over the next token, samples one, appends it, and repeats. Fluency comes from scale and training data. Trade-off to mention: it optimises plausibility, not truth, which is the root of hallucination.

2. What is a token and why does it matter?

A token is a sub-word chunk the tokenizer produces (about 4 English characters, or 0.75 of a word). Tokens matter because they set cost (APIs bill per token), limits (the context window is measured in tokens) and quirks (letter counting and arithmetic are unreliable because the model never sees individual characters). Non-English text and code usually cost more tokens per word.

3. What is a context window? Is it the same as memory?

It is the maximum tokens the model can attend to in one request, covering instructions, history, documents and the reply. It is not long-term memory: the model is stateless, and 'remembering' is the application re-sending history or stored notes. Long contexts cost money on every call, and models often use the middle of a long prompt less well ('lost in the middle').

4. Explain temperature, top-k and top-p.

They control how the next token is sampled from the model's probabilities. Temperature rescales the distribution: near 0 it is nearly deterministic (good for extraction, code and factual answers), higher values flatten it for variety. Top-k restricts choices to the k likeliest tokens; top-p keeps the smallest set whose probabilities sum to p, adapting to the model's confidence. I use low temperature for accuracy and moderate temperature plus top-p for creative tasks. Temperature 0 reduces randomness but is still not a guarantee of identical outputs across runs.

5. Pre-training vs fine-tuning vs RLHF?

Pre-training learns language and world knowledge by next-token prediction on trillions of tokens. Fine-tuning (instruction tuning) trains further on curated examples so the model follows instructions. RLHF and similar preference methods (DPO, RLAIF) push it toward answers humans or judges prefer: helpful, honest, safe. Together they turn a base autocomplete model into an assistant.

6. When would you fine-tune instead of using RAG?

Fine-tune to change behaviour: consistent tone, strict output format, domain style, or making a small model cheap and fast at one task. Use RAG for knowledge, especially facts that change, are private or need citations. Fine-tuning teaches style much better than facts, and facts go stale. Often the answer is both: RAG for facts, light tuning or prompting for behaviour.

7. What is a hallucination and why does it happen?

A fluent, confident output that is false or unsupported. It happens because the model is trained to produce plausible continuations, not verified truth; rare or missing facts get filled with plausible guesses; models are tuned to be helpful rather than to abstain; and there is a training cut-off. Mitigations: ground answers in retrieved documents, allow 'I do not know', require citations, lower temperature, use tools for maths and lookups, verify with a second pass, and keep humans in the loop for high stakes.

8. How would you measure hallucination?

Split into factual and faithfulness errors. For RAG the practical metric is faithfulness: is every claim in the answer supported by the retrieved context? Measure with an LLM-as-judge or NLI model over a golden set, including questions that have no answer in the data to test refusal accuracy. Calibrate the judge against human labels.

9. What is the difference between a base model, an instruct model and a reasoning model?

A base model only continues text. An instruct/chat model is tuned to follow instructions in a conversation. A reasoning model additionally spends extra tokens 'thinking' before answering, trading latency and cost for better multi-step accuracy. Choose by task: simple extraction wants a small instruct model; hard multi-step problems may justify a reasoning model.

10. What are the main levers for reducing LLM cost and latency?

Shrink the prompt (fewer, better chunks), cap output tokens, use prompt caching for repeated prefixes, cache answers semantically, route easy requests to a small model, stream so perceived latency drops, batch non-urgent work, and set budgets and alerts. Measure time-to-first-token and tokens per second separately.

Part B: Prompting and outputs

11. What makes a good prompt?

Clear task, relevant context, the desired output format (ideally with an example), explicit handling of uncertainty ('if not in the text, say so'), and delimiters that separate instructions from data. Iterate against an eval set, not by feel.

12. Few-shot vs zero-shot vs chain-of-thought?

Zero-shot just asks. Few-shot includes examples, which teach format and style quickly. Chain-of-thought asks for step-by-step reasoning, helping on multi-step problems. Reasoning models do this internally, so explicit CoT prompting matters less on them.

13. How do you get reliable JSON from a model?

Use structured outputs or a tool schema so the API constrains generation to your JSON Schema, otherwise JSON mode with a schema described in the prompt. Always validate on your side (pydantic/jsonschema), and on failure send the error back to the model for a retry. Keep schemas small and use enums.

14. What is prompt injection and how do you defend against it?

Untrusted text (a web page, email, document) containing instructions the model may obey. There is no prompt-only fix. Defend architecturally: least-privilege tools, no single agent holding private data plus untrusted input plus an outbound channel, human approval for consequential actions, output and argument validation, sandboxing, audit logging and red-team testing. Pattern filters are a weak extra layer only.

15. What is a system prompt and can users see or override it?

It is the standing instruction message that sets role, rules and format. Users can often coax it out, so never put secrets in it, and never rely on it alone for security; enforce important rules in code.

Part C: Embeddings and vector search

16. What is an embedding?

A vector of numbers, produced by a model, that represents the meaning of a piece of text (or image, audio, code) so that semantically similar inputs land near each other. It converts a fuzzy question (are these related?) into geometry (how close are these points?).

17. Cosine similarity vs dot product vs Euclidean distance?

Cosine is the angle between vectors and ignores magnitude, the default for text. Dot product is cosine times the lengths; for normalised vectors they are identical and the dot product is cheaper. Euclidean is straight-line distance and is affected by magnitude. Use whatever metric the embedding model was trained with, and normalise if it recommends it.

18. Why can't you mix embeddings from two different models?

Each model defines its own coordinate space, so vectors from different models are not comparable. If you change the model (or major version), you must re-embed the whole corpus and the queries with the same model.

19. How do you choose an embedding model?

Test on your own retrieval eval set rather than trusting leaderboards. Consider quality on your domain and language, dimensions (storage and speed), context length per input, cost or self-hosting needs, and licence. Also consider whether a truncatable (Matryoshka) model can cut storage.

20. What is chunking and how do you pick chunk size?

Splitting documents into pieces that are embedded and retrieved individually, because one vector per long document is blurry. Typical starting point is 200 to 800 tokens with 10 to 20 percent overlap, splitting on structure (headings, paragraphs) rather than raw character counts. Tune by measuring retrieval recall on a golden set. Smaller chunks are precise but lose context; larger chunks keep context but dilute the vector and waste prompt space.

21. What is ANN search and why not exact search?

Approximate nearest neighbour search uses an index (HNSW, IVF, PQ) to find very likely nearest vectors while examining a small fraction of the data. Exact search is O(n) per query, too slow at millions of vectors. You trade a little recall for orders of magnitude of speed, tuned via parameters like efSearch.

22. Explain HNSW in a few sentences.

A multi-layer proximity graph. Upper layers are sparse 'highways' that let the search jump quickly across the space; lower layers are dense. Search enters at the top, greedily moves to the closest neighbour, drops a layer, and repeats, ending in the densest layer with a local best-first search. It is fast and accurate but memory-hungry.

23. How do you handle metadata filtering and access control in vector search?

Store metadata (tenant, permissions, date, type) with each chunk and apply it in the query, ideally as an in-index filter rather than post-filtering, which can leave too few results. Enforce permissions at retrieval time; never retrieve text a user may not see and hope the model hides it.

24. When would you pick pgvector over a dedicated vector database?

When you already run PostgreSQL, the corpus is up to a few million vectors, and you value transactions, SQL joins and one less system to operate. Move to a dedicated engine for very large scale, heavy filtering, multi-tenant isolation at scale or specialised features.

Part D: Keyword search, hybrid search and reranking

25. What is BM25?

A probabilistic keyword ranking function. It scores a document by summing, over query terms, an IDF weight (rare terms count more) times a saturating term-frequency factor, normalised by document length. Two parameters: k1 controls frequency saturation and b controls length normalisation. It is fast, needs no model, and excels at exact terms, names and identifiers.

26. TF-IDF versus BM25?

Both weight terms by frequency and rarity. TF-IDF uses raw or log term frequency which keeps growing, and has no length normalisation by default. BM25 saturates term frequency (repeating a word 40 times is not 40 times better) and penalises long documents. BM25 usually ranks better.

27. Why does keyword search still matter when we have embeddings?

Embeddings blur exact tokens: error codes, SKUs, names, legal citations and rare jargon. BM25 finds them precisely. Embeddings handle paraphrase and synonyms that BM25 misses. They fail in opposite ways, so combining them is stronger than either.

28. What is hybrid search and how do you combine the scores?

Running keyword and vector retrieval and merging the results. Scores are on different scales, so the common method is Reciprocal Rank Fusion (sum of 1/(k+rank) across lists, k around 60), which uses ranks only. Alternatively normalise scores and take a weighted sum, which needs tuning.

29. Bi-encoder vs cross-encoder?

A bi-encoder embeds query and document separately, so document vectors are precomputed and search is fast, but the two never interact. A cross-encoder reads query and document together and outputs a relevance score, which is more accurate but needs one model pass per pair, so it is used only on a shortlist. Standard pipeline: bi-encoder/hybrid retrieval for recall, cross-encoder reranking for precision.

30. What is query rewriting / multi-query / HyDE?

Techniques that improve the query before retrieval. Rewriting turns a follow-up into a standalone question using chat history. Multi-query generates paraphrases and merges results. HyDE has the model write a hypothetical answer and searches with that, because answers resemble documents more than questions do. They cost an extra LLM call and can drift, so measure the gain.

Part E: RAG

31. Explain RAG end to end.

Offline: parse documents, chunk, embed, store in a vector index (plus BM25) with metadata. Online: take the question, optionally rewrite it, retrieve candidates with hybrid search, rerank to a handful, build a prompt with instructions and the retrieved context, have the LLM answer with citations (streaming), and optionally check faithfulness. The model stays frozen; knowledge lives in the index, so updates are just re-indexing.

32. RAG vs fine-tuning vs long context?

RAG for changing, private or citable knowledge. Fine-tuning for behaviour and format. Long context for one or few documents where simplicity wins, but it costs tokens on every call, degrades in the middle and does not scale to a whole corpus. Frequently combined.

33. A RAG bot gives a wrong answer. How do you debug it?

Locate the failing stage with logged intermediate data: (1) was the answer in the index at all (ingestion or parsing bug)? (2) was it in the top-k retrieved (chunking, embedding, missing BM25, filters, vague query)? (3) did it survive reranking and fit in the prompt? (4) if it was in the prompt, did the model use it (prompt, conflicting chunks, temperature)? Most failures are retrieval, so I inspect the retrieved chunks first.

34. How do you evaluate a RAG system?

Build a golden set of 50 to 200 real questions with reference answers and the chunks that contain them, including unanswerable ones. Evaluate retrieval separately (recall@k, MRR, nDCG) and generation (faithfulness, relevance, correctness, refusal accuracy), using LLM-as-judge calibrated on human labels. Run it in CI and turn every production failure into a new test case.

35. What are recall@k and MRR?

Recall@k is the fraction of relevant chunks that appear in the top k results, the key retrieval metric because a missing chunk cannot be used. MRR is the mean of 1/rank of the first relevant result, rewarding putting the right chunk near the top.

36. How do you make RAG answer 'I don't know' instead of guessing?

Prompt: answer only from the context, otherwise reply with a fixed refusal phrase. Add a retrieval-confidence threshold (if top scores are low, refuse before calling the model), require citations, evaluate refusal accuracy on unanswerable questions, and monitor for 'confident but unsupported' answers.

37. What are parent-child retrieval and contextual chunking?

Parent-child indexes small chunks for precise matching but returns the larger parent section to the model for context. Contextual chunking prepends each chunk with a short generated description of where it sits in the document, improving both embedding and keyword matching.

38. What is GraphRAG and when is it useful?

It builds a knowledge graph of entities and relations (and community summaries) from the corpus and retrieves over the graph. It helps with multi-hop, relationship and 'summarise the whole corpus' questions that flat chunk retrieval handles poorly, at higher ingestion cost and complexity.

39. How do you handle tables, PDFs and images in RAG?

Parsing quality often decides success. Use layout-aware parsers to keep tables as structured text (Markdown/CSV) rather than scrambled lines, extract headings for metadata, OCR scans, and use vision models to describe charts and figures. Verify by reading the parsed output, not the PDF.

40. How do you keep a RAG index fresh?

Incremental ingestion driven by change events or hashes, upserting new or changed chunks and deleting stale ones by document ID, with embedding-model version tracked so a model change triggers a full re-embed. Store timestamps so retrieval can prefer newer content.

41. How do you secure a multi-tenant RAG system?

Filter by tenant and user permissions in the retrieval query, isolate indexes or namespaces for strict tenants, never rely on the prompt to hide data, sanitise logs, treat retrieved text as untrusted (indirect prompt injection), and test cross-tenant leakage explicitly.

Warm-up coding question: cosine similarity

Interviewers love this one because it has edge cases. Write cosine similarity, handle the zero vector, and explain what the answer means.

import math

def cosine(a, b):
    if len(a) != len(b):
        raise ValueError("vectors must have the same length")
    dot = sum(x * y for x, y in zip(a, b))
    na = math.sqrt(sum(x * x for x in a))
    nb = math.sqrt(sum(y * y for y in b))
    if na == 0 or nb == 0:          # the edge case interviewers look for
        return 0.0
    return dot / (na * nb)

print(cosine([1, 0], [1, 0]))    # same direction  -> 1.0
print(cosine([1, 0], [0, 1]))    # unrelated       -> 0.0
print(cosine([1, 0], [-1, 0]))   # opposite        -> -1.0
print(cosine([0, 0], [1, 1]))    # zero vector     -> 0.0 (not a crash)

1.0
0.0
-1.0
0.0

# Write your solution here