Interview Special II: Agents, System Design and Live Scenarios

Agent and harness questions, five system design walkthroughs, live debugging scenarios and coding-round exercises.

• Advanced
• 45 min read
• 3 objectives

Before this lessonLesson 18: Interview Special I: LLM, Embeddings, RAG and Search Questions

Part II is where offers are won. It covers agents and harnesses, five full system design walkthroughs, live debugging scenarios (the 'real-time questions' interviewers throw at you), and the coding tasks that come up most. For system design, use this skeleton every time: clarify requirements, sketch the pipeline, dive into the risky parts, then cover evaluation, cost, latency and security. Saying the structure out loud is itself a signal of seniority.

Part A: Agents, tools and harnesses

1. What is an agent? How is it different from a workflow?

An agent is an LLM that decides its own next steps in a loop, using tools, until a goal is met. A workflow is LLM calls orchestrated by predefined code paths. Workflows are predictable and cheap; agents are flexible for open-ended tasks but slower, costlier and harder to control. I default to the simplest thing that works and add autonomy only where the steps genuinely cannot be listed in advance.

2. Describe the ReAct loop.

Reason, act, observe. The model emits a short thought and a tool call, the harness executes it and appends the observation, and the model continues until it produces a final answer or hits a limit. The loop plus tool feedback is what lets an agent correct itself.

3. How does tool calling work, and who executes the tool?

You describe tools with a name, description and JSON Schema. The model returns a structured request to call one. Your application executes it (never the model), appends the result as a tool message, and calls the model again. Because your code is in the middle, it is where you validate arguments, enforce permissions and log.

4. What is an agent harness?

Everything around the model that makes it an agent: the loop, tool registry and execution, context management (trimming, compaction, retrieval), memory, permissions and approvals, budgets and limits, streaming, and tracing. The same model behaves very differently in different harnesses, so quality is largely harness design.

5. What is context engineering?

Deciding exactly what the model sees each turn. Techniques: stable prefix for prompt caching, trimming and summarising tool outputs, compaction of old turns, just-in-time retrieval via search tools instead of preloading, scratchpad files, and delegating messy subtasks to sub-agents that return short results.

6. How do you stop an agent from looping or running away?

Hard step limit, token and dollar budgets, wall-clock timeouts, loop detection on repeated identical calls, a cancellation path, and clear stop conditions. Also design tools to return actionable errors so the model can change strategy instead of repeating.

7. Why do multi-step agents fail so often?

Errors compound: 95 percent per-step reliability is about 60 percent over ten steps and 36 percent over twenty. Plus context bloat, wrong tool choice, and prompt injection. Mitigate with short loops, verification steps, checkpoints, human approval on risky actions, and evals over full trajectories.

8. When would you use multiple agents?

When subtasks are parallelisable or need different tools, models or contexts: orchestrator-worker for broad research or repo-wide coding, handoffs for routing between specialised agents. Not by default: it multiplies cost and coordination complexity.

9. What is MCP and why does it exist?

The Model Context Protocol is an open JSON-RPC standard for connecting AI apps (clients) to tool and data servers, so an integration written once works in any MCP-capable app. Servers expose tools, resources and prompts. It solves the N-apps times M-integrations problem. Security still matters: a connected server is a capability and a trust boundary.

10. Explain the 'lethal trifecta'.

An agent that combines access to private data, exposure to untrusted content and the ability to communicate externally can be tricked by prompt injection into exfiltrating that data. Removing any one of the three breaks the attack, so I split privileges across separate agents and never give one agent all three.

11. LangGraph vs a plain while-loop?

A while-loop is fine for a simple agent. LangGraph models control flow as an explicit graph with shared state, conditional edges and checkpointing, which pays off with branching, retries, human approval pauses, resumable or long-running work, parallel branches and multi-agent composition, and it makes each step inspectable and testable.

12. LlamaIndex vs LangChain?

LlamaIndex is data-first: loaders, parsing, chunking, indexes, retrievers and query engines for RAG over varied sources. LangChain is a broad toolkit of model, prompt, parser, retriever and tool abstractions, with LangGraph for agents. They overlap, and many teams use pieces of each or neither.

13. How would you evaluate an agent?

Build tasks with verifiable outcomes and grade end state (did the tests pass, was the ticket resolved) plus trajectory quality (unnecessary steps, dangerous calls, cost, latency). Run each task multiple times because behaviour is stochastic, track pass rate and variance, and add every real failure as a new task.

14. How do you make an agent safe to deploy?

Least-privilege scoped tools, sandboxed execution, human approval for irreversible or external actions, input and output validation, prompt-injection testing, full audit logs, budgets and rate limits, and a kill switch.

Part B: System design walkthroughs

Design 1: A customer-support RAG chatbot for 10 million documents. Start with requirements: who are the users, latency target (say time-to-first-token under 1.5 seconds), accuracy and citation needs, multi-tenancy, languages, freshness. Then the pipeline:

• Ingestion: connectors, layout-aware parsing, dedupe, structure-based chunking (300 to 500 tokens, overlap), metadata (tenant, ACL, doc type, timestamps), embeddings plus a BM25 index. Incremental updates by content hash; deletions propagate.
• Storage: vector DB with HNSW and in-index metadata filters (10M chunks at 1,024 dims is roughly 40 GB raw, so consider quantisation), plus the text store and keyword index.
• Query path: auth, rate limit, query rewrite with history, hybrid retrieval (top 50), cross-encoder rerank (top 5), prompt assembly with citations, streamed generation over SSE, semantic cache in front.
• Safety: ACL filtering at retrieval, retrieved text treated as untrusted, PII redaction in logs, refusal when retrieval confidence is low.
• Quality: golden set in CI, online feedback, faithfulness judge on a sample, weekly failure review.
• Cost and latency: prompt caching, small model for rewrite and routing, larger model for hard questions, budget alerts.

Design 2: A coding agent that fixes failing tests. Tools: read file, search, edit, run tests (sandboxed). Loop with a step and token budget. Success signal is the test suite, which is what makes this a good agent problem. Context engineering: repo map, just-in-time file reads, trimmed test output. Safety: container sandbox, no network, allow-listed commands, changes proposed as a diff for review. Evaluate on a held-out set of real failing-test tasks and measure pass rate, cost and steps.

Design 3: Real-time voice assistant. Requirements: sub-second turn latency and interruptibility. Two-way audio needs WebRTC or WebSockets, not SSE. Pipeline: streaming speech-to-text, LLM streaming tokens, streaming text-to-speech, with voice-activity detection to allow barge-in (cancel generation when the user speaks). Discuss latency budget per stage, and what to do on dropped connections.

Design 4: 'Chat with your documents' for one user, 200 pages. Do not over-engineer. For one document, long-context with prompt caching may beat RAG on simplicity and quality; use RAG when the corpus exceeds context, cost matters, or you need citations at scale. Showing you can choose the simpler design is a strong signal.

Design 5: An AI support triage system (classification plus actions). This is a workflow, not an agent: classify (small model, structured output), route, look up order data via read-only tools, draft a reply, and require human approval before refunds above a threshold. Discuss evaluation with a labelled ticket set and monitoring for drift.

Part C: Live scenarios, the 'real-time questions'

Interviewers give you a situation and watch how you reason. Talk through: measure, hypothesise, isolate, fix, verify.

Scenarios

Scenario 1. 'Our RAG bot is wrong on about 8% of questions.'

First I would not touch the prompt. I would pull a sample of failing questions and, for each, read the retrieved chunks. That splits failures into: not indexed or badly parsed, not retrieved (chunking, no keyword matching for IDs, vague query), retrieved but ignored (prompt or conflicting context) or genuinely unanswerable (should have refused). I would compute recall@k on a labelled set to confirm retrieval is the main problem, then apply the fix for the biggest bucket, usually hybrid search, better chunking and a reranker, and re-measure.

Scenario 2. 'Responses take 12 seconds. Fix it.'

Break latency into spans with tracing: retrieval, rerank, time to first token, generation. Then act on the largest: stream tokens so users see output in under a second; cache or shrink the prompt and use prompt caching to cut time-to-first-token; parallelise independent steps (retrieval and query rewriting where possible); use a smaller model for rewrite and routing; cap output length; run reranking on fewer candidates. Report time-to-first-token and total time separately.

Scenario 3. 'The bill is 10 times what we forecast.'

Look at cost per request by feature and by prompt length. Common culprits: ever-growing conversation history re-sent each turn, too many retrieved chunks, an agent looping, retries without backoff, and everything going to the biggest model. Fixes: compaction and history summarisation, fewer chunks after reranking, step and budget limits, prompt and semantic caching, model routing, output caps, and per-user budgets with alerts.

Scenario 4. 'A user saw another customer's document in an answer.'

Treat as a security incident. Contain (disable the path if needed), then find the root cause: almost certainly retrieval without a tenant or permission filter, a shared cache key ignoring the tenant, or logs and embeddings mixing tenants. Fix by enforcing filters inside the vector query, namespacing caches by tenant, adding automated cross-tenant leakage tests, and reviewing what else shares the same index.

Scenario 5. 'The agent keeps repeating the same failing tool call.'

Add loop detection and a step limit, return richer tool errors that suggest alternatives, add a reflection step ('what have you tried, what will you change?') and escalate to a human after N failures. Examine the tool description: unclear tool docs are a frequent cause.

Scenario 6. 'Quality dropped after the model provider updated their model.'

That is why we pin model versions and keep an eval suite. Run the eval on old versus new, find which categories regressed, adjust prompts, and roll forward only when scores are back. Alert on eval scores and on production quality proxies like thumbs-down rate.

Scenario 7. 'The model ignores an instruction in the middle of a long prompt.'

Long context degrades in the middle. Shorten the context by retrieving fewer, better chunks, move critical instructions to the start and repeat key ones at the end, structure the prompt with clear sections, and test whether a smaller prompt reproduces the behaviour.

Scenario 8. 'Search cannot find product code XJ-4471 but finds everything else.'

Embeddings blur exact identifiers. Add BM25 or keyword search and fuse with RRF (hybrid), keep the code as its own token (do not split on hyphens during tokenisation), and consider a metadata or exact-match lookup path for identifiers.

Part D: The coding round

Typical tasks: implement cosine similarity and top-k retrieval, write RRF, chunk text with overlap, build BM25, implement a retry with backoff, parse an SSE stream, or write a tool-call loop. You have practised most of them in this course. Here are two more, with the traps interviewers look for.

Task: chunk text with overlap. Edge cases: overlap greater than or equal to chunk size (infinite loop), empty text, last short chunk.

def chunk_words(text, size, overlap):
    if size <= 0:
        raise ValueError("size must be positive")
    if not 0 <= overlap < size:
        raise ValueError("overlap must be >= 0 and < size")   # else the window never advances
    words = text.split()
    step = size - overlap
    chunks = []
    for start in range(0, max(len(words), 1), step):
        piece = words[start:start + size]
        if piece:
            chunks.append(" ".join(piece))
        if start + size >= len(words):
            break
    return chunks

text = "one two three four five six seven eight nine ten"
for c in chunk_words(text, size=4, overlap=1):
    print("|", c)
print(chunk_words("", 4, 1))

| one two three four
| four five six seven
| seven eight nine ten
[]

Task: efficient top-k. Do not sort everything when you only need the best k. A heap does it in O(n log k):

import heapq

def top_k(scored_items, k):
    """scored_items: iterable of (score, item). Returns the k best, highest score first."""
    return heapq.nlargest(k, scored_items)

docs = [(0.31, "shipping"), (0.92, "refunds"), (0.55, "warranty"), (0.87, "returns"), (0.12, "careers")]
print(top_k(docs, 2))

[(0.92, 'refunds'), (0.87, 'returns')]

Behavioural questions specific to AI roles

• "Tell me about a time an AI system failed in production." Use the structure: what happened, how you detected it, root cause, fix, and the eval or guardrail you added so it cannot recur.
• "How do you decide whether AI is the right solution?" Clear success metric, tolerance for errors, a non-AI baseline to beat, and a human fallback. Sometimes a rule or a SQL query wins.
• "How do you keep up with this field?" Read primary sources (model cards, provider changelogs, key papers), rebuild things from scratch to understand them, and keep a personal eval set to test new models on your own tasks.
• "How do you explain hallucinations to a non-technical stakeholder?" It is a very well-read autocomplete that always answers confidently. We reduce it by giving it the documents to read and telling it to admit when the answer is not there, and we measure how often it still goes wrong.

Red-flag answers to avoid

• "Just fine-tune it on our data" as the answer to a knowledge problem.
• "Set temperature to 0 and it will not hallucinate."
• "Use the biggest model for everything."
• Proposing an autonomous agent for a task a fixed workflow handles.
• No mention of evaluation, cost, latency or security. Always bring these up unprompted.
• Trusting the model to enforce permissions or hide data.

# Write your solution here