AI Interview Questions · Lesson 10 of 10
AI System Design, Multimodal and Interview Scenarios
Whiteboard designs for RAG, voice agents and coding agents; CLIP and diffusion; API vs self-host; ROI, latency vs quality and how to talk about risk with non-technical stakeholders.
- Advanced
- 15 min read
- 11 questions
Before this lessonLesson 9: Evaluation, Safety, Bias and Responsible AI
What you will learn
- Whiteboard end-to-end AI products with clear components, SLAs and failure modes
- Explain multimodal models (CLIP, VLMs, diffusion, voice stacks) with production trade-offs
- Answer behavioural and scenario questions with numbers, risks and decision criteria
Your Progress
0 of 10 lessons 0%
- Lessons0 / 10
- Completed0
- Est. time left~ 3 hours
Create a free account to keep your progress on every device.
Senior AI interviews almost always include a system design round: design ChatGPT-like chat, enterprise RAG, a voice agent, or a coding agent. Interviewers want components, data flow, latency and cost budgets, and what breaks first.
Multimodal and scenario questions test whether you can choose architecture under constraints and communicate risk. Draw the box diagram, state assumptions, and name one metric you would watch after launch.
The 11 questions in this lesson
- Design a production RAG assistant over millions of documents with per-user permissions. What are the main components and failure modes?
- Design a real-time voice AI agent (phone or app). Budget the latency and compare cascaded ASR+LLM+TTS with speech-to-speech models.
- Design an AI coding agent (IDE or CLI). What matters more: the model or the harness?
- What are multimodal AI models? How do vision-language models and CLIP-style embeddings work?
- Explain diffusion models for image generation at a high level. How do you speed them up and control them?
- What is FlashAttention, and why does it matter for long-context training and serving?
- API-hosted LLM versus self-hosting an open-source model: how do you decide?
- How do you measure ROI of an AI feature, and how do you choose between latency and quality?
- What is the Turing Test, and how is modern AI evaluation different from it?
- Explain a confusion matrix and when accuracy is the wrong headline metric.
- Your PM wants to ship an AI feature with a known 15% hallucination rate on edge cases. How do you respond?
100. Design a production RAG assistant over millions of documents with per-user permissions. What are the main components and failure modes?
System design
Start with requirements: query types, freshness, languages, citation needs, p95 latency, and who may see which documents. Then draw the pipeline and call out auth early — retrieval without ACLs is a security bug.
- Ingestion: parse PDFs and HTML, chunk with overlap, embed, write to a vector store plus a keyword index, keep document IDs and ACL tags in metadata.
- AuthZ: filter by user/group at retrieval time (preferred) or post-filter; never rely on the LLM to hide secrets.
- Retrieve → rerank → generate: hybrid search, top-k rerank, grounded prompt with "answer only from sources" and numbered citations.
- Eval and ops: golden set for faithfulness and recall@k, tracing, cost dashboards, kill switch.
Failure modes interviewers expect: wrong chunking, stale indexes after deletes, ACL leaks, lost-in-the-middle, citation fabrication, and cost blow-ups from huge contexts. Mitigate with hybrid search, reranking, permission tests, freshness jobs, and context budgets.
Follow-up: How would you prove a user cannot retrieve a document they are not allowed to see?
101. Design a real-time voice AI agent (phone or app). Budget the latency and compare cascaded ASR+LLM+TTS with speech-to-speech models.
System design
A usable voice agent needs barge-in, low time-to-first-audio, and clear turn-taking. Cascaded stacks are still the common production path.
| Stage | Typical budget | Notes |
|---|---|---|
| VAD / endpointing | 50–150 ms | Too eager cuts users off; too slow feels laggy |
| ASR (streaming) | 200–600 ms to usable text | Partials unlock earlier LLM starts |
| LLM TTFT | 200–800 ms | Small/fast model or speculative decoding helps |
| TTS time-to-first-byte | 100–400 ms | Stream audio; do not wait for full sentence |
| Network / jitter buffer | 50–200 ms | WebRTC or telephony stack |
- Cascaded ASR → LLM → TTS: modular, swappable vendors, easier tool use and logging; latency stacks and error compounds (ASR typo → wrong tool).
- Native speech-to-speech: lower theoretical latency and better prosody; harder to debug, fewer tool ecosystems, and still maturing for enterprise control.
Always design for barge-in (cancel TTS on new speech), timeouts, and a fallback to hold music or a human queue when any stage fails.
Follow-up: Where would you spend engineering time first if users complain the bot talks over them?
102. Design an AI coding agent (IDE or CLI). What matters more: the model or the harness?
System design
A coding agent is a model plus a harness: tools (read/write files, shell, tests, search), a loop with budgets, context packing, and safety rails. In interviews, strong candidates argue that harness quality often dominates raw model IQ for shipping patches.
- Context engineering: retrieve the right files, summarise long logs, keep a working set under the window.
- Verification: run tests/linters before declaring success; treat failing tests as observations, not as the end.
- Safety: sandbox shell, deny secrets paths, require approval for destructive git or production deploys.
- Eval: SWE-bench-style tasks plus your private repos; measure resolve rate, cost and human edits needed.
Trade-off line to say aloud: a weaker model in a great harness (good retrieval, tight tools, strong tests) often beats a frontier model with a naive chat loop.
Follow-up: How would you stop the agent from looping on the same failing edit?
103. What are multimodal AI models? How do vision-language models and CLIP-style embeddings work?
Core
Multimodal models handle more than text: images, audio, video, documents. A vision-language model (VLM) typically encodes images into tokens or embeddings that live in the same space the language model can attend over.
- CLIP-style contrastive training: pull matching image–text pairs together and push mismatches apart in a shared embedding space. That enables zero-shot classification and cross-modal search (text query → images).
- Image embeddings: vectors for photos or page screenshots; used for duplicate detection, visual search and multimodal RAG.
- Document VLMs: must handle layout, tables and multi-page inputs — often with OCR + layout features or native page encoders.
Failure mode: the model ignores the image and answers from prior text bias. Fix with prompts that force visual grounding, better image resolution, and eval sets that change only the image.
Follow-up: How would you build search that finds products by a photo and a text constraint together?
104. Explain diffusion models for image generation at a high level. How do you speed them up and control them?
Core
Diffusion models learn to reverse a gradual noising process: start from noise, iteratively denoising toward an image that matches a text condition (and optional controls).
- Sampling steps: more steps → usually better quality, higher latency/cost. Distilled or consistency-style samplers cut steps dramatically for production.
- Guidance: classifier-free guidance strength trades adherence to the prompt against diversity; too high looks oversaturated or brittle.
- Control: ControlNet, reference images, IP-adapters, inpainting masks and structured layouts improve precision when plain text is underspecified.
- Ops: batch on GPUs, cache text encoders, offer size presets, moderate inputs/outputs, and watermark or label synthetic media where required.
Interview soundbite: diffusion is iterative denoising guided by text embeddings; production is mostly about step count, guidance, caching and safety filters.
Follow-up: A user says the model ignores 'put the logo top-right'. What knobs do you change first?
105. What is FlashAttention, and why does it matter for long-context training and serving?
Deep dive
Standard attention materialises a full N×N score matrix, which is heavy on GPU memory bandwidth. FlashAttention computes attention in tiles that stay in fast on-chip memory, reducing HBM reads/writes without changing the math (exact attention, not an approximation).
- Why it matters: longer sequences become feasible; training and prefills use less memory and often run faster.
- What it does not fix alone: decode is still dominated by reading weights and the KV cache; you still need GQA/MQA, PagedAttention, quantisation and batching for serving.
- Related ideas: FlashAttention-2/3 improve parallelism; other kernels fuse ops similarly across vendors.
Say clearly: FlashAttention is an IO-aware implementation of attention, not a different model architecture.
Follow-up: If decode is still slow after enabling FlashAttention, what do you profile next?
106. API-hosted LLM versus self-hosting an open-source model: how do you decide?
Scenario
Frame it as a decision table, not a religion.
| Factor | Prefer API | Prefer self-host |
|---|---|---|
| Time to market | Days; no GPU ops | Weeks+ for serving, scaling, on-call |
| Data sensitivity | If vendor zero-retention and DPA fit | Strict residency, air-gapped, or regulated workloads |
| Traffic shape | Spiky or unknown demand | Steady high volume where GPUs amortise |
| Customisation | Prompting, light fine-tunes via vendor | Heavy LoRA/domain models, custom kernels |
| Latency control | Good enough regional endpoints | Need tight p95 on your VPC |
| Cost at scale | Pay per token; fine early | Often cheaper past a high, stable QPS |
Hybrid is common: API for peak and hard tasks, self-host a smaller model for easy classification or rewrite. Revisit the choice when volume, risk or latency SLOs change.
Follow-up: What numbers would you collect in a two-week spike to decide?
107. How do you measure ROI of an AI feature, and how do you choose between latency and quality?
Scenario
ROI needs a baseline and a decision metric tied to money or time: containment rate in support, minutes saved per ticket, conversion lift, engineer-hours automated, fraud dollars avoided. Subtract incremental model cost, review cost and incident risk.
- Define the counterfactual: what happens without the feature (human-only, rules, previous model).
- Instrument: online metric + guardrails (safety rate, escalation rate, latency, cost per success).
- Run an A/B or phased rollout with a pre-registered success threshold.
- Include failure cost: one bad medical/legal answer can wipe months of savings.
Latency vs quality: fix the user-visible SLA first (for example p95 TTFT < 2 s for chat). Then use cascading: small/fast model or cache for easy cases, larger model for hard ones; stream tokens so perceived latency drops even if total time is similar.
def monthly_roi(successes, value_per_success, model_cost, review_hours, review_rate):
benefit = successes * value_per_success
cost = model_cost + review_hours * review_rate
return benefit - cost
# Example: 8,000 auto-resolved tickets/month, $4 saved each, $1.2k model spend, 40h review at $50/h
print(monthly_roi(8000, 4.0, 1200, 40, 50))Follow-up: Your offline quality rose 4% but p95 latency doubled. Ship, cascade, or roll back?
108. What is the Turing Test, and how is modern AI evaluation different from it?
Warm-up
The Turing Test (imitation game, 1950) asks whether a machine's text behaviour is indistinguishable from a human's in conversation. It is a historical landmark, not a production acceptance test.
Modern evaluation is task-centred and quantitative: exact-match and F1 for extraction, pass@k for code, faithfulness for RAG, human or LLM-as-judge rubrics for open-ended chat, plus online A/B metrics. Passing as "human-like" can even be undesirable when you need calibrated uncertainty, citations and refusals.
Bridge line for interviews: the Turing Test asked about indistinguishability; we ship systems measured on usefulness, safety and reliability under explicit SLOs.
Follow-up: Why can a model that "sounds human" still fail a customer-support eval?
109. Explain a confusion matrix and when accuracy is the wrong headline metric.
Warm-up
For binary classification, the confusion matrix counts true positives (TP), false positives (FP), true negatives (TN) and false negatives (FN).
# Fraud detection toy example: 980 legit, 20 fraud
TP, FP, FN, TN = 16, 30, 4, 950
accuracy = (TP + TN) / (TP + FP + FN + TN)
precision = TP / (TP + FP)
recall = TP / (TP + FN)
f1 = 2 * precision * recall / (precision + recall)
print("accuracy %.3f precision %.3f recall %.3f F1 %.3f" % (accuracy, precision, recall, f1))
print("accuracy looks fine while missing 20% of fraud (FN=4)")- Precision: of predicted positives, how many were right — costly FP world (spam quarantine, wrong arrests).
- Recall: of actual positives, how many you caught — costly FN world (cancer, fraud, safety).
- F1: harmonic mean when you need balance; for ranking use PR-AUC or ROC-AUC.
Accuracy fails on imbalanced data because predicting the majority class scores high while missing the rare class that matters.
Follow-up: In medical screening, would you rather raise precision or recall first, and why?
110. Your PM wants to ship an AI feature with a known 15% hallucination rate on edge cases. How do you respond?
Behavioral
Do not block with vibes — reframe as risk, users and mitigations.
- Clarify the edge cases: how often do they appear in production traffic? 15% of 0.5% traffic differs from 15% of all sessions.
- Map harm: wrong tone vs wrong medical/legal/financial advice. Severity drives whether you need a hard gate.
- Propose controls: refuse when retrieval is weak, show citations, human review for high-risk intents, feature flag, shadow mode, kill switch.
- Agree a metric and date: for example "ship behind a flag to 5% users if faithfulness ≥ 95% on the golden set and zero critical red-team findings".
- Write it down: short risk note for stakeholders so the trade-off is explicit.
Tone: partner, not blocker. Offer a safer path that still learns from real traffic.
Follow-up: What would make you insist on delaying launch despite PM pressure?