Inference, Serving Performance and LLMOps

TTFT and throughput, continuous batching, PagedAttention, speculative decoding, quantisation, parallelism, caching, cost control, failover and observability.

• Advanced
• 17 min read
• 11 questions

Before this lessonLesson 7: Fine-Tuning, LoRA and Alignment (RLHF, DPO)

Companies that run models at scale care intensely about this lesson. It is where "I can call an API" becomes "I can keep a feature fast, available and affordable for a million users". The questions reward numbers and named techniques: time-to-first-token, tokens per second, continuous batching, KV cache, p99.

Even if you will only call hosted APIs, the same ideas apply to caching, routing, cost control and failure handling, so do not skip it.

The 11 questions in this lesson

78. How do you serve an LLM in production? Sketch the architecture.
79. Define TTFT, inter-token latency and throughput. Why is decoding memory-bandwidth-bound? Do the roofline maths.
80. What is continuous batching and PagedAttention, and why do engines like vLLM outperform naive serving?
81. What is speculative decoding and how much speed-up can it give?
82. Explain model quantisation. Compare INT8, INT4, FP8, PTQ and QAT. What happens when you push to 2 to 4 bits?
83. What are tensor, pipeline, data and expert parallelism? How do you serve a model larger than one GPU?
84. How do prompt caching and semantic caching work, and what are the risks?
85. How do you estimate and reduce the cost of running an LLM feature?
86. How do you handle rate limits, provider outages and failover? Explain retries with exponential backoff and circuit breakers.
87. What should you monitor and log in a production LLM application? What is LLM observability?
88. How do you release model, prompt and pipeline changes safely? What does CI/CD look like for AI applications?

78. How do you serve an LLM in production? Sketch the architecture.

Core

The pieces around the model matter as much as the model. A typical stack:

flowchart LR C[Clients] --> GW["API gateway: auth, quotas, rate limits, request validation"] GW --> CA{"Response cache: exact and semantic"} CA -- hit --> C CA -- miss --> RT["Router: pick model by task, cost, load, region"] RT --> ENG["Inference engine: vLLM, SGLang or TensorRT-LLM<br/>continuous batching, KV cache, quantised weights"] ENG --> GPU[(GPU pool with autoscaling)] RT -. fallback .-> EXT[Backup provider or smaller model] ENG --> OBS["Observability: traces, token counts, latency, cost, quality signals"] GW --> GR[Guardrails: input and output filters]

• Gateway: authentication, per-user and per-tenant quotas, token-based rate limiting, request size checks.
• Engine: use an optimised server (vLLM, SGLang, TensorRT-LLM, or llama.cpp for CPU and edge) rather than a naive framework loop: they provide continuous batching, paged KV cache, quantisation and parallelism.
• Streaming: return tokens as they are generated (server-sent events) so users see output immediately; it changes perceived latency far more than raw speed.
• Scaling: autoscale on queue depth or GPU utilisation, keep warm capacity to avoid cold starts (loading tens of GB of weights takes minutes), and separate latency-sensitive traffic from batch jobs.
• Reliability: health checks, timeouts, retries with backoff, circuit breakers and a fallback path.
• Safety and observability: guardrails, logging with PII controls, metrics and tracing, cost attribution per feature.

Build-versus-buy: hosted APIs give the best quality and zero ops at variable cost; self-hosting pays off for privacy or data residency, very high steady volume, latency control, or custom fine-tuned models.

Follow-up: When does self-hosting an open model become cheaper than an API, and what hidden costs do people forget?

79. Define TTFT, inter-token latency and throughput. Why is decoding memory-bandwidth-bound? Do the roofline maths.

Deep dive

• TTFT (time to first token): from request to first output token. Dominated by queueing plus the prefill of the prompt (compute-bound).
• Inter-token latency / TPOT: time between subsequent tokens. Set by the decode step (memory-bound). Users read at perhaps 5 to 10 tokens per second, so 20 to 50 tokens per second per user feels instant.
• Throughput: total tokens per second across all users on the hardware. What determines cost per token.

Example: an 80 GB H100 has about 3.35 TB/s of bandwidth. A 70B model in FP16 (140 GB) does not even fit on one GPU; in FP8 (70 GB) the ceiling is about 3.35e12 / 70e9 = 48 tokens per second for a single stream; in 4-bit (35 GB) about 95. A 8B model in FP16 (16 GB) reaches about 200. This is why batching is the key to economics: reading the weights once serves many requests at nearly the same cost, so throughput rises almost linearly with batch size until compute or KV-cache memory becomes the limit.

def max_tokens_per_sec(bandwidth_tb_s, model_gb):
    return bandwidth_tb_s * 1e12 / (model_gb * 1e9)

H100 = 3.35            # TB/s HBM bandwidth
for name, gb in [("8B  FP16", 16), ("70B FP8", 70), ("70B INT4", 35)]:
    single = max_tokens_per_sec(H100, gb)
    print("%-9s single stream ceiling %6.0f tok/s | batch of 32: about %6.0f tok/s aggregate" % (name, single, single * 32))
# batching amortises the weight reads; in practice you land below these ceilings (KV cache reads, kernel overheads)

Follow-up: Why does doubling the batch size barely change per-user latency at first, and when does it start to?

80. What is continuous batching and PagedAttention, and why do engines like vLLM outperform naive serving?

Deep dive

sequenceDiagram participant Q as Request queue participant G as GPU batch Note over G: Static: batch of 4 waits for the longest output Q->>G: A, B, C, D start together G-->>Q: B, C finished early - slots sit idle G-->>Q: A and D finish - only now can new requests start Note over G: Continuous: refill after every decode step Q->>G: E takes B's freed slot immediately Q->>G: F takes C's freed slot immediately

Related techniques worth naming: chunked prefill (split a long prompt so it does not stall other users' decoding, improving tail latency) and prefill-decode disaggregation (run the compute-bound and memory-bound phases on separate GPU pools tuned for each).

Follow-up: A few very long prompts are causing latency spikes for everyone. Which two techniques address it?

81. What is speculative decoding and how much speed-up can it give?

Deep dive

Decoding is slow because each token needs a full pass of a big model, but verifying several candidate tokens costs almost the same as generating one (the weights are read once either way). Speculative decoding exploits this: a small, fast draft model proposes the next gamma tokens, then the large target model checks all of them in one parallel pass. Tokens are accepted up to the first disagreement, the target supplies the corrected token, and the process repeats. A rejection-sampling rule guarantees the output distribution is exactly the same as the target model's, so quality is unchanged: it is a pure speed-up.

flowchart LR D["Draft model proposes 4 tokens quickly"] --> V["Target model verifies all 4 in one pass"] V --> A{"Which prefix agrees?"} A -- "3 accepted" --> N["Emit 3 tokens plus 1 corrected token = 4 tokens for one target pass"] N --> D

If the per-token acceptance probability is alpha and the draft length is gamma, the expected number of tokens produced per target pass is (1 - alpha^(gamma+1)) / (1 - alpha). Variants avoid a separate draft model: Medusa adds extra prediction heads, EAGLE drafts at the feature level, and n-gram / prompt-lookup speculation copies likely continuations from the prompt (excellent for editing and summarisation tasks). Gains are best at low batch sizes and for predictable text; at high batch sizes, where the GPU is already busy, the benefit shrinks.

def expected_tokens(alpha, gamma):
    return (1 - alpha ** (gamma + 1)) / (1 - alpha)

for alpha in (0.5, 0.7, 0.8, 0.9):
    print("acceptance %.1f -> %.2f tokens per target pass (gamma=4)" % (alpha, expected_tokens(alpha, 4)))

Follow-up: Why does speculative decoding help less when the server is already running large batches?

82. Explain model quantisation. Compare INT8, INT4, FP8, PTQ and QAT. What happens when you push to 2 to 4 bits?

Core

Quantisation stores weights (and sometimes activations and the KV cache) with fewer bits, cutting memory, memory traffic and often latency. Because decoding is memory-bound, halving the bytes per weight nearly doubles single-stream speed.

What breaks at 2 to 4 bits: outlier weights and activations (a few large values dominate the error), long-context and reasoning-heavy tasks degrade first, smaller models are hurt more, and average benchmark scores can hide task-specific regressions. So: quantise, then re-run your own evaluation set and compare with the full-precision baseline; consider mixed precision (keep sensitive layers, embeddings and the output head in higher precision); and quantise the KV cache separately for long contexts.

Follow-up: You quantised to INT4 and accuracy dropped 6 points. What do you try, in order?

83. What are tensor, pipeline, data and expert parallelism? How do you serve a model larger than one GPU?

Deep dive

Rule of thumb for a 70B model: it needs about 140 GB in 16-bit, so use tensor parallelism across 2 to 4 (or 8) GPUs inside one NVLink-connected node, then add data-parallel replicas for throughput. Quantising to FP8 or INT4 can bring it onto fewer GPUs. Pipeline parallelism is for models that exceed a node. For training, sharding methods such as FSDP and DeepSpeed ZeRO partition parameters, gradients and optimiser state across GPUs and gather them when needed, trading communication for memory.

Follow-up: Tensor parallelism across two nodes is slower than within one node. Why?

84. How do prompt caching and semantic caching work, and what are the risks?

Core

There are two very different caches:

• Prompt (prefix) caching: the provider or engine stores the computed KV cache for the beginning of a prompt so later requests with the same prefix skip its prefill. It cuts cost and TTFT substantially for long, repeated system prompts, tool definitions, few-shot examples and documents. To benefit, keep the static part identical and first, put user-specific content last, and do not change earlier tokens (even whitespace or timestamps) between calls.
• Response caching: store complete answers keyed by the request. Exact caching (hash of the normalised prompt) is safe and precise. Semantic caching embeds the query and returns a stored answer when a new query is similar enough (cosine above a threshold), saving whole LLM calls for FAQ-style traffic.

Risks of semantic caching: false hits ("cancel my order" versus "cancel my subscription" can be very close in embedding space and have different answers), stale answers, and leaking one user's personalised or private answer to another. Mitigate with a high threshold tuned on real traffic, a check on entities and intent, scoping the cache by tenant and permissions, time-to-live, not caching personalised or high-stakes flows, and monitoring the false-hit rate.

import math, re
from collections import Counter

def vec(text):
    return Counter(re.findall(r"[a-z]+", text.lower()))

def cosine(a, b):
    num = sum(a[t] * b[t] for t in a)
    den = math.sqrt(sum(v * v for v in a.values())) * math.sqrt(sum(v * v for v in b.values()))
    return num / den if den else 0.0

cache = [("how do i reset my password", "Go to Settings, then Security, then Reset password.")]

def lookup(query, threshold=0.7):
    q = vec(query)
    best = max(cache, key=lambda item: cosine(q, vec(item[0])))
    score = cosine(q, vec(best[0]))
    return (best[1], round(score, 2)) if score >= threshold else (None, round(score, 2))

print(lookup("how can i reset my password"))     # hit
print(lookup("how do i delete my account"))      # miss: must call the model

Follow-up: How would you pick and monitor the similarity threshold in production?

85. How do you estimate and reduce the cost of running an LLM feature?

Core

Estimate from traffic: monthly cost = requests x (avg input tokens x input price + avg output tokens x output price), then add retries, evaluation traffic, embedding and reranking calls, vector database and hosting costs, and engineering time. Always model the p50 and p95 prompt sizes, because long-tail requests dominate the bill, and use cost per successful task rather than cost per call.

def monthly_cost(requests_per_day, in_tokens, out_tokens, price_in_per_m, price_out_per_m, cache_hit_rate=0.0):
    billable = requests_per_day * 30 * (1 - cache_hit_rate)
    return billable * (in_tokens * price_in_per_m + out_tokens * price_out_per_m) / 1e6

# example prices only - replace with your provider's current rates
big   = monthly_cost(200_000, 3000, 400, price_in_per_m=3.00, price_out_per_m=15.00)
small = monthly_cost(200_000, 3000, 400, price_in_per_m=0.25, price_out_per_m=1.25)
routed = 0.7 * small + 0.3 * big                                   # 70% of traffic handled by the small model
cached = monthly_cost(200_000, 3000, 400, 3.00, 15.00, cache_hit_rate=0.25)
print("all on large model : $%9.0f / month" % big)
print("all on small model : $%9.0f / month" % small)
print("70/30 routed       : $%9.0f / month" % routed)
print("large + 25%% cached : $%9.0f / month" % cached)

Levers, roughly in order of payoff: route easy requests to small models and hard ones to large; cache (prefix, exact, semantic); shorten prompts and outputs (fewer, reranked chunks; concise formats); batch non-urgent work through discounted batch endpoints; quantise or distil a self-hosted model for high volume; cap tokens, steps and retries; and track spend per feature and per tenant with alerts so a runaway loop is caught in minutes, not at month end. Verify each change against your quality benchmark.

Follow-up: Finance says the LLM bill doubled with flat traffic. Where do you look first?

86. How do you handle rate limits, provider outages and failover? Explain retries with exponential backoff and circuit breakers.

Core

Providers rate-limit by requests and by tokens per minute, and they have outages, so design for graceful degradation:

• Retry transient errors (429, 500, 503, timeouts) with exponential backoff plus jitter: wait a random time up to min(cap, base x 2^attempt). Jitter prevents thousands of clients retrying in lockstep (a thundering herd). Honour Retry-After headers; cap attempts; do not retry client errors (400s) or non-idempotent actions without an idempotency key.
• Client-side rate limiting: a token-bucket limiter that meters tokens (not just requests), with queues and priorities so interactive traffic beats batch jobs.
• Circuit breaker: after N consecutive failures stop calling the provider for a cool-down period, fail fast, and probe periodically to recover, so a dead dependency does not exhaust your threads and timeouts.
• Fallbacks: a secondary provider or region (behind a gateway that normalises the API), a smaller local model, a cached answer, or a clear degraded experience ("answers are limited right now"). Test the fallback path regularly, and re-check quality, prompts and safety on the fallback model.
• Backpressure and timeouts: bounded queues, per-request deadlines, and load shedding so you fail some requests cleanly rather than all of them slowly.

import random

def backoff_schedule(attempts, base=0.5, cap=20.0, seed=4):
    rng = random.Random(seed)
    waits = []
    for n in range(attempts):
        ceiling = min(cap, base * (2 ** n))
        waits.append(round(rng.uniform(0, ceiling), 2))      # "full jitter": random between 0 and the ceiling
    return waits

print("no jitter, ceilings :", [min(20.0, 0.5 * 2 ** n) for n in range(7)])
print("full-jitter waits   :", backoff_schedule(7))
# two clients with different seeds retry at different moments instead of together
print("another client      :", backoff_schedule(7, seed=9))

Follow-up: Your provider is down for 40 minutes. What does the user see, and what does your system do?

87. What should you monitor and log in a production LLM application? What is LLM observability?

Core

Classic service metrics are necessary but not sufficient, because an LLM system can be fast, error-free and wrong. Track four families:

Add alerts on budgets and anomalies (a spike in tokens per request signals a loop or prompt bloat), sample real traffic into the evaluation pipeline to catch drift, and mind privacy: logs contain user data, so apply retention limits, access control and redaction.

Follow-up: Users say answers got worse this week but every dashboard is green. What do you check?

88. How do you release model, prompt and pipeline changes safely? What does CI/CD look like for AI applications?

Core

AI changes are riskier than code changes because behaviour is statistical: a change can improve the average and break an important segment. So the release pipeline adds evaluation gates on top of the usual tests:

1. Version everything: prompts, model ids (pin exact versions, never a moving alias), retrieval configuration, embedding model, tool schemas and evaluation data.
2. Offline gate in CI: run the regression set (golden questions, edge cases, adversarial inputs, safety tests) and compare metrics against the current production version; block on regressions in quality, safety, latency or cost.
3. Shadow mode: send live traffic to the candidate without showing results, and compare with production.
4. Canary and A/B: release to a small share of users with guardrail metrics and automatic rollback; use feature flags so rollback is a switch, not a deploy.
5. Monitor after release on both system and quality metrics; keep the previous version warm for instant rollback.
6. Handle provider deprecations: when a provider retires a model, treat the migration as a release with a full evaluation.

Statistical care matters: compare variants on enough traffic and with confidence intervals (or paired evaluations with the same inputs), watch segment-level metrics, and beware novelty effects.

Follow-up: A/B results show +2% satisfaction but +30% cost. How do you decide?

Sources and further reading

• Kwon et al., Efficient Memory Management for LLM Serving with PagedAttention (vLLM)
• Leviathan et al., Fast Inference from Transformers via Speculative Decoding
• Yu et al., Orca: Continuous Batching
• Frantar et al., GPTQ
• Lin et al., AWQ: Activation-aware Weight Quantization
• AWS Architecture Blog: Exponential Backoff and Jitter
• AI Engineering interview questions (Outcome School, Apache-2.0)