System Design · Lesson 5 of 18
Queues, APIs and Reliability
Message queues, idempotency, retries, circuit breakers, WebSockets and API design.
- Advanced
- 50 min read
- 8 objectives
Before this lessonLesson 4: Choosing and Scaling Databases
What you will learn
- Decide what runs synchronously and what goes to a queue
- Choose between a queue and a log, and design partitions and consumer groups
- Build idempotent consumers for at-least-once delivery, with DLQs and backpressure
- Publish events safely with the transactional outbox
- Pick REST, gRPC, GraphQL, SSE, WebSocket or long polling per client
- Design cursors, versioning, idempotency keys, errors and rate-limit headers
- Apply timeouts, jittered retries, circuit breakers, bulkheads and load shedding
- Set SLIs, SLOs and error budgets with burn-rate alerts
Your Progress
0 of 18 lessons 0%
- Lessons0 / 18
- Completed0
- Est. time left~ 16 hours
Progress is saved in this browser. No account needed.
Tip: pressing Next marks this lesson complete automatically.
Every system in this course is a set of services talking to each other and to clients. This lesson is about how they talk and what happens when the conversation goes wrong: when to call synchronously and when to hand work to a queue, how queues differ from logs like Kafka, how ordering and delivery guarantees actually work, how to publish events without losing them, which API style fits which client, the small API design details that separate a professional API from a fragile one, and the reliability patterns — timeouts, retries with jitter, circuit breakers, bulkheads, load shedding — that keep one failure from becoming an outage. It closes with SLIs, SLOs and error budgets, the numbers that tell you how reliable is reliable enough.
Each concept has a worked example, and the Python runs in your browser. The database side of reliability (replication, transactions, sagas) is in Lesson 4; the case studies that lean hardest on this lesson are notifications, chat, video and the food delivery capstone.
Step 1 — Synchronous or asynchronous?
A synchronous call means the caller waits for the answer: the user's HTTP request stays open while your service calls payments, inventory and email. An asynchronous hand-off means the caller records the intent (a message on a queue) and returns; a worker does the job later. The question to ask of every step in a request is: does the user need the result of this step before we can answer them?
Checkout is the classic worked example. The user needs to know the payment was authorised and the stock reserved — synchronous. They do not need the confirmation email to have been sent, the analytics event recorded, the loyalty points credited or the warehouse notified before the "Thank you" page renders — all asynchronous. Moving those four out of the request path is not a micro-optimisation; it changes the availability and latency of checkout.
MIN_PER_MONTH = 30 * 24 * 60
# A request that synchronously calls N services, each 99.9% available,
# is only up when ALL of them are up at the same moment.
print("services in the sync path availability downtime/month")
for n in (1, 3, 5, 10):
a = 0.999 ** n
print(f"{n:>10} {a:8.3%} {(1 - a) * MIN_PER_MONTH:6.0f} min")
# Tail latency amplifies too: if a page fans out to N backends in parallel,
# the page is slow whenever ANY backend has a p99-slow response.
print("\nparallel calls chance the page hits at least one p99-slow call")
for n in (1, 10, 50, 100):
print(f"{n:>10} {1 - 0.99 ** n:6.1%}")
# The async version: the request path only needs the API and the queue.
checkout_sync = 0.999 ** 5 # api, payments, inventory, email, analytics
checkout_async = 0.999 ** 3 * 0.9999 # api, payments, inventory + a durable queue
print(f"\ncheckout sync over 5 services: {checkout_sync:.3%}")
print(f"checkout with email+analytics async: {checkout_async:.3%}")services in the sync path availability downtime/month
1 99.900% 43 min
3 99.700% 129 min
5 99.501% 216 min
10 99.004% 430 min
parallel calls chance the page hits at least one p99-slow call
1 1.0%
10 9.6%
50 39.5%
100 63.4%
checkout sync over 5 services: 99.501%
checkout with email+analytics async: 99.690%
Availability multiplies along a synchronous chain: five 99.9% services in series give you 99.5%, or three and a half hours of downtime a month. And fan-out amplifies tail latency: a page that calls 100 backends in parallel hits at least one backend's p99 on 63% of loads, so your p50 page is built from their p99s. Taking non-essential work off the path removes both its latency and its failures from the user's request.
What async buys, and what it costs
- Buys: spike absorption (10,000 sign-ups in a minute become a steady drain), failure isolation (the email provider is down, messages wait, nobody sees a 500), independent scaling of workers, and fan-out to many consumers from one event.
- Costs: eventual consistency (the email arrives in 2 seconds, or 20 minutes during a backlog), duplicates to handle (Step 4), a new component to operate and monitor, and harder debugging — a request is now a trail across processes, so you need trace ids in message headers.
- Wrong answer: "make everything async". If the user needs the answer (is my payment accepted?), a queue just makes them poll for it. Request/response is the right tool for queries and for commands whose result the caller needs.
Step 2 — Queues versus logs
"Message broker" covers two different designs. A queue (Amazon SQS, RabbitMQ, ActiveMQ, Google Cloud Tasks) hands each message to one consumer and deletes it once acknowledged: it is a to-do list shared by workers. A log (Apache Kafka, Amazon Kinesis, Apache Pulsar, Redpanda) appends messages to an ordered, retained sequence; consumers track their own position (offset) and reading does not remove anything: it is a newspaper archive that any number of readers can read at their own pace, and re-read.
offsets 0..N, kept 7 days)] LOG -->|group billing
at offset 1041| B[Billing] LOG -->|group analytics
at offset 870| A[Analytics] LOG -->|new group
replays from 0| N[Search indexer] end
Left: a queue deletes a message once one worker acknowledges it. Right: a log keeps messages; each consumer group has its own offset, so billing and analytics read the same events independently, and either can rewind.
| Queue (SQS, RabbitMQ) | Log (Kafka, Kinesis, Pulsar) | |
|---|---|---|
| Consumption | Competing consumers; each message to one worker | Each consumer group reads every message; within a group, each partition goes to one consumer |
| After processing | Acked message is deleted | Nothing is deleted; the group's offset advances |
| Retention and replay | Until consumed (SQS max 14 days); no replay | By time or size (days to forever); rewind to reprocess or bootstrap a new consumer |
| Ordering | Best effort (SQS standard); FIFO queues order per message group | Strict within a partition |
| Per-message control | Individual ack, delay, visibility timeout, per-message retry and DLQ | Offset per partition: one stuck message blocks everything behind it in that partition |
| Routing | RabbitMQ exchanges: topic, fanout, header routing; priorities | By topic and key only |
| Throughput | Thousands to tens of thousands/s per queue (SQS scales further horizontally) | Millions/s per cluster; sequential disk writes and batching |
| Scaling consumers | Add workers any time | Up to the partition count per group |
| Best for | Task/job queues: send email, resize image, charge card, call a webhook | Event streams: order events, clickstreams, CDC, metrics, event sourcing, feeding many systems |
Worked choices. Thumbnail generation for uploaded photos: a queue — each job is independent, needs per-job retry, and nobody will replay it. "Order placed" consumed by billing, email, fraud, analytics and a search indexer: a log — five consumers, each at its own pace, and next quarter's new recommendation service can replay the last 30 days to build its state. Publish/subscribe fan-out to a handful of queues (Amazon SNS to several SQS queues, or a RabbitMQ fanout exchange) is the middle ground when you want fan-out but per-message retries.
Step 3 — Partitions, consumer groups and ordering
A Kafka topic is split into partitions, each an independent ordered log on a different broker. A producer chooses a partition by hashing the message key; messages with the same key always land in the same partition, in the order they were produced. A consumer group divides a topic's partitions among its members: each partition is read by exactly one consumer of the group at a time. Those two rules together give the only ordering guarantee you get — per key — and they cap the group's parallelism at the partition count.
import hashlib
from collections import defaultdict
PARTITIONS = 6
def partition_for(key):
# Kafka's default partitioner is murmur2(key) % partitions; md5 stands in here
return int(hashlib.md5(key.encode()).hexdigest(), 16) % PARTITIONS
# Events for three orders, produced interleaved. Keyed by order_id.
events = [("o1", "CREATED"), ("o2", "CREATED"), ("o1", "PAID"), ("o3", "CREATED"),
("o2", "PAID"), ("o1", "SHIPPED"), ("o3", "PAID"), ("o2", "CANCELLED")]
log = defaultdict(list) # partition -> ordered list of records
for key, ev in events:
log[partition_for(key)].append((key, ev))
for p in sorted(log):
print(f"partition {p}: {log[p]}")
def assign(consumers):
"""Round-robin assignment: each partition goes to exactly ONE consumer in the group."""
out = defaultdict(list)
for p in range(PARTITIONS):
out[consumers[p % len(consumers)]].append(p)
return dict(out)
print("\ngroup 'billing' with 2 consumers:", assign(["c1", "c2"]))
print("group 'billing' with 3 consumers:", assign(["c1", "c2", "c3"]))
a = assign([f"c{i}" for i in range(1, 9)])
idle = 8 - len(a)
print(f"group 'billing' with 8 consumers: {len(a)} busy, {idle} idle "
f"(parallelism is capped by the partition count)")
# Every order's events stay in order, because one key -> one partition -> one consumer.
for oid in ("o1", "o2", "o3"):
seq = [ev for key, ev in log[partition_for(oid)] if key == oid]
print(f"{oid} as seen by its consumer: {seq}")partition 0: [('o3', 'CREATED'), ('o3', 'PAID')]
partition 4: [('o1', 'CREATED'), ('o2', 'CREATED'), ('o1', 'PAID'), ('o2', 'PAID'), ('o1', 'SHIPPED'), ('o2', 'CANCELLED')]
group 'billing' with 2 consumers: {'c1': [0, 2, 4], 'c2': [1, 3, 5]}
group 'billing' with 3 consumers: {'c1': [0, 3], 'c2': [1, 4], 'c3': [2, 5]}
group 'billing' with 8 consumers: 6 busy, 2 idle (parallelism is capped by the partition count)
o1 as seen by its consumer: ['CREATED', 'PAID', 'SHIPPED']
o2 as seen by its consumer: ['CREATED', 'PAID', 'CANCELLED']
o3 as seen by its consumer: ['CREATED', 'PAID']
- Choose the key by what must stay ordered. Order events keyed by
order_idkeep CREATED → PAID → SHIPPED in sequence. Keying bycustomer_idalso works and additionally orders a customer's orders relative to each other. No key (round-robin) maximises spread and gives no ordering. - Global ordering requires a single partition, which caps you at one consumer. Almost no business process needs it; say so if asked.
- Choose the partition count with headroom (for example, 3–4x today's consumer count). Adding partitions later changes
hash(key) % partitions, so existing keys move and per-key ordering breaks across the change. - Hot keys make hot partitions. One merchant producing 40% of events will pin one partition and one consumer at 100%. Split that key (
merchant_id + sub-bucket) when its events do not need mutual ordering. - Rebalances happen whenever a consumer joins, leaves or stalls past its poll timeout. Partitions move, and messages processed but not yet committed are redelivered to the new owner — one more reason consumers must be idempotent.
- Queues have an equivalent: SQS FIFO orders within a
MessageGroupIdand only one message per group is in flight at a time; RabbitMQ has single-active-consumer and consistent-hash exchanges.
Step 4 — Delivery semantics and idempotent consumers
A consumer does two things per message: the work and the acknowledgement (ack, or offset commit). A crash can land between them. Where you place the ack decides the guarantee:
- At-most-once — ack first, then work. A crash after the ack loses the message. Acceptable for metrics samples or a presence ping; never for money.
- At-least-once — work first, then ack. A crash before the ack means the message is redelivered and the work happens twice. This is the default of SQS, RabbitMQ and Kafka consumers, and what you should assume.
- Exactly-once — not something the network can give you end to end. What you can build is effectively-once: at-least-once delivery plus an idempotent consumer, so duplicates have no additional effect.
def consume(mode):
"""Four messages, each meaning 'add $10 to the balance'. The consumer
crashes once, around message m2; the broker redelivers anything unacked."""
queue = ["m1", "m2", "m3", "m4"]
acked, seen = set(), set()
state = {"balance": 0, "crash_pending": True}
def handle(msg):
if mode == "effectively-once":
if msg in seen: # duplicate delivery: skip the effect
return
seen.add(msg) # real life: INSERT INTO processed(msg_id)
state["balance"] += 10 # in the SAME DB transaction as this update
def crash_here(msg):
if state["crash_pending"] and msg == "m2":
state["crash_pending"] = False
return True
return False
while len(acked) < len(queue):
for msg in [m for m in queue if m not in acked]: # (re)poll unacked messages
if mode == "at-most-once":
acked.add(msg) # ack first...
if crash_here(msg):
break # ...crash before the work: m2 is gone forever
handle(msg)
else:
handle(msg) # work first...
if crash_here(msg):
break # ...crash before the ack: m2 comes back
acked.add(msg)
return state["balance"]
for mode in ("at-most-once", "at-least-once", "effectively-once"):
print(f"{mode:17} balance = ${consume(mode)} (expected $40)")at-most-once balance = $30 (expected $40) at-least-once balance = $50 (expected $40) effectively-once balance = $40 (expected $40)
How to make a consumer idempotent
- Naturally idempotent operations.
SET status = 'SHIPPED'can run twice;balance = balance + 10cannot. Prefer absolute writes ("set to X") over relative ones ("add X") where the domain allows. - A dedupe table in the same transaction.
INSERT INTO processed_messages(message_id)with a unique constraint, in the same database transaction as the effect; a duplicate hits the constraint and the whole transaction is skipped. The key must be a stable id assigned by the producer, not something regenerated on retry. - Conditional updates with versions.
UPDATE orders SET state='PAID', seq=17 WHERE id=? AND seq < 17— duplicates and stale, out-of-order events both become no-ops. - Idempotency keys passed downstream. When the effect is an external call (a payment provider, an email API), pass a key derived from the message id so the provider dedupes too.
- Kafka's exactly-once (idempotent producer plus transactions) covers read-from-Kafka, write-to-Kafka pipelines such as Kafka Streams. It does not cover your database or an email API; those still need the techniques above.
Step 5 — Retries, poison messages and dead-letter queues
Failures come in two kinds. Transient failures (a timeout, a throttled downstream, a deadlock) succeed if you retry later. Permanent failures — a poison message with malformed JSON, a reference to a deleted user, a bug your code hits every time — will fail forever, and if you retry them forever they burn capacity and, in a log, block the partition behind them. A dead-letter queue (DLQ) separates the two: after N attempts the message is moved aside with its error, so the main flow keeps moving and a human or a tool can deal with it.
MAX_RECEIVES = 3 # SQS maxReceiveCount / RabbitMQ delivery-limit
attempts = {}
def handler(msg):
n = attempts[msg["id"]] = attempts.get(msg["id"], 0) + 1
if msg["id"] == "img-2" and n <= 2:
raise TimeoutError("thumbnail service timed out") # transient
if msg["body"] is None:
raise ValueError("body is not valid JSON") # poison: will never work
return "ok"
queue = [{"id": "img-1", "body": "cat.jpg"}, {"id": "img-2", "body": "dog.jpg"},
{"id": "img-3", "body": None}, {"id": "img-4", "body": "fox.jpg"}]
receives, dlq, tick = {}, [], 0
while queue:
tick += 1
msg = queue.pop(0)
receives[msg["id"]] = receives.get(msg["id"], 0) + 1
try:
handler(msg)
print(f"t{tick:<2} {msg['id']} delivered (receive #{receives[msg['id']]})")
except Exception as e:
if receives[msg["id"]] >= MAX_RECEIVES:
dlq.append({**msg, "error": repr(e), "receives": receives[msg["id"]]})
print(f"t{tick:<2} {msg['id']} -> DLQ after {receives[msg['id']]} receives: {e}")
else:
queue.append(msg) # becomes visible again after the visibility timeout
print(f"t{tick:<2} {msg['id']} failed ({e}); will be redelivered")
print("\nDLQ:", dlq)t1 img-1 delivered (receive #1)
t2 img-2 failed (thumbnail service timed out); will be redelivered
t3 img-3 failed (body is not valid JSON); will be redelivered
t4 img-4 delivered (receive #1)
t5 img-2 failed (thumbnail service timed out); will be redelivered
t6 img-3 failed (body is not valid JSON); will be redelivered
t7 img-2 delivered (receive #3)
t8 img-3 -> DLQ after 3 receives: body is not valid JSON
DLQ: [{'id': 'img-3', 'body': None, 'error': "ValueError('body is not valid JSON')", 'receives': 3}]
- Visibility timeout (SQS) or unacked redelivery (RabbitMQ): a received message is hidden, not deleted; if the worker does not ack within the timeout it reappears. Set it comfortably above the p99 processing time, or long jobs get processed twice concurrently; extend it (heartbeat) for jobs of unpredictable length.
- Back off between attempts (Step 10) — retrying a throttled API immediately just gets throttled again. SQS delay queues, RabbitMQ delayed/TTL queues, and in Kafka a chain of retry topics (
orders.retry.1m,orders.retry.10m) implement delayed retries. - Fail fast on permanent errors. A validation error should go straight to the DLQ rather than waste three attempts; classify exceptions.
- The DLQ is not a bin. Alert on its depth, record the error, stack trace, attempt count and original metadata, and build a redrive tool to replay messages after the fix. A DLQ nobody watches is data loss with extra steps.
- Ordering caveat: moving one message aside lets later messages for the same key overtake it. When per-key order matters, park the whole key (divert its later messages too) until the first one is resolved.
Step 6 — Backpressure: when producers outrun consumers
A queue absorbs a spike only if the spike ends. When arrivals exceed processing capacity, the backlog grows linearly, and by Little's law the waiting time grows with it: delay = backlog ÷ throughput. Backpressure is any mechanism that pushes the overload back toward the source instead of letting it silently accumulate.
# A flash sale: producers publish 3,000 msg/s for 10 minutes, then 500 msg/s.
# Each consumer processes 100 msg/s; we run 20 of them (2,000 msg/s capacity).
CONSUMERS, PER_CONSUMER = 20, 100
capacity = CONSUMERS * PER_CONSUMER
def arrivals(t):
return 3_000 if t < 600 else 500
def simulate(queue_limit=None, minutes=40):
backlog, peak, rejected, drained_at = 0, 0, 0, None
for t in range(minutes * 60):
backlog += arrivals(t)
if queue_limit is not None and backlog > queue_limit:
rejected += backlog - queue_limit # producer gets "queue full" -> 429 / 503
backlog = queue_limit
backlog = max(0, backlog - capacity)
peak = max(peak, backlog)
if t >= 600 and backlog == 0 and drained_at is None:
drained_at = t
return peak, rejected, drained_at
peak, rejected, drained = simulate()
print(f"unbounded queue: peak backlog {peak:,} msgs, "
f"drained {(drained - 600) / 60:.0f} min after the spike ends")
# Little's law: time waiting in queue = backlog / throughput
print(f" worst queueing delay = {peak / capacity / 60:.0f} min "
"(fine for emails, fatal for 'your ride is here')")
peak, rejected, drained = simulate(queue_limit=120_000)
print(f"bounded at 120k: peak {peak:,}, rejected {rejected:,} msgs, "
f"worst delay {peak / capacity:.0f} s")
# Scaling consumers only helps once capacity exceeds the peak arrival rate.
for n in (20, 25, 30):
cap = n * PER_CONSUMER
growth = max(0, 3_000 - cap) * 600
print(f"{n} consumers: backlog grows {max(0, 3_000 - cap):,}/s -> "
f"{growth:,} msgs, delay {growth / cap:,.0f} s")unbounded queue: peak backlog 600,000 msgs, drained 7 min after the spike ends worst queueing delay = 5 min (fine for emails, fatal for 'your ride is here') bounded at 120k: peak 118,000, rejected 482,000 msgs, worst delay 59 s 20 consumers: backlog grows 1,000/s -> 600,000 msgs, delay 300 s 25 consumers: backlog grows 500/s -> 300,000 msgs, delay 120 s 30 consumers: backlog grows 0/s -> 0 msgs, delay 0 s
The numbers make the trade explicit. An unbounded queue never rejects anything but delivers messages five minutes late — fine for newsletters, useless for "your driver has arrived". A bounded queue caps the delay at about a minute but rejects 482,000 messages, which producers must handle (retry later, degrade, or tell the user). And adding consumers only helps once total capacity exceeds the peak arrival rate — at 25 consumers the backlog still grows, just more slowly.
| Strategy | How | Use when |
|---|---|---|
| Buffer | Durable queue with generous retention | Spikes are short and delay is acceptable |
| Scale consumers | Autoscale on queue depth or lag in seconds, not CPU | The bottleneck is the consumer, not a downstream database it hammers |
| Bound and reject | Max queue length; producer gets 429/503 or a blocked send | Stale work is worthless; protect latency for what you accept |
| Rate-limit producers | Token bucket per tenant at the API edge (rate limiter lesson) | One noisy tenant should not flood everyone |
| Drop or sample | Keep 1 in 10 metrics points; drop superseded updates (only the newest location matters) | Messages are samples, or newer ones replace older |
| Pull, not push | Consumers fetch at their own pace (Kafka, SQS long polling); gRPC and reactive streams have flow-control windows | Always prefer it; push-based consumers need explicit credit/prefetch limits (RabbitMQ prefetch) |
Step 7 — The transactional outbox
A service that must both update its database and publish an event faces the dual-write problem. Commit then publish, and a crash in between leaves an order nobody hears about. Publish then commit, and a failed commit leaves an event for an order that does not exist. There is no ordering of two separate systems that is safe, and 2PC across a database and a broker is rarely available or wise.
The fix is to make it one write. In the same local transaction as the business change, insert the event into an outbox table. A separate relay process reads unsent outbox rows, publishes them, and marks them sent. The event is published if and only if the transaction committed — at least once, so consumers dedupe on the event id.
One local transaction writes both the order and its event. The relay publishes afterwards; if it crashes before marking a row sent, the row is published again, and the consumer's dedupe on event_id absorbs the duplicate.
import itertools
class World:
def __init__(self):
self.db_orders, self.outbox, self.broker, self.ids = [], [], [], itertools.count(1)
def dual_write(w, crash_between):
"""Anti-pattern: commit to the DB, then publish to the broker."""
oid = next(w.ids)
w.db_orders.append(oid) # COMMIT
if crash_between:
return # process killed / broker unreachable
w.broker.append(("order.placed", oid))
def with_outbox(w):
"""Order row and event row in ONE local transaction."""
oid = next(w.ids)
w.db_orders.append(oid) # BEGIN; INSERT order;
w.outbox.append({"event_id": f"evt-{oid}", "order": oid, "sent": False}) # INSERT outbox; COMMIT
def relay(w, crash_before_mark=False):
"""Separate process: publish unsent rows, then mark them sent."""
for row in w.outbox:
if not row["sent"]:
w.broker.append(("order.placed", row["order"], row["event_id"]))
if crash_before_mark:
return # published, but not marked -> resent later
row["sent"] = True
w = World()
dual_write(w, crash_between=False)
dual_write(w, crash_between=True)
print("dual write: orders", w.db_orders, "events", [e[1] for e in w.broker],
"-> order 2 exists but nobody was told")
w = World()
with_outbox(w); with_outbox(w)
relay(w, crash_before_mark=True) # relay dies mid-way
relay(w) # restarts, resends the unmarked row
print("outbox: orders", w.db_orders, "events", [e[2] for e in w.broker])
seen, applied = set(), []
for _, order, event_id in w.broker: # the consumer dedupes by event_id
if event_id not in seen:
seen.add(event_id); applied.append(order)
print("consumer applied:", applied, "-> nothing lost, duplicate absorbed")dual write: orders [1, 2] events [1] -> order 2 exists but nobody was told outbox: orders [1, 2] events ['evt-1', 'evt-1', 'evt-2'] consumer applied: [1, 2] -> nothing lost, duplicate absorbed
- Relay by polling — a loop running
SELECT ... WHERE published_at IS NULL ORDER BY id LIMIT 500 FOR UPDATE SKIP LOCKED. Simple; adds some load and around 100 ms–1 s of delay. - Relay by change data capture — Debezium tails the database's write-ahead log and publishes outbox inserts. Lower latency, no polling load, preserves commit order; one more system to run.
- Housekeeping — delete or partition published rows after a few days, or the table grows forever.
- The mirror image, the inbox: the consumer records incoming event ids in its own database in the same transaction as the effect. Outbox on the producer plus inbox on the consumer is the standard recipe for effectively-once across services — and the reliable way to publish saga steps from Lesson 4.
Step 8 — API styles: REST, gRPC, GraphQL and the realtime family
The same operation -- "get order 88231 with its items" -- in each style.
REST (HTTP + JSON, resources and verbs)
GET /v1/orders/88231
200 OK ETag: "v7" Cache-Control: private, max-age=5
{ "id": "88231", "state": "PAID", "items": [{ "sku": "B-12", "qty": 2 }] }
+ cacheable by URL, debuggable with curl, universal client support
- over/under-fetching: the mobile app wanted 3 fields, got 30; or needs 3 calls
gRPC (HTTP/2 + Protocol Buffers, generated stubs)
service Orders { rpc GetOrder(GetOrderRequest) returns (Order); }
message Order { string id = 1; State state = 2; repeated Item items = 3; }
+ binary (3-10x smaller than JSON), typed contracts, streaming both ways,
deadlines propagate across hops -- ideal between internal services
- browsers need gRPC-Web + a proxy; not human-readable; no URL caching
GraphQL (one endpoint, the client names the fields)
POST /graphql
{ order(id: "88231") { state items { sku qty } restaurant { name } } }
+ one round trip for a nested screen; clients evolve without new endpoints
- every query is a POST to one URL (CDN caching needs persisted queries);
a naive resolver does N+1 database calls (use DataLoader batching);
a malicious query can be arbitrarily expensive (depth/cost limits)
WebSocket (full-duplex frames over one upgraded TCP connection)
GET /v1/orders/88231/live Upgrade: websocket
<- {"type":"state","state":"PICKED_UP"} -> {"type":"ping"}
+ both sides send any time, tiny per-message overhead: chat, games, editors
- stateful: sticky connections, custom reconnect/resume, harder to load balance
Server-Sent Events (one-way server push over plain HTTP)
GET /v1/orders/88231/events Accept: text/event-stream
id: 41
event: state
data: {"state":"PICKED_UP"}
+ plain HTTP (works through most proxies), auto-reconnect with Last-Event-ID
- server -> client only; text only; ~6 connections per domain on HTTP/1.1
Long polling (a GET the server holds open until there is news)
GET /v1/orders/88231/events?after=41&wait=30
... server holds up to 30 s ... 200 [{"id":42,"state":"DELIVERED"}] | 204
+ works everywhere, even through the most hostile corporate proxy
- a new request per event; the server holds one request per client| Style | Transport | Direction | Best for | Watch out for |
|---|---|---|---|---|
| REST | HTTP/1.1 or 2, JSON | Request/response | Public APIs, CRUD resources, anything cacheable | Chatty clients, over-fetching; versioning discipline |
| gRPC | HTTP/2, Protobuf | Unary + client/server/bidi streaming | Service-to-service, low latency, polyglot internal contracts, mobile with generated clients | Browser support needs a proxy; L7 load balancing needed (long-lived HTTP/2 connections) |
| GraphQL | HTTP POST, JSON | Request/response (+ subscriptions) | Aggregating many backends for varied front-ends (a backend-for-frontend) | N+1 resolvers, query cost limits, caching |
| Short polling | HTTP | Client asks repeatedly | Rare updates, trivial implementation, CDN-cacheable status | Mostly empty responses; delay up to the interval |
| Long polling | HTTP (held request) | Server replies when there is news | Hostile networks, fallback transport, simple notification feeds | One held request per client; re-request per event |
| Server-Sent Events | HTTP streaming, text | Server → client | Live scores, dashboards, notifications, LLM token streaming | One-way; HTTP/1.1 connection limits per domain |
| WebSocket | Upgraded TCP connection | Full duplex | Chat, multiplayer games, collaborative editing, trading | Stateful connections: sticky routing, reconnect and resume logic, pub/sub backplane |
| Webhooks | HTTP POST to the client's server | Server → server | Notifying third parties (payment succeeded) | Sign payloads, retry with backoff, receivers must dedupe |
Polling, long polling, SSE and WebSocket compared on one timeline
Short polling asks on a timer and usually hears nothing. Long polling parks a request until news arrives. SSE and WebSocket keep one connection open and push; only WebSocket lets the client send on the same connection.
CLIENTS = 1_000_000
EVENTS_PER_CLIENT_PER_MIN = 0.5 # e.g. a notification badge updates every 2 min
events_per_s = CLIENTS * EVENTS_PER_CLIENT_PER_MIN / 60
print(f"useful events: {events_per_s:,.0f}/s\n")
# Short polling every 5 s: every client asks, whether or not anything happened
poll_rps = CLIENTS / 5
print(f"short poll /5s : {poll_rps:>9,.0f} req/s, {events_per_s / poll_rps:5.1%} carry data, "
f"worst-case delay 5 s")
# Long polling: one request is parked per client until an event or a 30 s timeout
lp_rps = events_per_s + CLIENTS / 30 # re-request after each event or each timeout
print(f"long poll 30s : {lp_rps:>9,.0f} req/s, {CLIENTS:,} requests held open, delay ~0")
# SSE / WebSocket: one persistent connection, the server pushes each event
KB_PER_CONN = 30 # kernel buffers + TLS + app state, roughly
CONNS_PER_NODE = 100_000
print(f"SSE/WebSocket : {events_per_s:>9,.0f} pushes/s, {CLIENTS:,} connections "
f"= ~{CLIENTS * KB_PER_CONN / 1e6:.0f} GB RAM across "
f"{CLIENTS // CONNS_PER_NODE} nodes (+ headroom)")useful events: 8,333/s short poll /5s : 200,000 req/s, 4.2% carry data, worst-case delay 5 s long poll 30s : 41,667 req/s, 1,000,000 requests held open, delay ~0 SSE/WebSocket : 8,333 pushes/s, 1,000,000 connections = ~30 GB RAM across 10 nodes (+ headroom)
For a million clients receiving one event every two minutes, short polling generates 200,000 requests per second, 96% of them empty. Long polling cuts that fivefold. A persistent connection sends only the 8,333 useful messages per second — but holds a million connections, around 30 GB of memory across a dedicated tier. That is why realtime systems run a separate, stateful connection gateway (deployed independently of the stateless API so a deploy does not drop every socket), a pub/sub backplane such as Redis pub/sub or Kafka so any API server can reach a user connected to any gateway node, and a registry of which node holds which user. The chat lesson builds this.
Step 9 — API design details that matter in interviews and in production
## Pagination: opaque cursor, not page numbers
GET /v1/orders?limit=50&cursor=eyJ0cyI6MTcwMDAwMDAwMCwiaWQiOjg4MjMxfQ
200 { "data": [...50 orders...],
"next_cursor": "eyJ0cyI6MTY5OTk5OTEyMCwiaWQiOjg3OTAyfQ", # null on the last page
"has_more": true }
# The cursor encodes the sort key of the last row (created_at, id). It is
# opaque (base64, optionally signed) so clients cannot build or depend on it.
## Versioning
GET /v1/orders/88231 # major version in the path, for breaking changes
Stripe-Version: 2024-06-20 # or date-pinned versions per API key
# Additive changes (new optional field, new endpoint, new enum value that
# clients were told to tolerate) ship WITHOUT a new version.
# Breaking: removing/renaming a field, changing a type or a default, new
# required input. Run v1 and v2 side by side; announce a sunset date;
# send "Deprecation" and "Sunset" response headers; measure who still calls v1.
## Idempotency keys on every non-idempotent POST
POST /v1/payments
Idempotency-Key: 5f0c1d3e-7a4b-4c1e-9f66-0b1a6c2d9e11 # client-generated UUID
{ "order_id": "o-77", "amount_minor": 4999, "currency": "EUR" }
-> 201 first time; the SAME 201 body on any retry with the same key (for 24 h)
-> 409 if the first request is still in progress
-> 422 if the key is reused with a different body
## Errors: one machine-readable format everywhere (RFC 9457 problem+json)
HTTP/1.1 409 Conflict
Content-Type: application/problem+json
{ "type": "https://api.example.com/errors/order-not-cancellable",
"title": "Order can no longer be cancelled",
"status": 409,
"detail": "Order 88231 was picked up at 12:41; cancellation closes at pickup.",
"instance": "/v1/orders/88231/cancel",
"code": "ORDER_NOT_CANCELLABLE", # stable string clients switch on
"request_id": "req_7Hc2kQ" } # the support team greps logs for this
## Status codes that carry meaning
400 malformed | 401 no/invalid credentials | 403 authenticated but not allowed
404 not found | 409 state conflict | 412 If-Match precondition failed (stale ETag)
422 valid JSON, invalid values | 429 rate limited | 500 our bug
502/504 an upstream failed or timed out | 503 overloaded or in maintenance
# Clients should retry: 429, 502, 503, 504 (with backoff, honouring Retry-After)
# Clients must NOT blindly retry: 400, 401, 403, 404, 409, 422
## Rate limit headers (IETF RateLimit fields; many APIs still use X-RateLimit-*)
HTTP/1.1 200 OK
RateLimit-Policy: 100;w=60 # 100 requests per 60 s window
RateLimit: limit=100, remaining=37, reset=21
HTTP/1.1 429 Too Many Requests
Retry-After: 21
Content-Type: application/problem+json
{ "type": ".../errors/rate-limited", "status": 429, "detail": "100 requests per minute" }Worked example: why cursors, not offsets
import base64, json
# A feed ordered newest first. Each row: (created_at, id)
feed = [(100 - i, f"p{100 - i}") for i in range(12)] # p100 ... p89
def offset_page(rows, offset, limit=4):
return [r[1] for r in rows[offset:offset + limit]]
def encode(row):
return base64.urlsafe_b64encode(json.dumps(row).encode()).decode()
def keyset_page(rows, cursor=None, limit=4):
# WHERE (created_at, id) < (:c_ts, :c_id) ORDER BY created_at DESC, id DESC LIMIT :limit
if cursor:
c = tuple(json.loads(base64.urlsafe_b64decode(cursor)))
rows = [r for r in rows if r < c]
page = rows[:limit]
return [r[1] for r in page], encode(page[-1])
print("offset page 1:", offset_page(feed, 0))
p1, cur = keyset_page(feed)
print("keyset page 1:", p1, " next_cursor =", cur)
# Two new posts arrive at the top before the user scrolls.
feed = [(102, "p102"), (101, "p101")] + feed
print("offset page 2:", offset_page(feed, 4), " <- p98, p97 shown twice")
p2, _ = keyset_page(feed, cur)
print("keyset page 2:", p2, " <- continues exactly where page 1 ended")
# Why offsets are also slow: OFFSET 1,000,000 still reads and discards 1M rows.
# The keyset query is an index seek to the cursor, whatever page you are on.offset page 1: ['p100', 'p99', 'p98', 'p97'] keyset page 1: ['p100', 'p99', 'p98', 'p97'] next_cursor = Wzk3LCAicDk3Il0= offset page 2: ['p98', 'p97', 'p96', 'p95'] <- p98, p97 shown twice keyset page 2: ['p96', 'p95', 'p94', 'p93'] <- continues exactly where page 1 ended
Offset pagination breaks on any list that changes while someone reads it: two new posts push two old ones onto page 2, and the user sees them twice (a deletion makes them skip rows instead). It is also slow at depth, because OFFSET 1000000 reads and discards a million rows. A keyset cursor — "rows strictly after (created_at, id) of the last row I saw" — is an index seek at any depth and is stable under inserts. Include a unique tiebreaker (id) in the sort, or rows with equal timestamps get skipped. Offsets are still fine for small, static admin tables where "jump to page 7" is a real need.
Worked example: an idempotency-key handler
import hashlib, json
store = {} # idempotency_key -> {"hash", "state", "response"}; a DB table in real life
charges = [] # side effect we must not repeat
def handle_payment(key, user_id, body):
req_hash = hashlib.sha256(json.dumps(body, sort_keys=True).encode()).hexdigest()
rec = store.get((user_id, key)) # keys are scoped per user/API client
if rec is None:
# INSERT ... ON CONFLICT DO NOTHING: exactly one concurrent request wins the claim
store[(user_id, key)] = {"hash": req_hash, "state": "IN_PROGRESS", "response": None}
charges.append(body["amount_minor"])
resp = (201, {"payment_id": f"pay_{len(charges)}", "status": "succeeded"})
store[(user_id, key)].update(state="DONE", response=resp)
return resp
if rec["hash"] != req_hash:
return (422, {"type": "/errors/idempotency-key-reuse",
"detail": "key was used with a different request body"})
if rec["state"] == "IN_PROGRESS":
return (409, {"type": "/errors/request-in-progress", "retry_after_s": 1})
return rec["response"] # replay the ORIGINAL response, byte for byte
def show(label, resp):
print(f"{label + ':':22}{resp}")
body = {"amount_minor": 4_999, "currency": "EUR", "order_id": "o-77"}
show("1st attempt", handle_payment("k-1", "u-9", body))
show("retry (timeout)", handle_payment("k-1", "u-9", body))
show("same key, new body", handle_payment("k-1", "u-9", {**body, "amount_minor": 9_999}))
# Simulate a second request arriving while the first is still talking to the PSP
body2 = {"amount_minor": 1_500, "currency": "EUR", "order_id": "o-78"}
h2 = hashlib.sha256(json.dumps(body2, sort_keys=True).encode()).hexdigest()
store[("u-9", "k-2")] = {"hash": h2, "state": "IN_PROGRESS", "response": None}
show("concurrent duplicate", handle_payment("k-2", "u-9", body2))
print("charges made:", charges)1st attempt: (201, {'payment_id': 'pay_1', 'status': 'succeeded'})
retry (timeout): (201, {'payment_id': 'pay_1', 'status': 'succeeded'})
same key, new body: (422, {'type': '/errors/idempotency-key-reuse', 'detail': 'key was used with a different request body'})
concurrent duplicate: (409, {'type': '/errors/request-in-progress', 'retry_after_s': 1})
charges made: [4999]
The details that make this correct in production: the key is claimed with an atomic insert on a unique index before any side effect (so two concurrent retries cannot both proceed); the stored request hash turns accidental key reuse into a clear 422 rather than a silently wrong replay; the stored response is replayed exactly, including the status code; keys are scoped per client and expire after 24–48 hours; and a request that crashed mid-way (stuck IN_PROGRESS) is recovered by checking the downstream system with the same key rather than blindly re-running. The food delivery capstone uses exactly this table for order placement.
Versioning, briefly
Most changes should be additive and need no new version: new optional request fields, new response fields, new endpoints. That only works if clients are told to ignore unknown fields and tolerate new enum values — write that into the API contract from day one. Breaking changes get a new major version (/v2/) or a new dated version (the Stripe model, where each API key is pinned to the version it was created with and upgrades explicitly). Keep old versions running with a published sunset date, send Deprecation and Sunset headers, and measure traffic per version before turning anything off. For gRPC, never reuse a Protobuf field number; mark removed fields reserved.
Step 10 — Reliability patterns
Every remote call can fail, be slow, or — worst of all — succeed without you finding out. The patterns below exist to keep one sick dependency from taking down everything that calls it. They are layered: timeouts bound the wait, retries recover transient failures, circuit breakers stop retrying the hopeless, bulkheads contain the damage, load shedding protects the core, and graceful degradation decides what the user sees.
Timeouts and deadlines
Every network call needs a timeout; the default in many HTTP clients is infinite, which is how one hung dependency eventually consumes every thread in the caller. Set timeouts from the dependency's measured latency (a little above its p99.9), not from a round number. Worked example: an API with a 1,000 ms budget calls auth (p99 20 ms), then the order service (p99 150 ms), which calls the database (p99 40 ms). Timeouts of 50 ms for auth and 300 ms for the order service leave room for one retry within the budget. Pass the remaining deadline down the chain (gRPC does this natively), so a downstream service does not start work the caller has already given up on.
Retries with exponential backoff and jitter
Retry only what is safe (idempotent requests, or requests with an idempotency key) and only on errors that might succeed next time (timeouts, 429, 502, 503, 504 — never 400 or 422). Wait longer after each attempt, and randomise the wait. The randomisation is the part candidates forget, and it matters most: when a dependency hiccups, all clients fail at the same instant, and deterministic backoff makes them all retry at the same instants too.
import random
from collections import Counter
random.seed(42)
CLIENTS, BASE, CAP, TRIES = 1_000, 1.0, 20.0, 5 # seconds
def schedule(strategy):
"""Times (s) at which each client retries after all failing together at t=0."""
times = []
for _ in range(CLIENTS):
t = 0.0
for attempt in range(TRIES):
ceiling = min(CAP, BASE * 2 ** attempt) # 1, 2, 4, 8, 16 (capped at 20)
if strategy == "fixed 1s":
wait = 1.0
elif strategy == "exponential":
wait = ceiling
elif strategy == "exp + full jitter":
wait = random.uniform(0, ceiling)
else: # equal jitter
wait = ceiling / 2 + random.uniform(0, ceiling / 2)
t += wait
times.append(t)
return times
print(f"{CLIENTS} clients fail at once, then retry {TRIES} times each\n")
print(f"{'strategy':20} {'peak retries in any 100 ms':>28} {'last retry at':>14}")
for s in ("fixed 1s", "exponential", "exp + full jitter", "exp + equal jitter"):
times = schedule(s)
peak = max(Counter(int(t * 10) for t in times).values())
print(f"{s:20} {peak:>28,} {max(times):>13.1f}s")1000 clients fail at once, then retry 5 times each strategy peak retries in any 100 ms last retry at fixed 1s 1,000 5.0s exponential 1,000 31.0s exp + full jitter 154 28.0s exp + equal jitter 211 29.6s
Plain exponential backoff spreads retries over time but not within each retry wave: all 1,000 clients still arrive in the same 100 ms, again and again, which can keep a recovering service down. Full jitter (sleep = random(0, min(cap, base * 2^attempt))) cuts the peak by more than six times. Two more rules: retry in one layer only — three layers each retrying three times turns one failure into 27 calls — and use a retry budget (retries may add at most, say, 10% on top of normal traffic) so retries cannot become the overload.
Circuit breakers
When a dependency is clearly down, retrying it with a 2-second timeout on every request wastes 2 seconds of every request and a thread each. A circuit breaker watches the failure rate and, past a threshold, opens: calls fail immediately, without touching the network, and the caller uses a fallback. After a cool-down it goes half-open and lets a few trial requests through; if they succeed it closes, if not it opens again.
calls pass through
count failures] -->|5 consecutive failures
or half of the last 20 fail| O[OPEN
fail fast
use fallback] O -->|cool-down of 10 s elapsed| H[HALF_OPEN
allow trial calls] H -->|2 trials succeed| C H -->|any trial fails| O
Closed passes calls and counts failures. Open fails fast for a cool-down. Half-open lets trial calls decide whether the dependency has recovered.
class CircuitOpen(Exception):
pass
class CircuitBreaker:
"""CLOSED -> (N consecutive failures) -> OPEN -> (cool-down) -> HALF_OPEN
HALF_OPEN -> (trial succeeds k times) -> CLOSED, or (trial fails) -> OPEN"""
def __init__(self, clock, failure_threshold=5, cooldown_s=10, half_open_successes=2):
self.clock, self.threshold = clock, failure_threshold
self.cooldown, self.needed = cooldown_s, half_open_successes
self.state, self.failures, self.successes, self.opened_at = "CLOSED", 0, 0, None
self.log = []
def _go(self, state):
self.log.append(f"t={self.clock():<3} {self.state:>9} -> {state}")
self.state = state
def call(self, fn):
if self.state == "OPEN":
if self.clock() - self.opened_at < self.cooldown:
raise CircuitOpen("fail fast") # no network call at all
self._go("HALF_OPEN") # let trial requests through
self.successes = 0
try:
result = fn()
except Exception:
self.failures += 1
if self.state == "HALF_OPEN" or self.failures >= self.threshold:
self._go("OPEN")
self.opened_at, self.failures = self.clock(), 0
raise
if self.state == "HALF_OPEN":
self.successes += 1
if self.successes >= self.needed:
self._go("CLOSED")
self.failures = 0
return result
# --- simulation: the recommendations service is down from t=3 s to t=24 s ---
now = 0
wasted_calls = 0 # calls that hit the dead service
def recommendations():
global wasted_calls
if 3 <= now < 24:
wasted_calls += 1
raise TimeoutError("timed out after 2 s")
return ["item-1", "item-2"]
cb = CircuitBreaker(clock=lambda: now)
served = {"live": 0, "fallback": 0}
for now in range(40): # one request per second
try:
cb.call(recommendations)
served["live"] += 1
except (CircuitOpen, TimeoutError):
served["fallback"] += 1 # e.g. cached bestsellers
print("\n".join(cb.log))
print(f"\n40 requests: {served}")
print(f"calls that waited 2 s on the dead service: {wasted_calls} with the breaker, "
f"21 without it")t=7 CLOSED -> OPEN
t=17 OPEN -> HALF_OPEN
t=17 HALF_OPEN -> OPEN
t=27 OPEN -> HALF_OPEN
t=28 HALF_OPEN -> CLOSED
40 requests: {'live': 16, 'fallback': 24}
calls that waited 2 s on the dead service: 6 with the breaker, 21 without it
The dependency was down for 21 seconds; only 6 requests paid the full timeout, and the other 15 got an instant fallback. The price is visible too: the dependency recovered at t=24, but three more requests got the fallback until the cool-down ended at t=27 — cool-down length trades recovery speed against hammering a service that is still sick. Production breakers (Resilience4j, Polly, Envoy outlier detection, Hystrix historically) use a failure rate over a sliding window with a minimum number of calls, count slow calls as failures, and keep one breaker per dependency — and ideally per endpoint, so one broken endpoint does not cut off a healthy one.
Bulkheads
Named after the watertight compartments of a ship: partition resources so that a flood in one compartment does not sink the rest. Give each dependency its own connection pool, thread pool or concurrency limit. Worked example: a product page service has 200 worker threads and calls reviews, pricing and inventory. The reviews service starts taking 10 seconds per call; without bulkheads all 200 threads are soon waiting on reviews and pricing requests cannot be served either. With a 30-thread limit on reviews, reviews calls beyond 30 are rejected immediately (fall back to "reviews unavailable"), and the other 170 threads keep serving the page. The same idea applies at larger scale: separate clusters for internal batch jobs and user traffic, and per-tenant capacity pools so one customer's surge cannot starve the others.
Load shedding
When a service is past its capacity, trying to serve everything means serving everything slowly, and then timing out on everything — goodput collapses toward zero. Load shedding deliberately rejects some requests early and cheaply (a fast 503 with Retry-After) so the ones it accepts complete within their deadline. Shed by priority: in the food delivery capstone the order is place-order, then tracking, then browse, then recommendations, then analytics. Signals to shed on: in-flight request count or queue wait time above a limit (better than CPU), or adaptive concurrency limits that back off as latency rises. Drop requests whose deadline has already passed — the client has given up. Under overload, serving the newest requests first (LIFO) serves users still waiting rather than ones who already left.
Graceful degradation
Decide in advance what each feature becomes when its dependency fails, so an outage turns into a slightly worse page rather than an error page.
| Dependency down | Degraded behaviour |
|---|---|
| Recommendations | Show cached bestsellers for the category |
| Personalised feed ranking | Chronological or popularity order from cache |
| Search cluster | Prefix match on names from the primary database, fewer filters |
| Reviews service | Hide the reviews section; keep the product and "Buy" button |
| Payment provider A | Route to provider B; or accept the order and capture payment asynchronously |
| Email provider | Queue the messages; the send happens after recovery |
| Feature-flag service | Use the last known flags cached locally, or safe defaults |
Make degradations switchable with feature flags ("kill switches"), and test them — a fallback path that has never run in production is a second outage waiting to happen. Chaos experiments and game days exist to exercise exactly these paths.
Step 11 — SLIs, SLOs and error budgets
- SLI (service level indicator) — a measured ratio of good events to valid events, as close to the user as you can measure it. For example, "the proportion of checkout requests that returned a non-5xx response in under 800 ms", measured at the load balancer.
- SLO (service level objective) — the target for an SLI over a window: 99.9% over a rolling 30 days.
- SLA (service level agreement) — a contract with consequences (credits, refunds) if missed. Set it looser than the SLO, so you breach the internal target long before the contractual one.
- Error budget — 100% minus the SLO. It is the amount of unreliability you are allowed to spend on deploys, experiments and incidents. When it is exhausted, reliability work takes priority over launches; when there is plenty left, move faster.
MIN_30D = 30 * 24 * 60
print("SLO error budget / 30 days failed requests allowed at 2,000 req/s")
for slo in (0.99, 0.999, 0.9995, 0.9999):
budget = 1 - slo
reqs = 2_000 * 86_400 * 30 * budget
print(f"{slo:<8.2%} {budget * MIN_30D:8.1f} min of full outage {reqs:>14,.0f}")
# Burn rate = observed error rate / budgeted error rate.
# Burn rate 1 spends exactly the whole budget in 30 days.
slo = 0.999
print(f"\nSLO {slo:.1%}: time until the 30-day budget is gone")
for err in (0.001, 0.002, 0.0144, 0.05, 0.5):
burn = err / (1 - slo)
print(f" error rate {err:6.2%} -> burn rate {burn:5.1f}x -> budget gone in "
f"{30 * 24 / burn:6.1f} h")
# A common paging rule: burn rate > 14.4 over the last 1 h (AND over the last 5 min)
# means 2% of the monthly budget went in one hour -- wake someone up.
print(f"\n14.4x for 1 hour consumes {14.4 / (30 * 24):.0%} of the monthly budget")SLO error budget / 30 days failed requests allowed at 2,000 req/s 99.00% 432.0 min of full outage 51,840,000 99.90% 43.2 min of full outage 5,184,000 99.95% 21.6 min of full outage 2,592,000 99.99% 4.3 min of full outage 518,400 SLO 99.9%: time until the 30-day budget is gone error rate 0.10% -> burn rate 1.0x -> budget gone in 720.0 h error rate 0.20% -> burn rate 2.0x -> budget gone in 360.0 h error rate 1.44% -> burn rate 14.4x -> budget gone in 50.0 h error rate 5.00% -> burn rate 50.0x -> budget gone in 14.4 h error rate 50.00% -> burn rate 500.0x -> budget gone in 1.4 h 14.4x for 1 hour consumes 2% of the monthly budget
Each additional nine cuts the budget tenfold: 99.99% leaves four minutes a month, less than a single slow rollback, which means automated rollbacks, multi-region failover and change freezes. Do not promise more nines than your dependencies have — a service built on a 99.95% database cannot credibly offer 99.99% without redundancy around it. And alert on burn rate, not on individual errors: page when the budget is burning at 14.4x over the last hour (and still over the last five minutes, so the page stops once the problem does), open a ticket for a slow 1x–2x burn over days. This pages on what users feel and ignores CPU spikes that users never notice.
Choosing SLIs for a system in an interview: availability and latency for every user-facing request path; freshness for pipelines ("99% of events visible in search within 60 s"); durability for storage; and a business SLI or two (orders completed per checkout started) that catches failures every technical dashboard shows as green.
The trade-off summary — how to talk about communication and reliability
- Sync for what the user must wait for, async for everything else. Every synchronous dependency multiplies into your availability and tail latency.
- Queue for jobs, log for events. Per-message retries and DLQs versus retention, replay and many independent consumers.
- Ordering is per key. Choose the partition key by what must stay ordered, and leave headroom in the partition count.
- Assume at-least-once and make consumers idempotent with dedupe tables, versioned conditional writes and downstream idempotency keys.
- Never dual-write. Transactional outbox on the producer, inbox or dedupe on the consumer.
- Bound everything: queue sizes, timeouts, retries, concurrency per dependency. Unbounded things are where outages hide.
- Retries need backoff, jitter, a budget and a single owning layer; circuit breakers and bulkheads stop a sick dependency from spreading; load shedding and degradation decide who gets served when you cannot serve everyone.
- Pick the API style per client: REST for public and cacheable, gRPC between services, GraphQL for aggregating front-ends, SSE for one-way push, WebSocket for two-way realtime, long polling as the fallback.
- Define reliability as SLOs with error budgets, and alert on burn rate.
// Write your solution here
// Write your solution here
// Write your solution here
// Write your solution here
Finished reading? Mark this lesson complete to track your progress.
