System Design · Lesson 5 of 18

Queues, APIs and Reliability

Message queues, idempotency, retries, circuit breakers, WebSockets and API design.

  • Advanced
  • 50 min read
  • 8 objectives

Before this lessonLesson 4: Choosing and Scaling Databases

What you will learn

  • Decide what runs synchronously and what goes to a queue
  • Choose between a queue and a log, and design partitions and consumer groups
  • Build idempotent consumers for at-least-once delivery, with DLQs and backpressure
  • Publish events safely with the transactional outbox
  • Pick REST, gRPC, GraphQL, SSE, WebSocket or long polling per client
  • Design cursors, versioning, idempotency keys, errors and rate-limit headers
  • Apply timeouts, jittered retries, circuit breakers, bulkheads and load shedding
  • Set SLIs, SLOs and error budgets with burn-rate alerts

Your Progress

0 of 18 lessons 0%

  • Lessons0 / 18
  • Completed0
  • Est. time left~ 16 hours

Progress is saved in this browser. No account needed.

Tip: pressing Next marks this lesson complete automatically.

Every system in this course is a set of services talking to each other and to clients. This lesson is about how they talk and what happens when the conversation goes wrong: when to call synchronously and when to hand work to a queue, how queues differ from logs like Kafka, how ordering and delivery guarantees actually work, how to publish events without losing them, which API style fits which client, the small API design details that separate a professional API from a fragile one, and the reliability patterns — timeouts, retries with jitter, circuit breakers, bulkheads, load shedding — that keep one failure from becoming an outage. It closes with SLIs, SLOs and error budgets, the numbers that tell you how reliable is reliable enough.

Each concept has a worked example, and the Python runs in your browser. The database side of reliability (replication, transactions, sagas) is in Lesson 4; the case studies that lean hardest on this lesson are notifications, chat, video and the food delivery capstone.

Step 1 — Synchronous or asynchronous?

A synchronous call means the caller waits for the answer: the user's HTTP request stays open while your service calls payments, inventory and email. An asynchronous hand-off means the caller records the intent (a message on a queue) and returns; a worker does the job later. The question to ask of every step in a request is: does the user need the result of this step before we can answer them?

Checkout is the classic worked example. The user needs to know the payment was authorised and the stock reserved — synchronous. They do not need the confirmation email to have been sent, the analytics event recorded, the loyalty points credited or the warehouse notified before the "Thank you" page renders — all asynchronous. Moving those four out of the request path is not a micro-optimisation; it changes the availability and latency of checkout.

MIN_PER_MONTH = 30 * 24 * 60

# A request that synchronously calls N services, each 99.9% available,
# is only up when ALL of them are up at the same moment.
print("services in the sync path   availability   downtime/month")
for n in (1, 3, 5, 10):
    a = 0.999 ** n
    print(f"{n:>10}                  {a:8.3%}      {(1 - a) * MIN_PER_MONTH:6.0f} min")

# Tail latency amplifies too: if a page fans out to N backends in parallel,
# the page is slow whenever ANY backend has a p99-slow response.
print("\nparallel calls   chance the page hits at least one p99-slow call")
for n in (1, 10, 50, 100):
    print(f"{n:>10}       {1 - 0.99 ** n:6.1%}")

# The async version: the request path only needs the API and the queue.
checkout_sync = 0.999 ** 5            # api, payments, inventory, email, analytics
checkout_async = 0.999 ** 3 * 0.9999  # api, payments, inventory + a durable queue
print(f"\ncheckout sync over 5 services:     {checkout_sync:.3%}")
print(f"checkout with email+analytics async: {checkout_async:.3%}")
Output
services in the sync path   availability   downtime/month
         1                   99.900%          43 min
         3                   99.700%         129 min
         5                   99.501%         216 min
        10                   99.004%         430 min

parallel calls   chance the page hits at least one p99-slow call
         1         1.0%
        10         9.6%
        50        39.5%
       100        63.4%

checkout sync over 5 services:     99.501%
checkout with email+analytics async: 99.690%

Availability multiplies along a synchronous chain: five 99.9% services in series give you 99.5%, or three and a half hours of downtime a month. And fan-out amplifies tail latency: a page that calls 100 backends in parallel hits at least one backend's p99 on 63% of loads, so your p50 page is built from their p99s. Taking non-essential work off the path removes both its latency and its failures from the user's request.

What async buys, and what it costs

  • Buys: spike absorption (10,000 sign-ups in a minute become a steady drain), failure isolation (the email provider is down, messages wait, nobody sees a 500), independent scaling of workers, and fan-out to many consumers from one event.
  • Costs: eventual consistency (the email arrives in 2 seconds, or 20 minutes during a backlog), duplicates to handle (Step 4), a new component to operate and monitor, and harder debugging — a request is now a trail across processes, so you need trace ids in message headers.
  • Wrong answer: "make everything async". If the user needs the answer (is my payment accepted?), a queue just makes them poll for it. Request/response is the right tool for queries and for commands whose result the caller needs.

Step 2 — Queues versus logs

"Message broker" covers two different designs. A queue (Amazon SQS, RabbitMQ, ActiveMQ, Google Cloud Tasks) hands each message to one consumer and deletes it once acknowledged: it is a to-do list shared by workers. A log (Apache Kafka, Amazon Kinesis, Apache Pulsar, Redpanda) appends messages to an ordered, retained sequence; consumers track their own position (offset) and reading does not remove anything: it is a newspaper archive that any number of readers can read at their own pace, and re-read.

flowchart LR subgraph Q[Queue - SQS or RabbitMQ] P1[Producer] --> QQ[(jobs)] QQ -->|msg 1| W1[Worker A] QQ -->|msg 2| W2[Worker B] QQ -->|msg 3| W1 end subgraph L[Log - Kafka] P2[Producer] --> LOG[(order.events
offsets 0..N, kept 7 days)] LOG -->|group billing
at offset 1041| B[Billing] LOG -->|group analytics
at offset 870| A[Analytics] LOG -->|new group
replays from 0| N[Search indexer] end

Left: a queue deletes a message once one worker acknowledges it. Right: a log keeps messages; each consumer group has its own offset, so billing and analytics read the same events independently, and either can rewind.

Queue (SQS, RabbitMQ)Log (Kafka, Kinesis, Pulsar)
ConsumptionCompeting consumers; each message to one workerEach consumer group reads every message; within a group, each partition goes to one consumer
After processingAcked message is deletedNothing is deleted; the group's offset advances
Retention and replayUntil consumed (SQS max 14 days); no replayBy time or size (days to forever); rewind to reprocess or bootstrap a new consumer
OrderingBest effort (SQS standard); FIFO queues order per message groupStrict within a partition
Per-message controlIndividual ack, delay, visibility timeout, per-message retry and DLQOffset per partition: one stuck message blocks everything behind it in that partition
RoutingRabbitMQ exchanges: topic, fanout, header routing; prioritiesBy topic and key only
ThroughputThousands to tens of thousands/s per queue (SQS scales further horizontally)Millions/s per cluster; sequential disk writes and batching
Scaling consumersAdd workers any timeUp to the partition count per group
Best forTask/job queues: send email, resize image, charge card, call a webhookEvent streams: order events, clickstreams, CDC, metrics, event sourcing, feeding many systems

Worked choices. Thumbnail generation for uploaded photos: a queue — each job is independent, needs per-job retry, and nobody will replay it. "Order placed" consumed by billing, email, fraud, analytics and a search indexer: a log — five consumers, each at its own pace, and next quarter's new recommendation service can replay the last 30 days to build its state. Publish/subscribe fan-out to a handful of queues (Amazon SNS to several SQS queues, or a RabbitMQ fanout exchange) is the middle ground when you want fan-out but per-message retries.

Step 3 — Partitions, consumer groups and ordering

A Kafka topic is split into partitions, each an independent ordered log on a different broker. A producer chooses a partition by hashing the message key; messages with the same key always land in the same partition, in the order they were produced. A consumer group divides a topic's partitions among its members: each partition is read by exactly one consumer of the group at a time. Those two rules together give the only ordering guarantee you get — per key — and they cap the group's parallelism at the partition count.

import hashlib
from collections import defaultdict

PARTITIONS = 6

def partition_for(key):
    # Kafka's default partitioner is murmur2(key) % partitions; md5 stands in here
    return int(hashlib.md5(key.encode()).hexdigest(), 16) % PARTITIONS

# Events for three orders, produced interleaved. Keyed by order_id.
events = [("o1", "CREATED"), ("o2", "CREATED"), ("o1", "PAID"), ("o3", "CREATED"),
          ("o2", "PAID"), ("o1", "SHIPPED"), ("o3", "PAID"), ("o2", "CANCELLED")]
log = defaultdict(list)                       # partition -> ordered list of records
for key, ev in events:
    log[partition_for(key)].append((key, ev))

for p in sorted(log):
    print(f"partition {p}: {log[p]}")

def assign(consumers):
    """Round-robin assignment: each partition goes to exactly ONE consumer in the group."""
    out = defaultdict(list)
    for p in range(PARTITIONS):
        out[consumers[p % len(consumers)]].append(p)
    return dict(out)

print("\ngroup 'billing' with 2 consumers:", assign(["c1", "c2"]))
print("group 'billing' with 3 consumers:", assign(["c1", "c2", "c3"]))
a = assign([f"c{i}" for i in range(1, 9)])
idle = 8 - len(a)
print(f"group 'billing' with 8 consumers: {len(a)} busy, {idle} idle "
      f"(parallelism is capped by the partition count)")

# Every order's events stay in order, because one key -> one partition -> one consumer.
for oid in ("o1", "o2", "o3"):
    seq = [ev for key, ev in log[partition_for(oid)] if key == oid]
    print(f"{oid} as seen by its consumer: {seq}")
Output
partition 0: [('o3', 'CREATED'), ('o3', 'PAID')]
partition 4: [('o1', 'CREATED'), ('o2', 'CREATED'), ('o1', 'PAID'), ('o2', 'PAID'), ('o1', 'SHIPPED'), ('o2', 'CANCELLED')]

group 'billing' with 2 consumers: {'c1': [0, 2, 4], 'c2': [1, 3, 5]}
group 'billing' with 3 consumers: {'c1': [0, 3], 'c2': [1, 4], 'c3': [2, 5]}
group 'billing' with 8 consumers: 6 busy, 2 idle (parallelism is capped by the partition count)
o1 as seen by its consumer: ['CREATED', 'PAID', 'SHIPPED']
o2 as seen by its consumer: ['CREATED', 'PAID', 'CANCELLED']
o3 as seen by its consumer: ['CREATED', 'PAID']
  • Choose the key by what must stay ordered. Order events keyed by order_id keep CREATED → PAID → SHIPPED in sequence. Keying by customer_id also works and additionally orders a customer's orders relative to each other. No key (round-robin) maximises spread and gives no ordering.
  • Global ordering requires a single partition, which caps you at one consumer. Almost no business process needs it; say so if asked.
  • Choose the partition count with headroom (for example, 3–4x today's consumer count). Adding partitions later changes hash(key) % partitions, so existing keys move and per-key ordering breaks across the change.
  • Hot keys make hot partitions. One merchant producing 40% of events will pin one partition and one consumer at 100%. Split that key (merchant_id + sub-bucket) when its events do not need mutual ordering.
  • Rebalances happen whenever a consumer joins, leaves or stalls past its poll timeout. Partitions move, and messages processed but not yet committed are redelivered to the new owner — one more reason consumers must be idempotent.
  • Queues have an equivalent: SQS FIFO orders within a MessageGroupId and only one message per group is in flight at a time; RabbitMQ has single-active-consumer and consistent-hash exchanges.

Step 4 — Delivery semantics and idempotent consumers

A consumer does two things per message: the work and the acknowledgement (ack, or offset commit). A crash can land between them. Where you place the ack decides the guarantee:

  • At-most-once — ack first, then work. A crash after the ack loses the message. Acceptable for metrics samples or a presence ping; never for money.
  • At-least-once — work first, then ack. A crash before the ack means the message is redelivered and the work happens twice. This is the default of SQS, RabbitMQ and Kafka consumers, and what you should assume.
  • Exactly-once — not something the network can give you end to end. What you can build is effectively-once: at-least-once delivery plus an idempotent consumer, so duplicates have no additional effect.
def consume(mode):
    """Four messages, each meaning 'add $10 to the balance'. The consumer
    crashes once, around message m2; the broker redelivers anything unacked."""
    queue = ["m1", "m2", "m3", "m4"]
    acked, seen = set(), set()
    state = {"balance": 0, "crash_pending": True}

    def handle(msg):
        if mode == "effectively-once":
            if msg in seen:                 # duplicate delivery: skip the effect
                return
            seen.add(msg)                   # real life: INSERT INTO processed(msg_id)
        state["balance"] += 10              # in the SAME DB transaction as this update

    def crash_here(msg):
        if state["crash_pending"] and msg == "m2":
            state["crash_pending"] = False
            return True
        return False

    while len(acked) < len(queue):
        for msg in [m for m in queue if m not in acked]:   # (re)poll unacked messages
            if mode == "at-most-once":
                acked.add(msg)              # ack first...
                if crash_here(msg):
                    break                   # ...crash before the work: m2 is gone forever
                handle(msg)
            else:
                handle(msg)                 # work first...
                if crash_here(msg):
                    break                   # ...crash before the ack: m2 comes back
                acked.add(msg)
    return state["balance"]

for mode in ("at-most-once", "at-least-once", "effectively-once"):
    print(f"{mode:17} balance = ${consume(mode)}   (expected $40)")
Output
at-most-once      balance = $30   (expected $40)
at-least-once     balance = $50   (expected $40)
effectively-once  balance = $40   (expected $40)

How to make a consumer idempotent

  • Naturally idempotent operations. SET status = 'SHIPPED' can run twice; balance = balance + 10 cannot. Prefer absolute writes ("set to X") over relative ones ("add X") where the domain allows.
  • A dedupe table in the same transaction. INSERT INTO processed_messages(message_id) with a unique constraint, in the same database transaction as the effect; a duplicate hits the constraint and the whole transaction is skipped. The key must be a stable id assigned by the producer, not something regenerated on retry.
  • Conditional updates with versions. UPDATE orders SET state='PAID', seq=17 WHERE id=? AND seq < 17 — duplicates and stale, out-of-order events both become no-ops.
  • Idempotency keys passed downstream. When the effect is an external call (a payment provider, an email API), pass a key derived from the message id so the provider dedupes too.
  • Kafka's exactly-once (idempotent producer plus transactions) covers read-from-Kafka, write-to-Kafka pipelines such as Kafka Streams. It does not cover your database or an email API; those still need the techniques above.

Step 5 — Retries, poison messages and dead-letter queues

Failures come in two kinds. Transient failures (a timeout, a throttled downstream, a deadlock) succeed if you retry later. Permanent failures — a poison message with malformed JSON, a reference to a deleted user, a bug your code hits every time — will fail forever, and if you retry them forever they burn capacity and, in a log, block the partition behind them. A dead-letter queue (DLQ) separates the two: after N attempts the message is moved aside with its error, so the main flow keeps moving and a human or a tool can deal with it.

MAX_RECEIVES = 3            # SQS maxReceiveCount / RabbitMQ delivery-limit

attempts = {}
def handler(msg):
    n = attempts[msg["id"]] = attempts.get(msg["id"], 0) + 1
    if msg["id"] == "img-2" and n <= 2:
        raise TimeoutError("thumbnail service timed out")      # transient
    if msg["body"] is None:
        raise ValueError("body is not valid JSON")            # poison: will never work
    return "ok"

queue = [{"id": "img-1", "body": "cat.jpg"}, {"id": "img-2", "body": "dog.jpg"},
         {"id": "img-3", "body": None}, {"id": "img-4", "body": "fox.jpg"}]
receives, dlq, tick = {}, [], 0
while queue:
    tick += 1
    msg = queue.pop(0)
    receives[msg["id"]] = receives.get(msg["id"], 0) + 1
    try:
        handler(msg)
        print(f"t{tick:<2} {msg['id']} delivered (receive #{receives[msg['id']]})")
    except Exception as e:
        if receives[msg["id"]] >= MAX_RECEIVES:
            dlq.append({**msg, "error": repr(e), "receives": receives[msg["id"]]})
            print(f"t{tick:<2} {msg['id']} -> DLQ after {receives[msg['id']]} receives: {e}")
        else:
            queue.append(msg)       # becomes visible again after the visibility timeout
            print(f"t{tick:<2} {msg['id']} failed ({e}); will be redelivered")

print("\nDLQ:", dlq)
Output
t1  img-1 delivered (receive #1)
t2  img-2 failed (thumbnail service timed out); will be redelivered
t3  img-3 failed (body is not valid JSON); will be redelivered
t4  img-4 delivered (receive #1)
t5  img-2 failed (thumbnail service timed out); will be redelivered
t6  img-3 failed (body is not valid JSON); will be redelivered
t7  img-2 delivered (receive #3)
t8  img-3 -> DLQ after 3 receives: body is not valid JSON

DLQ: [{'id': 'img-3', 'body': None, 'error': "ValueError('body is not valid JSON')", 'receives': 3}]
  • Visibility timeout (SQS) or unacked redelivery (RabbitMQ): a received message is hidden, not deleted; if the worker does not ack within the timeout it reappears. Set it comfortably above the p99 processing time, or long jobs get processed twice concurrently; extend it (heartbeat) for jobs of unpredictable length.
  • Back off between attempts (Step 10) — retrying a throttled API immediately just gets throttled again. SQS delay queues, RabbitMQ delayed/TTL queues, and in Kafka a chain of retry topics (orders.retry.1m, orders.retry.10m) implement delayed retries.
  • Fail fast on permanent errors. A validation error should go straight to the DLQ rather than waste three attempts; classify exceptions.
  • The DLQ is not a bin. Alert on its depth, record the error, stack trace, attempt count and original metadata, and build a redrive tool to replay messages after the fix. A DLQ nobody watches is data loss with extra steps.
  • Ordering caveat: moving one message aside lets later messages for the same key overtake it. When per-key order matters, park the whole key (divert its later messages too) until the first one is resolved.

Step 6 — Backpressure: when producers outrun consumers

A queue absorbs a spike only if the spike ends. When arrivals exceed processing capacity, the backlog grows linearly, and by Little's law the waiting time grows with it: delay = backlog ÷ throughput. Backpressure is any mechanism that pushes the overload back toward the source instead of letting it silently accumulate.

# A flash sale: producers publish 3,000 msg/s for 10 minutes, then 500 msg/s.
# Each consumer processes 100 msg/s; we run 20 of them (2,000 msg/s capacity).
CONSUMERS, PER_CONSUMER = 20, 100
capacity = CONSUMERS * PER_CONSUMER

def arrivals(t):
    return 3_000 if t < 600 else 500

def simulate(queue_limit=None, minutes=40):
    backlog, peak, rejected, drained_at = 0, 0, 0, None
    for t in range(minutes * 60):
        backlog += arrivals(t)
        if queue_limit is not None and backlog > queue_limit:
            rejected += backlog - queue_limit     # producer gets "queue full" -> 429 / 503
            backlog = queue_limit
        backlog = max(0, backlog - capacity)
        peak = max(peak, backlog)
        if t >= 600 and backlog == 0 and drained_at is None:
            drained_at = t
    return peak, rejected, drained_at

peak, rejected, drained = simulate()
print(f"unbounded queue: peak backlog {peak:,} msgs, "
      f"drained {(drained - 600) / 60:.0f} min after the spike ends")
# Little's law: time waiting in queue = backlog / throughput
print(f"  worst queueing delay = {peak / capacity / 60:.0f} min "
      "(fine for emails, fatal for 'your ride is here')")

peak, rejected, drained = simulate(queue_limit=120_000)
print(f"bounded at 120k: peak {peak:,}, rejected {rejected:,} msgs, "
      f"worst delay {peak / capacity:.0f} s")

# Scaling consumers only helps once capacity exceeds the peak arrival rate.
for n in (20, 25, 30):
    cap = n * PER_CONSUMER
    growth = max(0, 3_000 - cap) * 600
    print(f"{n} consumers: backlog grows {max(0, 3_000 - cap):,}/s -> "
          f"{growth:,} msgs, delay {growth / cap:,.0f} s")
Output
unbounded queue: peak backlog 600,000 msgs, drained 7 min after the spike ends
  worst queueing delay = 5 min (fine for emails, fatal for 'your ride is here')
bounded at 120k: peak 118,000, rejected 482,000 msgs, worst delay 59 s
20 consumers: backlog grows 1,000/s -> 600,000 msgs, delay 300 s
25 consumers: backlog grows 500/s -> 300,000 msgs, delay 120 s
30 consumers: backlog grows 0/s -> 0 msgs, delay 0 s

The numbers make the trade explicit. An unbounded queue never rejects anything but delivers messages five minutes late — fine for newsletters, useless for "your driver has arrived". A bounded queue caps the delay at about a minute but rejects 482,000 messages, which producers must handle (retry later, degrade, or tell the user). And adding consumers only helps once total capacity exceeds the peak arrival rate — at 25 consumers the backlog still grows, just more slowly.

StrategyHowUse when
BufferDurable queue with generous retentionSpikes are short and delay is acceptable
Scale consumersAutoscale on queue depth or lag in seconds, not CPUThe bottleneck is the consumer, not a downstream database it hammers
Bound and rejectMax queue length; producer gets 429/503 or a blocked sendStale work is worthless; protect latency for what you accept
Rate-limit producersToken bucket per tenant at the API edge (rate limiter lesson)One noisy tenant should not flood everyone
Drop or sampleKeep 1 in 10 metrics points; drop superseded updates (only the newest location matters)Messages are samples, or newer ones replace older
Pull, not pushConsumers fetch at their own pace (Kafka, SQS long polling); gRPC and reactive streams have flow-control windowsAlways prefer it; push-based consumers need explicit credit/prefetch limits (RabbitMQ prefetch)

Step 7 — The transactional outbox

A service that must both update its database and publish an event faces the dual-write problem. Commit then publish, and a crash in between leaves an order nobody hears about. Publish then commit, and a failed commit leaves an event for an order that does not exist. There is no ordering of two separate systems that is safe, and 2PC across a database and a broker is rarely available or wise.

The fix is to make it one write. In the same local transaction as the business change, insert the event into an outbox table. A separate relay process reads unsent outbox rows, publishes them, and marks them sent. The event is published if and only if the transaction committed — at least once, so consumers dedupe on the event id.

sequenceDiagram participant S as Order service participant D as Orders DB participant R as Outbox relay participant K as Kafka participant C as Consumer S->>D: BEGIN S->>D: INSERT INTO orders ... S->>D: INSERT INTO outbox event_id=evt-2, topic=order.placed S->>D: COMMIT - both rows or neither R->>D: SELECT unsent rows ORDER BY id LIMIT 500 R->>K: produce evt-2, key=order_id K-->>R: ack R->>D: UPDATE outbox SET published_at=now() K->>C: evt-2 C->>C: INSERT processed evt-2, apply effect, same txn

One local transaction writes both the order and its event. The relay publishes afterwards; if it crashes before marking a row sent, the row is published again, and the consumer's dedupe on event_id absorbs the duplicate.

import itertools

class World:
    def __init__(self):
        self.db_orders, self.outbox, self.broker, self.ids = [], [], [], itertools.count(1)

def dual_write(w, crash_between):
    """Anti-pattern: commit to the DB, then publish to the broker."""
    oid = next(w.ids)
    w.db_orders.append(oid)                     # COMMIT
    if crash_between:
        return                                  # process killed / broker unreachable
    w.broker.append(("order.placed", oid))

def with_outbox(w):
    """Order row and event row in ONE local transaction."""
    oid = next(w.ids)
    w.db_orders.append(oid)                     # BEGIN; INSERT order;
    w.outbox.append({"event_id": f"evt-{oid}", "order": oid, "sent": False})  # INSERT outbox; COMMIT

def relay(w, crash_before_mark=False):
    """Separate process: publish unsent rows, then mark them sent."""
    for row in w.outbox:
        if not row["sent"]:
            w.broker.append(("order.placed", row["order"], row["event_id"]))
            if crash_before_mark:
                return                          # published, but not marked -> resent later
            row["sent"] = True

w = World()
dual_write(w, crash_between=False)
dual_write(w, crash_between=True)
print("dual write: orders", w.db_orders, "events", [e[1] for e in w.broker],
      "-> order 2 exists but nobody was told")

w = World()
with_outbox(w); with_outbox(w)
relay(w, crash_before_mark=True)                # relay dies mid-way
relay(w)                                        # restarts, resends the unmarked row
print("outbox:     orders", w.db_orders, "events", [e[2] for e in w.broker])

seen, applied = set(), []
for _, order, event_id in w.broker:             # the consumer dedupes by event_id
    if event_id not in seen:
        seen.add(event_id); applied.append(order)
print("consumer applied:", applied, "-> nothing lost, duplicate absorbed")
Output
dual write: orders [1, 2] events [1] -> order 2 exists but nobody was told
outbox:     orders [1, 2] events ['evt-1', 'evt-1', 'evt-2']
consumer applied: [1, 2] -> nothing lost, duplicate absorbed
  • Relay by polling — a loop running SELECT ... WHERE published_at IS NULL ORDER BY id LIMIT 500 FOR UPDATE SKIP LOCKED. Simple; adds some load and around 100 ms–1 s of delay.
  • Relay by change data capture — Debezium tails the database's write-ahead log and publishes outbox inserts. Lower latency, no polling load, preserves commit order; one more system to run.
  • Housekeeping — delete or partition published rows after a few days, or the table grows forever.
  • The mirror image, the inbox: the consumer records incoming event ids in its own database in the same transaction as the effect. Outbox on the producer plus inbox on the consumer is the standard recipe for effectively-once across services — and the reliable way to publish saga steps from Lesson 4.

Step 8 — API styles: REST, gRPC, GraphQL and the realtime family

The same operation -- "get order 88231 with its items" -- in each style.

REST (HTTP + JSON, resources and verbs)
  GET /v1/orders/88231
  200 OK   ETag: "v7"   Cache-Control: private, max-age=5
  { "id": "88231", "state": "PAID", "items": [{ "sku": "B-12", "qty": 2 }] }
  + cacheable by URL, debuggable with curl, universal client support
  - over/under-fetching: the mobile app wanted 3 fields, got 30; or needs 3 calls

gRPC (HTTP/2 + Protocol Buffers, generated stubs)
  service Orders { rpc GetOrder(GetOrderRequest) returns (Order); }
  message Order { string id = 1; State state = 2; repeated Item items = 3; }
  + binary (3-10x smaller than JSON), typed contracts, streaming both ways,
    deadlines propagate across hops -- ideal between internal services
  - browsers need gRPC-Web + a proxy; not human-readable; no URL caching

GraphQL (one endpoint, the client names the fields)
  POST /graphql
  { order(id: "88231") { state items { sku qty } restaurant { name } } }
  + one round trip for a nested screen; clients evolve without new endpoints
  - every query is a POST to one URL (CDN caching needs persisted queries);
    a naive resolver does N+1 database calls (use DataLoader batching);
    a malicious query can be arbitrarily expensive (depth/cost limits)

WebSocket (full-duplex frames over one upgraded TCP connection)
  GET /v1/orders/88231/live   Upgrade: websocket
  <- {"type":"state","state":"PICKED_UP"}   -> {"type":"ping"}
  + both sides send any time, tiny per-message overhead: chat, games, editors
  - stateful: sticky connections, custom reconnect/resume, harder to load balance

Server-Sent Events (one-way server push over plain HTTP)
  GET /v1/orders/88231/events   Accept: text/event-stream
  id: 41
  event: state
  data: {"state":"PICKED_UP"}
  + plain HTTP (works through most proxies), auto-reconnect with Last-Event-ID
  - server -> client only; text only; ~6 connections per domain on HTTP/1.1

Long polling (a GET the server holds open until there is news)
  GET /v1/orders/88231/events?after=41&wait=30
  ... server holds up to 30 s ... 200 [{"id":42,"state":"DELIVERED"}] | 204
  + works everywhere, even through the most hostile corporate proxy
  - a new request per event; the server holds one request per client
StyleTransportDirectionBest forWatch out for
RESTHTTP/1.1 or 2, JSONRequest/responsePublic APIs, CRUD resources, anything cacheableChatty clients, over-fetching; versioning discipline
gRPCHTTP/2, ProtobufUnary + client/server/bidi streamingService-to-service, low latency, polyglot internal contracts, mobile with generated clientsBrowser support needs a proxy; L7 load balancing needed (long-lived HTTP/2 connections)
GraphQLHTTP POST, JSONRequest/response (+ subscriptions)Aggregating many backends for varied front-ends (a backend-for-frontend)N+1 resolvers, query cost limits, caching
Short pollingHTTPClient asks repeatedlyRare updates, trivial implementation, CDN-cacheable statusMostly empty responses; delay up to the interval
Long pollingHTTP (held request)Server replies when there is newsHostile networks, fallback transport, simple notification feedsOne held request per client; re-request per event
Server-Sent EventsHTTP streaming, textServer → clientLive scores, dashboards, notifications, LLM token streamingOne-way; HTTP/1.1 connection limits per domain
WebSocketUpgraded TCP connectionFull duplexChat, multiplayer games, collaborative editing, tradingStateful connections: sticky routing, reconnect and resume logic, pub/sub backplane
WebhooksHTTP POST to the client's serverServer → serverNotifying third parties (payment succeeded)Sign payloads, retry with backoff, receivers must dedupe

Polling, long polling, SSE and WebSocket compared on one timeline

sequenceDiagram participant C as Client participant S as Server Note over C,S: Short polling C->>S: GET /updates S-->>C: 200 empty C->>S: GET /updates, 5 s later S-->>C: 200 one event Note over C,S: Long polling C->>S: GET /updates?wait=30 Note right of S: holds the request until an event S-->>C: 200 one event C->>S: GET /updates?wait=30, immediately Note over C,S: Server-Sent Events C->>S: GET /stream, Accept text/event-stream S-->>C: event 1 S-->>C: event 2 Note over C,S: WebSocket C->>S: HTTP Upgrade websocket S-->>C: 101 Switching Protocols C->>S: frame - typing S-->>C: frame - new message

Short polling asks on a timer and usually hears nothing. Long polling parks a request until news arrives. SSE and WebSocket keep one connection open and push; only WebSocket lets the client send on the same connection.

CLIENTS = 1_000_000
EVENTS_PER_CLIENT_PER_MIN = 0.5          # e.g. a notification badge updates every 2 min

events_per_s = CLIENTS * EVENTS_PER_CLIENT_PER_MIN / 60
print(f"useful events: {events_per_s:,.0f}/s\n")

# Short polling every 5 s: every client asks, whether or not anything happened
poll_rps = CLIENTS / 5
print(f"short poll /5s : {poll_rps:>9,.0f} req/s, {events_per_s / poll_rps:5.1%} carry data, "
      f"worst-case delay 5 s")

# Long polling: one request is parked per client until an event or a 30 s timeout
lp_rps = events_per_s + CLIENTS / 30     # re-request after each event or each timeout
print(f"long poll 30s  : {lp_rps:>9,.0f} req/s, {CLIENTS:,} requests held open, delay ~0")

# SSE / WebSocket: one persistent connection, the server pushes each event
KB_PER_CONN = 30                          # kernel buffers + TLS + app state, roughly
CONNS_PER_NODE = 100_000
print(f"SSE/WebSocket  : {events_per_s:>9,.0f} pushes/s, {CLIENTS:,} connections "
      f"= ~{CLIENTS * KB_PER_CONN / 1e6:.0f} GB RAM across "
      f"{CLIENTS // CONNS_PER_NODE} nodes (+ headroom)")
Output
useful events: 8,333/s

short poll /5s :   200,000 req/s,  4.2% carry data, worst-case delay 5 s
long poll 30s  :    41,667 req/s, 1,000,000 requests held open, delay ~0
SSE/WebSocket  :     8,333 pushes/s, 1,000,000 connections = ~30 GB RAM across 10 nodes (+ headroom)

For a million clients receiving one event every two minutes, short polling generates 200,000 requests per second, 96% of them empty. Long polling cuts that fivefold. A persistent connection sends only the 8,333 useful messages per second — but holds a million connections, around 30 GB of memory across a dedicated tier. That is why realtime systems run a separate, stateful connection gateway (deployed independently of the stateless API so a deploy does not drop every socket), a pub/sub backplane such as Redis pub/sub or Kafka so any API server can reach a user connected to any gateway node, and a registry of which node holds which user. The chat lesson builds this.

Step 9 — API design details that matter in interviews and in production

## Pagination: opaque cursor, not page numbers
GET /v1/orders?limit=50&cursor=eyJ0cyI6MTcwMDAwMDAwMCwiaWQiOjg4MjMxfQ
200 { "data": [...50 orders...],
      "next_cursor": "eyJ0cyI6MTY5OTk5OTEyMCwiaWQiOjg3OTAyfQ",   # null on the last page
      "has_more": true }
  # The cursor encodes the sort key of the last row (created_at, id). It is
  # opaque (base64, optionally signed) so clients cannot build or depend on it.

## Versioning
GET /v1/orders/88231                      # major version in the path, for breaking changes
Stripe-Version: 2024-06-20                # or date-pinned versions per API key
  # Additive changes (new optional field, new endpoint, new enum value that
  # clients were told to tolerate) ship WITHOUT a new version.
  # Breaking: removing/renaming a field, changing a type or a default, new
  # required input. Run v1 and v2 side by side; announce a sunset date;
  # send "Deprecation" and "Sunset" response headers; measure who still calls v1.

## Idempotency keys on every non-idempotent POST
POST /v1/payments
Idempotency-Key: 5f0c1d3e-7a4b-4c1e-9f66-0b1a6c2d9e11   # client-generated UUID
{ "order_id": "o-77", "amount_minor": 4999, "currency": "EUR" }
  -> 201 first time; the SAME 201 body on any retry with the same key (for 24 h)
  -> 409 if the first request is still in progress
  -> 422 if the key is reused with a different body

## Errors: one machine-readable format everywhere (RFC 9457 problem+json)
HTTP/1.1 409 Conflict
Content-Type: application/problem+json
{ "type": "https://api.example.com/errors/order-not-cancellable",
  "title": "Order can no longer be cancelled",
  "status": 409,
  "detail": "Order 88231 was picked up at 12:41; cancellation closes at pickup.",
  "instance": "/v1/orders/88231/cancel",
  "code": "ORDER_NOT_CANCELLABLE",          # stable string clients switch on
  "request_id": "req_7Hc2kQ" }              # the support team greps logs for this

## Status codes that carry meaning
400 malformed | 401 no/invalid credentials | 403 authenticated but not allowed
404 not found | 409 state conflict | 412 If-Match precondition failed (stale ETag)
422 valid JSON, invalid values | 429 rate limited | 500 our bug
502/504 an upstream failed or timed out | 503 overloaded or in maintenance
  # Clients should retry: 429, 502, 503, 504 (with backoff, honouring Retry-After)
  # Clients must NOT blindly retry: 400, 401, 403, 404, 409, 422

## Rate limit headers (IETF RateLimit fields; many APIs still use X-RateLimit-*)
HTTP/1.1 200 OK
RateLimit-Policy: 100;w=60                 # 100 requests per 60 s window
RateLimit: limit=100, remaining=37, reset=21

HTTP/1.1 429 Too Many Requests
Retry-After: 21
Content-Type: application/problem+json
{ "type": ".../errors/rate-limited", "status": 429, "detail": "100 requests per minute" }

Worked example: why cursors, not offsets

import base64, json

# A feed ordered newest first. Each row: (created_at, id)
feed = [(100 - i, f"p{100 - i}") for i in range(12)]          # p100 ... p89

def offset_page(rows, offset, limit=4):
    return [r[1] for r in rows[offset:offset + limit]]

def encode(row):
    return base64.urlsafe_b64encode(json.dumps(row).encode()).decode()

def keyset_page(rows, cursor=None, limit=4):
    # WHERE (created_at, id) < (:c_ts, :c_id) ORDER BY created_at DESC, id DESC LIMIT :limit
    if cursor:
        c = tuple(json.loads(base64.urlsafe_b64decode(cursor)))
        rows = [r for r in rows if r < c]
    page = rows[:limit]
    return [r[1] for r in page], encode(page[-1])

print("offset page 1:", offset_page(feed, 0))
p1, cur = keyset_page(feed)
print("keyset page 1:", p1, " next_cursor =", cur)

# Two new posts arrive at the top before the user scrolls.
feed = [(102, "p102"), (101, "p101")] + feed

print("offset page 2:", offset_page(feed, 4), " <- p98, p97 shown twice")
p2, _ = keyset_page(feed, cur)
print("keyset page 2:", p2, " <- continues exactly where page 1 ended")

# Why offsets are also slow: OFFSET 1,000,000 still reads and discards 1M rows.
# The keyset query is an index seek to the cursor, whatever page you are on.
Output
offset page 1: ['p100', 'p99', 'p98', 'p97']
keyset page 1: ['p100', 'p99', 'p98', 'p97']  next_cursor = Wzk3LCAicDk3Il0=
offset page 2: ['p98', 'p97', 'p96', 'p95']  <- p98, p97 shown twice
keyset page 2: ['p96', 'p95', 'p94', 'p93']  <- continues exactly where page 1 ended

Offset pagination breaks on any list that changes while someone reads it: two new posts push two old ones onto page 2, and the user sees them twice (a deletion makes them skip rows instead). It is also slow at depth, because OFFSET 1000000 reads and discards a million rows. A keyset cursor — "rows strictly after (created_at, id) of the last row I saw" — is an index seek at any depth and is stable under inserts. Include a unique tiebreaker (id) in the sort, or rows with equal timestamps get skipped. Offsets are still fine for small, static admin tables where "jump to page 7" is a real need.

Worked example: an idempotency-key handler

import hashlib, json

store = {}          # idempotency_key -> {"hash", "state", "response"}; a DB table in real life
charges = []        # side effect we must not repeat

def handle_payment(key, user_id, body):
    req_hash = hashlib.sha256(json.dumps(body, sort_keys=True).encode()).hexdigest()
    rec = store.get((user_id, key))           # keys are scoped per user/API client
    if rec is None:
        # INSERT ... ON CONFLICT DO NOTHING: exactly one concurrent request wins the claim
        store[(user_id, key)] = {"hash": req_hash, "state": "IN_PROGRESS", "response": None}
        charges.append(body["amount_minor"])
        resp = (201, {"payment_id": f"pay_{len(charges)}", "status": "succeeded"})
        store[(user_id, key)].update(state="DONE", response=resp)
        return resp
    if rec["hash"] != req_hash:
        return (422, {"type": "/errors/idempotency-key-reuse",
                      "detail": "key was used with a different request body"})
    if rec["state"] == "IN_PROGRESS":
        return (409, {"type": "/errors/request-in-progress", "retry_after_s": 1})
    return rec["response"]                    # replay the ORIGINAL response, byte for byte

def show(label, resp):
    print(f"{label + ':':22}{resp}")

body = {"amount_minor": 4_999, "currency": "EUR", "order_id": "o-77"}
show("1st attempt", handle_payment("k-1", "u-9", body))
show("retry (timeout)", handle_payment("k-1", "u-9", body))
show("same key, new body", handle_payment("k-1", "u-9", {**body, "amount_minor": 9_999}))
# Simulate a second request arriving while the first is still talking to the PSP
body2 = {"amount_minor": 1_500, "currency": "EUR", "order_id": "o-78"}
h2 = hashlib.sha256(json.dumps(body2, sort_keys=True).encode()).hexdigest()
store[("u-9", "k-2")] = {"hash": h2, "state": "IN_PROGRESS", "response": None}
show("concurrent duplicate", handle_payment("k-2", "u-9", body2))
print("charges made:", charges)
Output
1st attempt:          (201, {'payment_id': 'pay_1', 'status': 'succeeded'})
retry (timeout):      (201, {'payment_id': 'pay_1', 'status': 'succeeded'})
same key, new body:   (422, {'type': '/errors/idempotency-key-reuse', 'detail': 'key was used with a different request body'})
concurrent duplicate: (409, {'type': '/errors/request-in-progress', 'retry_after_s': 1})
charges made: [4999]

The details that make this correct in production: the key is claimed with an atomic insert on a unique index before any side effect (so two concurrent retries cannot both proceed); the stored request hash turns accidental key reuse into a clear 422 rather than a silently wrong replay; the stored response is replayed exactly, including the status code; keys are scoped per client and expire after 24–48 hours; and a request that crashed mid-way (stuck IN_PROGRESS) is recovered by checking the downstream system with the same key rather than blindly re-running. The food delivery capstone uses exactly this table for order placement.

Versioning, briefly

Most changes should be additive and need no new version: new optional request fields, new response fields, new endpoints. That only works if clients are told to ignore unknown fields and tolerate new enum values — write that into the API contract from day one. Breaking changes get a new major version (/v2/) or a new dated version (the Stripe model, where each API key is pinned to the version it was created with and upgrades explicitly). Keep old versions running with a published sunset date, send Deprecation and Sunset headers, and measure traffic per version before turning anything off. For gRPC, never reuse a Protobuf field number; mark removed fields reserved.

Step 10 — Reliability patterns

Every remote call can fail, be slow, or — worst of all — succeed without you finding out. The patterns below exist to keep one sick dependency from taking down everything that calls it. They are layered: timeouts bound the wait, retries recover transient failures, circuit breakers stop retrying the hopeless, bulkheads contain the damage, load shedding protects the core, and graceful degradation decides what the user sees.

Timeouts and deadlines

Every network call needs a timeout; the default in many HTTP clients is infinite, which is how one hung dependency eventually consumes every thread in the caller. Set timeouts from the dependency's measured latency (a little above its p99.9), not from a round number. Worked example: an API with a 1,000 ms budget calls auth (p99 20 ms), then the order service (p99 150 ms), which calls the database (p99 40 ms). Timeouts of 50 ms for auth and 300 ms for the order service leave room for one retry within the budget. Pass the remaining deadline down the chain (gRPC does this natively), so a downstream service does not start work the caller has already given up on.

Retries with exponential backoff and jitter

Retry only what is safe (idempotent requests, or requests with an idempotency key) and only on errors that might succeed next time (timeouts, 429, 502, 503, 504 — never 400 or 422). Wait longer after each attempt, and randomise the wait. The randomisation is the part candidates forget, and it matters most: when a dependency hiccups, all clients fail at the same instant, and deterministic backoff makes them all retry at the same instants too.

import random
from collections import Counter

random.seed(42)
CLIENTS, BASE, CAP, TRIES = 1_000, 1.0, 20.0, 5     # seconds

def schedule(strategy):
    """Times (s) at which each client retries after all failing together at t=0."""
    times = []
    for _ in range(CLIENTS):
        t = 0.0
        for attempt in range(TRIES):
            ceiling = min(CAP, BASE * 2 ** attempt)   # 1, 2, 4, 8, 16 (capped at 20)
            if strategy == "fixed 1s":
                wait = 1.0
            elif strategy == "exponential":
                wait = ceiling
            elif strategy == "exp + full jitter":
                wait = random.uniform(0, ceiling)
            else:                                     # equal jitter
                wait = ceiling / 2 + random.uniform(0, ceiling / 2)
            t += wait
            times.append(t)
    return times

print(f"{CLIENTS} clients fail at once, then retry {TRIES} times each\n")
print(f"{'strategy':20} {'peak retries in any 100 ms':>28} {'last retry at':>14}")
for s in ("fixed 1s", "exponential", "exp + full jitter", "exp + equal jitter"):
    times = schedule(s)
    peak = max(Counter(int(t * 10) for t in times).values())
    print(f"{s:20} {peak:>28,} {max(times):>13.1f}s")
Output
1000 clients fail at once, then retry 5 times each

strategy               peak retries in any 100 ms  last retry at
fixed 1s                                    1,000           5.0s
exponential                                 1,000          31.0s
exp + full jitter                             154          28.0s
exp + equal jitter                            211          29.6s

Plain exponential backoff spreads retries over time but not within each retry wave: all 1,000 clients still arrive in the same 100 ms, again and again, which can keep a recovering service down. Full jitter (sleep = random(0, min(cap, base * 2^attempt))) cuts the peak by more than six times. Two more rules: retry in one layer only — three layers each retrying three times turns one failure into 27 calls — and use a retry budget (retries may add at most, say, 10% on top of normal traffic) so retries cannot become the overload.

Circuit breakers

When a dependency is clearly down, retrying it with a 2-second timeout on every request wastes 2 seconds of every request and a thread each. A circuit breaker watches the failure rate and, past a threshold, opens: calls fail immediately, without touching the network, and the caller uses a fallback. After a cool-down it goes half-open and lets a few trial requests through; if they succeed it closes, if not it opens again.

flowchart LR C[CLOSED
calls pass through
count failures] -->|5 consecutive failures
or half of the last 20 fail| O[OPEN
fail fast
use fallback] O -->|cool-down of 10 s elapsed| H[HALF_OPEN
allow trial calls] H -->|2 trials succeed| C H -->|any trial fails| O

Closed passes calls and counts failures. Open fails fast for a cool-down. Half-open lets trial calls decide whether the dependency has recovered.

class CircuitOpen(Exception):
    pass

class CircuitBreaker:
    """CLOSED -> (N consecutive failures) -> OPEN -> (cool-down) -> HALF_OPEN
       HALF_OPEN -> (trial succeeds k times) -> CLOSED, or (trial fails) -> OPEN"""
    def __init__(self, clock, failure_threshold=5, cooldown_s=10, half_open_successes=2):
        self.clock, self.threshold = clock, failure_threshold
        self.cooldown, self.needed = cooldown_s, half_open_successes
        self.state, self.failures, self.successes, self.opened_at = "CLOSED", 0, 0, None
        self.log = []

    def _go(self, state):
        self.log.append(f"t={self.clock():<3} {self.state:>9} -> {state}")
        self.state = state

    def call(self, fn):
        if self.state == "OPEN":
            if self.clock() - self.opened_at < self.cooldown:
                raise CircuitOpen("fail fast")          # no network call at all
            self._go("HALF_OPEN")                        # let trial requests through
            self.successes = 0
        try:
            result = fn()
        except Exception:
            self.failures += 1
            if self.state == "HALF_OPEN" or self.failures >= self.threshold:
                self._go("OPEN")
                self.opened_at, self.failures = self.clock(), 0
            raise
        if self.state == "HALF_OPEN":
            self.successes += 1
            if self.successes >= self.needed:
                self._go("CLOSED")
        self.failures = 0
        return result

# --- simulation: the recommendations service is down from t=3 s to t=24 s ---
now = 0
wasted_calls = 0                                    # calls that hit the dead service
def recommendations():
    global wasted_calls
    if 3 <= now < 24:
        wasted_calls += 1
        raise TimeoutError("timed out after 2 s")
    return ["item-1", "item-2"]

cb = CircuitBreaker(clock=lambda: now)
served = {"live": 0, "fallback": 0}
for now in range(40):                               # one request per second
    try:
        cb.call(recommendations)
        served["live"] += 1
    except (CircuitOpen, TimeoutError):
        served["fallback"] += 1                     # e.g. cached bestsellers

print("\n".join(cb.log))
print(f"\n40 requests: {served}")
print(f"calls that waited 2 s on the dead service: {wasted_calls} with the breaker, "
      f"21 without it")
Output
t=7      CLOSED -> OPEN
t=17       OPEN -> HALF_OPEN
t=17  HALF_OPEN -> OPEN
t=27       OPEN -> HALF_OPEN
t=28  HALF_OPEN -> CLOSED

40 requests: {'live': 16, 'fallback': 24}
calls that waited 2 s on the dead service: 6 with the breaker, 21 without it

The dependency was down for 21 seconds; only 6 requests paid the full timeout, and the other 15 got an instant fallback. The price is visible too: the dependency recovered at t=24, but three more requests got the fallback until the cool-down ended at t=27 — cool-down length trades recovery speed against hammering a service that is still sick. Production breakers (Resilience4j, Polly, Envoy outlier detection, Hystrix historically) use a failure rate over a sliding window with a minimum number of calls, count slow calls as failures, and keep one breaker per dependency — and ideally per endpoint, so one broken endpoint does not cut off a healthy one.

Bulkheads

Named after the watertight compartments of a ship: partition resources so that a flood in one compartment does not sink the rest. Give each dependency its own connection pool, thread pool or concurrency limit. Worked example: a product page service has 200 worker threads and calls reviews, pricing and inventory. The reviews service starts taking 10 seconds per call; without bulkheads all 200 threads are soon waiting on reviews and pricing requests cannot be served either. With a 30-thread limit on reviews, reviews calls beyond 30 are rejected immediately (fall back to "reviews unavailable"), and the other 170 threads keep serving the page. The same idea applies at larger scale: separate clusters for internal batch jobs and user traffic, and per-tenant capacity pools so one customer's surge cannot starve the others.

Load shedding

When a service is past its capacity, trying to serve everything means serving everything slowly, and then timing out on everything — goodput collapses toward zero. Load shedding deliberately rejects some requests early and cheaply (a fast 503 with Retry-After) so the ones it accepts complete within their deadline. Shed by priority: in the food delivery capstone the order is place-order, then tracking, then browse, then recommendations, then analytics. Signals to shed on: in-flight request count or queue wait time above a limit (better than CPU), or adaptive concurrency limits that back off as latency rises. Drop requests whose deadline has already passed — the client has given up. Under overload, serving the newest requests first (LIFO) serves users still waiting rather than ones who already left.

Graceful degradation

Decide in advance what each feature becomes when its dependency fails, so an outage turns into a slightly worse page rather than an error page.

Dependency downDegraded behaviour
RecommendationsShow cached bestsellers for the category
Personalised feed rankingChronological or popularity order from cache
Search clusterPrefix match on names from the primary database, fewer filters
Reviews serviceHide the reviews section; keep the product and "Buy" button
Payment provider ARoute to provider B; or accept the order and capture payment asynchronously
Email providerQueue the messages; the send happens after recovery
Feature-flag serviceUse the last known flags cached locally, or safe defaults

Make degradations switchable with feature flags ("kill switches"), and test them — a fallback path that has never run in production is a second outage waiting to happen. Chaos experiments and game days exist to exercise exactly these paths.

Step 11 — SLIs, SLOs and error budgets

  • SLI (service level indicator) — a measured ratio of good events to valid events, as close to the user as you can measure it. For example, "the proportion of checkout requests that returned a non-5xx response in under 800 ms", measured at the load balancer.
  • SLO (service level objective) — the target for an SLI over a window: 99.9% over a rolling 30 days.
  • SLA (service level agreement) — a contract with consequences (credits, refunds) if missed. Set it looser than the SLO, so you breach the internal target long before the contractual one.
  • Error budget — 100% minus the SLO. It is the amount of unreliability you are allowed to spend on deploys, experiments and incidents. When it is exhausted, reliability work takes priority over launches; when there is plenty left, move faster.
MIN_30D = 30 * 24 * 60

print("SLO       error budget / 30 days    failed requests allowed at 2,000 req/s")
for slo in (0.99, 0.999, 0.9995, 0.9999):
    budget = 1 - slo
    reqs = 2_000 * 86_400 * 30 * budget
    print(f"{slo:<8.2%}  {budget * MIN_30D:8.1f} min of full outage   {reqs:>14,.0f}")

# Burn rate = observed error rate / budgeted error rate.
# Burn rate 1 spends exactly the whole budget in 30 days.
slo = 0.999
print(f"\nSLO {slo:.1%}: time until the 30-day budget is gone")
for err in (0.001, 0.002, 0.0144, 0.05, 0.5):
    burn = err / (1 - slo)
    print(f"  error rate {err:6.2%} -> burn rate {burn:5.1f}x -> budget gone in "
          f"{30 * 24 / burn:6.1f} h")

# A common paging rule: burn rate > 14.4 over the last 1 h (AND over the last 5 min)
# means 2% of the monthly budget went in one hour -- wake someone up.
print(f"\n14.4x for 1 hour consumes {14.4 / (30 * 24):.0%} of the monthly budget")
Output
SLO       error budget / 30 days    failed requests allowed at 2,000 req/s
99.00%       432.0 min of full outage       51,840,000
99.90%        43.2 min of full outage        5,184,000
99.95%        21.6 min of full outage        2,592,000
99.99%         4.3 min of full outage          518,400

SLO 99.9%: time until the 30-day budget is gone
  error rate  0.10% -> burn rate   1.0x -> budget gone in  720.0 h
  error rate  0.20% -> burn rate   2.0x -> budget gone in  360.0 h
  error rate  1.44% -> burn rate  14.4x -> budget gone in   50.0 h
  error rate  5.00% -> burn rate  50.0x -> budget gone in   14.4 h
  error rate 50.00% -> burn rate 500.0x -> budget gone in    1.4 h

14.4x for 1 hour consumes 2% of the monthly budget

Each additional nine cuts the budget tenfold: 99.99% leaves four minutes a month, less than a single slow rollback, which means automated rollbacks, multi-region failover and change freezes. Do not promise more nines than your dependencies have — a service built on a 99.95% database cannot credibly offer 99.99% without redundancy around it. And alert on burn rate, not on individual errors: page when the budget is burning at 14.4x over the last hour (and still over the last five minutes, so the page stops once the problem does), open a ticket for a slow 1x–2x burn over days. This pages on what users feel and ignores CPU spikes that users never notice.

Choosing SLIs for a system in an interview: availability and latency for every user-facing request path; freshness for pipelines ("99% of events visible in search within 60 s"); durability for storage; and a business SLI or two (orders completed per checkout started) that catches failures every technical dashboard shows as green.

The trade-off summary — how to talk about communication and reliability

  • Sync for what the user must wait for, async for everything else. Every synchronous dependency multiplies into your availability and tail latency.
  • Queue for jobs, log for events. Per-message retries and DLQs versus retention, replay and many independent consumers.
  • Ordering is per key. Choose the partition key by what must stay ordered, and leave headroom in the partition count.
  • Assume at-least-once and make consumers idempotent with dedupe tables, versioned conditional writes and downstream idempotency keys.
  • Never dual-write. Transactional outbox on the producer, inbox or dedupe on the consumer.
  • Bound everything: queue sizes, timeouts, retries, concurrency per dependency. Unbounded things are where outages hide.
  • Retries need backoff, jitter, a budget and a single owning layer; circuit breakers and bulkheads stop a sick dependency from spreading; load shedding and degradation decide who gets served when you cannot serve everyone.
  • Pick the API style per client: REST for public and cacheable, gRPC between services, GraphQL for aggregating front-ends, SSE for one-way push, WebSocket for two-way realtime, long polling as the fallback.
  • Define reliability as SLOs with error budgets, and alert on burn rate.
// Write your solution here
// Write your solution here
// Write your solution here
// Write your solution here

Finished reading? Mark this lesson complete to track your progress.

Up next · Lesson 6Low-Level Design: Classes, APIs and ConcurrencyTurn a feature into entities, interfaces, a state machine and thread-safe code.