System Design · Lesson 12 of 18

Chat System (WhatsApp / Messenger)

WebSockets, presence, message ordering, delivery receipts and group fan-out.

  • Advanced
  • 65 min read
  • 5 objectives

Before this lessonLesson 11: News Feed (Twitter / Instagram)

What you will learn

  • Pick a connection protocol and design the wire frames and acks
  • Guarantee per-conversation order and exactly-once display
  • Model storage, partitioning and multi-device sync
  • Scale groups, channels, receipts, presence and media
  • Reason about E2E encryption, deploys and failure modes

Your Progress

0 of 18 lessons 0%

  • Lessons0 / 18
  • Completed0
  • Est. time left~ 16 hours

Progress is saved in this browser. No account needed.

Tip: pressing Next marks this lesson complete automatically.

Design a messaging system like WhatsApp, Messenger, Signal or Telegram: one-to-one and group chat, delivered and read receipts, online presence, history on every device, and the promise users actually care about — a message the app said it sent is never lost, never shown twice, and never shown out of order.

Chat looks like a CRUD app with a socket bolted on. It is not. It is three unusual problems stacked together:

  • A stateful connection fleet. Hundreds of millions of long-lived sockets, each pinned to one machine. Almost every other system in this course is stateless at the edge; this one cannot be, and that changes how you route, deploy and fail over.
  • An ordered, exactly-once-displayed log per conversation. Networks drop, duplicate and reorder. Phones go into tunnels mid-send. The design must turn an unreliable transport into a reliable, ordered log that every device of every member agrees on.
  • A fan-out engine with very different shapes. A 1:1 message goes to one person, a family group to twelve, a broadcast channel to a million. One write path cannot serve all three efficiently.

The sentence that frames the interview: "The source of truth is a per-conversation append-only log with server-assigned sequence numbers. Sockets are only a fast notification path on top of it; anything the socket misses, the client repairs by syncing from its last sequence number." Everything below hangs off that idea.

Step 1 — Clarify the requirements

Functional, in scope for v1

  • 1:1 chat with text, emoji and replies; messages persist and are available on every device the user owns.
  • Group chat up to 1,024 members; admins add and remove members.
  • Receipts: sent (one tick), delivered (two ticks), read (blue ticks), per message, with group read receipts aggregated.
  • Presence: online / last seen, and typing indicators.
  • Offline delivery: messages to an offline user wait on the server and trigger a push notification.
  • Media: images, voice notes, video and documents up to 100 MB.
  • Multi-device: one phone plus up to four linked devices (desktop, web, tablet), each fully in sync.

Out of scope (say it)

Voice and video calls (a separate real-time media system: WebRTC, TURN relays, jitter buffers), stories, payments, bots and business APIs, message search on the server, spam and abuse ML, and contact discovery. Mention end-to-end encryption as a design constraint you will sketch, because it changes what the server is allowed to know.

Non-functional

  • Scale: 500M daily active users, 40 messages per user per day, 25% of DAU connected at the evening peak.
  • Latency: sender-to-recipient p99 under 300 ms when both are online in the same region, under 1 s cross-region. Server ack to sender p99 under 150 ms.
  • Durability: once the sender sees one tick, the message survives any single machine, rack or availability-zone failure. This is the non-negotiable requirement.
  • Ordering: every member sees one conversation in the same order, on every device. There is no global order across conversations and nobody needs one.
  • Exactly-once display: delivery is at-least-once under the hood; duplicates are removed by idempotency keys so the user never sees the same message twice.
  • Availability: 99.99% for send and receive. When forced to choose for a single conversation, prefer consistency of the log (reject the send and let the client retry) over accepting writes that could fork the order.

Step 2 — Back-of-the-envelope estimation

DAU               = 500_000_000
MSGS_PER_USER_DAY = 40
SECONDS_PER_DAY   = 86_400
PEAK_FACTOR       = 3            # evening peak vs daily average

# --- Message rate ---
msgs_day  = DAU * MSGS_PER_USER_DAY
msgs_avg  = msgs_day / SECONDS_PER_DAY
msgs_peak = msgs_avg * PEAK_FACTOR
print(f"messages/day: {msgs_day/1e9:.0f} B   avg {msgs_avg:,.0f}/s   peak {msgs_peak:,.0f}/s")

# --- Deliveries, not sends, are the fan-out load (groups multiply) ---
group_share, avg_group_size = 0.30, 20
deliveries = msgs_avg * ((1 - group_share) * 1 + group_share * (avg_group_size - 1))
print(f"deliveries: {deliveries:,.0f}/s avg  ({deliveries/msgs_avg:.1f}x sends)")

# --- Receipts: each delivery produces a 'delivered' and usually a 'read' event ---
receipts = deliveries * 1.8
print(f"receipt events: {receipts:,.0f}/s avg")

# --- Storage: server keeps history (non-E2E variant) ---
body_bytes, meta_bytes, rf = 100, 60, 3
per_day_tb = msgs_day * (body_bytes + meta_bytes) / 1e12
print(f"storage: {per_day_tb:.1f} TB/day raw, {per_day_tb*365*5*rf/1e3:,.1f} PB for 5y x RF{rf}")

# --- Connections ---
concurrent_frac = 0.25           # share of DAU holding a socket at peak
sockets = DAU * concurrent_frac
per_box = 250_000                # tuned epoll box, ~10-20 KB kernel+app memory each
boxes = sockets / per_box
print(f"concurrent sockets: {sockets/1e6:.0f} M -> {boxes:,.0f} connection servers "
      f"(+50% headroom = {boxes*1.5:,.0f})")

# --- Heartbeats alone are a real load ---
hb_interval = 30
print(f"heartbeats: {sockets/hb_interval/1e6:.1f} M/s  (never write these to a database)")

# --- Bandwidth ---
avg_wire_bytes = 300             # framed, TLS, headers, JSON/protobuf envelope
print(f"egress: {deliveries*avg_wire_bytes*8/1e9:.1f} Gbit/s avg for text alone")
Output
messages/day: 20 B   avg 231,481/s   peak 694,444/s
deliveries: 1,481,481/s avg  (6.4x sends)
receipt events: 2,666,667/s avg
storage: 3.2 TB/day raw, 17.5 PB for 5y x RF3
concurrent sockets: 125 M -> 500 connection servers (+50% headroom = 750)
heartbeats: 4.2 M/s  (never write these to a database)
egress: 3.6 Gbit/s avg for text alone

What the numbers actually decide

  • Deliveries, not sends, are the load. 230k sends/s sounds like the headline, but groups multiply it to about 1.5M deliveries/s, and receipts add another 2.7M events/s on top. Receipts are the biggest event stream in the system — which is why Step 10 compresses them into watermarks.
  • 125M concurrent sockets forces a dedicated connection tier. A tuned Linux box with epoll, small buffers and a lean per-connection struct holds 200k–1M idle sockets; a conservative 50k per box means 2,500 machines, a tuned 250k means 500. Either way this is a separate, stateful fleet with its own deploy process (Step 17), not the API servers.
  • 4M heartbeats/s must never touch a database. They are absorbed in the connection server's memory, and only state changes (online to offline) are published.
  • 3 TB/day and ~17 PB over five years with replication means a horizontally partitioned store built for small appends and range scans: wide-column (Cassandra, ScyllaDB, HBase, Bigtable). A relay-only design stores only the undelivered backlog — a few TB total.
  • Bandwidth is modest for text (a few Gbit/s). Media dominates bytes by two orders of magnitude, which is why media never flows through the chat servers at all (Step 14).

Step 3 — Choosing the connection protocol

The receiver has to learn about a new message. There are two families: the client pulls ("anything for me?") or the server pushes over a connection the client keeps open.

Why pull loses

With plain polling every client asks every few seconds. To get 1 s latency, 125M clients poll once a second: 125M requests/s, over 99% of them returning nothing, each paying for TLS, headers, auth and a database or cache lookup. And the latency is still up to a full interval. Pull is the wrong model for a system whose defining requirement is "instant".

Long polling: push made out of pull

The client sends a request and the server holds it open until a message arrives or a timeout (typically 30–60 s) expires. When it returns, the client immediately opens the next one. Latency becomes near-instant and empty responses drop to one per timeout. The costs: one message per round trip (a burst of ten messages means ten request cycles or careful batching), a gap between responses where messages must be buffered server-side, re-sent headers on every cycle, and no way for the client to send on the same channel. It is an excellent fallback for hostile networks, and the default in older designs, but not the primary transport today.

The comparison

TransportDirectionLatencyOverhead per messageWorks through proxiesVerdict for chat
Short pollingClient pullUp to the poll intervalFull HTTP request, mostly emptyEverywhereNo — wasteful and slow
Long pollingServer push (simulated)Near-instant, gaps between pollsFull HTTP response per batchEverywhereFallback only
Server-Sent EventsServer to client onlyInstantA few bytes of framingMostly (it is plain HTTP)Good for feeds and notifications; sends still need separate POSTs
WebSocketFull duplexInstant2–14 bytes of framingUsually; some corporate proxies break itDefault for web and mobile
MQTT (over TCP or WebSocket)Full duplex, pub/subInstant2-byte fixed header, tiny keepaliveNeeds port 443 / WS wrappingExcellent for mobile: built-in QoS levels, small keepalives, session resume (Messenger used it)

Choose WebSocket (or MQTT over WebSocket) on port 443, with long polling as the fallback. Justify it: full duplex means sends, acks, receipts and typing share one connection; framing overhead is a few bytes instead of hundreds; and TLS is negotiated once per connection instead of once per request. On mobile, the protocol's keepalive cost matters as much as its throughput — every radio wake-up costs battery, so heartbeats are tuned to just under the carrier's NAT timeout (often 30–60 s) and the app relies on OS push notifications while backgrounded.

Step 4 — The wire protocol and API

Most candidates skip this and hand-wave "send over the socket". Writing out the frames is how you show you understand where messages get lost.

## Every frame on the socket is an envelope. Protobuf on the wire; JSON shown here.
{ "t": "<frame type>", "id": "<frame id, for acking this frame>", ...payload }

## Client -> server
HELLO      { device_id, auth_token, app_version,
             cursors: { "c:alice:bob": 4182, "g:9x2": 77, ... } }   # last contiguous seq
SEND       { client_msg_id: "a-7f1", conv_id, kind: "text"|"media"|"reaction",
             body | ciphertext, media_ref?, reply_to_seq? }
DELIV_ACK  { conv_id, up_to_seq }      # "this device has stored everything <= seq"
READ       { conv_id, up_to_seq }      # "the user has seen everything <= seq"
TYPING     { conv_id, state: "on"|"off" }        # ephemeral, never stored
SUB_PRES   { user_ids: [...] }         # presence for the chats on screen right now
PING       { }                          # app-level heartbeat every 30 s

## Server -> client
WELCOME    { session_id, server_time, resume: true|false }
SERVER_ACK { client_msg_id, conv_id, seq, server_ts }   # persisted: show ONE tick
NACK       { client_msg_id, code: "NOT_A_MEMBER"|"TOO_LARGE"|"RATE_LIMITED",
             retry_after_ms? }                              # final: do not retry blindly
MSG        { conv_id, seq, sender, sender_device, client_msg_id, server_ts, body }
RECEIPT    { conv_id, user_id, kind: "delivered"|"read", up_to_seq }
PRESENCE   { user_id, state: "online"|"offline", last_seen? }
SYNC_HINT  { conv_id, latest_seq }      # "you are behind, pull via the sync API"
GOAWAY     { reconnect_after_ms, reason: "deploy"|"rebalance" }   # graceful drain
PONG       { }

## Ack semantics (the part to say out loud)
SERVER_ACK  = durable on the server (quorum write succeeded). Before this, the client
              keeps the message in a local outbox and retries with the SAME client_msg_id.
DELIV_ACK   = durable on the receiving DEVICE (its local DB). Cumulative, per conversation.
READ        = human-level event. Cumulative. Also implies delivered.
None of these are TCP acks. TCP only tells you bytes reached a kernel buffer.

## HTTPS side (stateless API fleet, not the socket)
GET  /v1/sync?conv_id=c:alice:bob&after_seq=4182&limit=200
     -> 200 { messages: [...], next_after_seq, has_more }
GET  /v1/conversations?updated_after=<ts>&limit=50       # the chat list on a cold start
     -> 200 { conversations: [{ conv_id, latest_seq, unread, last_msg_preview }] }
GET  /v1/history?conv_id=...&before_seq=4001&limit=50     # scrolling up
POST /v1/media/uploads { mime, size, sha256 }
     -> 201 { media_id, upload_url (pre-signed, 15 min) } | 413 TOO_LARGE
POST /v1/groups { name, member_ids[] } -> 201 { conv_id } | 422 TOO_MANY_MEMBERS
POST /v1/devices { device_id, platform, push_token, identity_key, prekeys[] } -> 201

Protocol decisions worth defending

  • Three different acks, three different meanings. SERVER_ACK means durable on the server; DELIV_ACK means durable on a device; READ means a human saw it. Collapsing them ("the server acks") is how designs end up showing two ticks for a message sitting in a dead socket.
  • Acks are cumulative per conversation (up_to_seq), like TCP's cumulative ACK. One frame acknowledges a burst of 40 messages; lost ack frames are healed by the next one.
  • The client generates client_msg_id (a UUID or device-scoped counter) before the first attempt and reuses it on every retry. The server turns it into exactly-once.
  • HELLO carries cursors. The reconnecting device tells the server where it is; the server replies with SYNC_HINTs for conversations that moved, and the client pulls the gap over HTTPS. Bulk catch-up does not clog the socket, and it can be served by the stateless fleet and caches.
  • NACK is final and typed. "You are not a member of this group" must stop the retry loop; "rate limited" must carry retry_after_ms.
  • GOAWAY exists from day one so the server can drain connections politely during deploys instead of cutting them.

Step 5 — Data model

-- =========================================================
-- MESSAGES  (wide-column: Cassandra / ScyllaDB / HBase / Bigtable)
-- =========================================================
messages (
  conv_id        TEXT,         -- "c:<lo_uid>:<hi_uid>" for 1:1 (deterministic), "g:<id>" for groups
  bucket         INT,          -- seq / 50_000: caps a partition at ~50k rows, ~10 MB
  seq            BIGINT,       -- dense, per conversation, assigned by the conversation owner
  sender_id      BIGINT,
  sender_device  INT,
  client_msg_id  TEXT,         -- the sender's idempotency key
  kind           TINYINT,      -- text, media, reaction, edit, delete (tombstone), system
  body           BLOB,         -- plaintext, or ciphertext when E2E (server cannot read it)
  media_ref      TEXT,         -- object-store key; bytes never live here
  server_ts      TIMESTAMP,    -- for display only, NEVER for ordering
  PRIMARY KEY ((conv_id, bucket), seq)
) WITH CLUSTERING ORDER BY (seq DESC);  -- "latest 50" is the first 50 rows of one partition

-- Dedupe: (conv, sender, client_msg_id) -> seq. Checked before a seq is allocated.
msg_dedupe (
  conv_id TEXT, sender_id BIGINT, client_msg_id TEXT, seq BIGINT,
  PRIMARY KEY ((conv_id), sender_id, client_msg_id)
) WITH default_time_to_live = 604800;   -- 7 days: longer than any client retries

conversations (
  conv_id     TEXT PRIMARY KEY,
  type        TINYINT,          -- direct | group | channel
  latest_seq  BIGINT,           -- the conversation's counter (see Step 9)
  member_count INT,
  created_at  TIMESTAMP
);

conv_members (                  -- who is in a group; read on every group send
  conv_id  TEXT, user_id BIGINT, role TINYINT, joined_seq BIGINT,
  PRIMARY KEY ((conv_id), user_id)
);

-- The per-user index: "which conversations do I have, newest first".
-- Partitioned by USER id: one user's chat list is one partition read.
user_conversations (
  user_id      BIGINT,
  conv_id      TEXT,
  latest_seq   BIGINT,          -- bumped on fan-out; unread = latest_seq - read_seq
  read_seq     BIGINT,          -- the user's read watermark (also the read receipt)
  delivered_seq BIGINT,
  muted_until  TIMESTAMP,
  PRIMARY KEY ((user_id), conv_id)
);

devices (                       -- multi-device: each device syncs independently
  user_id BIGINT, device_id INT, platform TEXT, push_token TEXT,
  identity_key BLOB, last_seen TIMESTAMP,
  PRIMARY KEY ((user_id), device_id)
);

-- =========================================================
-- EPHEMERAL  (Redis — rebuilt from reconnects if lost; never the source of truth)
-- =========================================================
HSET  session:u:42        dev:1 "cs-0317:sess-9a1" dev:2 "cs-0122:sess-44f"
EXPIRE session:u:42       90                   -- refreshed by every heartbeat
SETEX presence:u:42       60 "online"          -- absent key == offline
SET   last_seen:u:42      1758612000           -- written on disconnect, coarse
SADD  pres_subs:u:42      "cs-0317" "cs-0801"  -- which connection servers care about 42
LPUSH recent:c:alice:bob  <msg>  ; LTRIM 0 49  -- hot cache: last 50 messages

Why a wide-column store, and why not the alternatives

  • The access pattern is "append small rows" and "read a contiguous range of one conversation". A log-structured merge tree (Cassandra, ScyllaDB, HBase) turns every write into a sequential memtable append, and stores a partition's rows physically sorted by seq, so "the last 50 messages" is one seek and one sequential read. See Lesson 10: Key-Value Store for how LSM trees and quorum replication work.
  • Why not MySQL/Postgres? A relational database carries you a long way at small scale — Discord started on MongoDB and later moved to Cassandra and then ScyllaDB for exactly this workload — but 700k inserts/s at peak and petabytes of history means hundreds of manually sharded instances, B-tree page splits on hot tables and painful resharding. The relational features you pay for (joins, multi-row transactions) are ones the message path does not need.
  • Why not a document store keyed by message id? Because the primary key decides the partition. Partitioning by message_id scatters one conversation across every shard, so rendering one chat becomes a scatter-gather over the whole cluster. Never do this.
  • Partition by conversation or by user? Both, for different tables. messages is partitioned by conv_id, so a group message is stored once, not once per member. user_conversations is partitioned by user_id, so a user's chat list is one partition read. A pure "partition everything by user id" design (store each message in every recipient's mailbox) is simple for 1:1 and relay-only systems, but writes N copies of every group message.
  • The bucket in the partition key stops a ten-year-old family group from growing a single partition to gigabytes. Because the bucket is derived from seq, the client knows exactly which bucket to read next when scrolling up.
  • server_ts is for display only. Ordering is seq. Clocks on different servers disagree by milliseconds to seconds; two messages in the same second would render in different orders on different devices.

Step 6 — High-level design

flowchart TB A[Phone / desktop / web] -->|WebSocket, TLS 443| LB[L4 load balancer] A -->|HTTPS: sync, history, media| API[Stateless API fleet] A -->|PUT bytes via pre-signed URL| OBJ[(Object storage + CDN)] LB --> CS[Connection servers
hold sockets, heartbeats] CS <--> SR[(Session registry
Redis: user to server)] CS --> MS[Message service
dedupe, assign seq, persist] API --> MS MS --> DB[(Wide-column store
messages, members, inbox)] MS --> RC[(Recent-message cache)] MS --> K[(Log / Kafka
partitioned by conv_id)] K --> FO[Fan-out workers] FO --> SR FO -->|deliver to the right box| CS FO --> PN[Push notification service] PN --> APNS[APNs / FCM] CS <--> PR[Presence service
Redis, pub/sub] API --> DB API --> RC

Two edges into the system: long-lived sockets for real-time frames, stateless HTTPS for bulk sync, history and media. The message service is the only writer of the log. Fan-out and push hang off the log, so a slow push provider never slows the sender's ack.

What each component owns

  • L4 load balancer — spreads new TCP connections across connection servers (least-connections). It does not need to know users; the session registry does.
  • Connection servers — terminate TLS and WebSocket, authenticate the HELLO, hold sockets and heartbeats in memory, register user/device → server in the session registry, forward client frames to the message service, and write server frames to sockets. Deliberately thin: no business logic, so they rarely need deploying.
  • Session registry — answers "which connection server holds user 42's devices right now?" A Redis hash per user with a TTL refreshed by heartbeats. This is the answer to "how do we know which server holds the connection".
  • Message service — the heart. Checks membership and rate limits, dedupes by client_msg_id, assigns the next seq, writes to the store at quorum, appends to the log, returns SERVER_ACK.
  • Log (Kafka or similar) — partitioned by conv_id, which preserves per-conversation order for downstream consumers and decouples fan-out from the write.
  • Fan-out workers — expand a conversation into member devices, look them up in the registry, deliver to online ones, bump user_conversations.latest_seq and unread counts, and hand offline ones to push.
  • Push notification service — batches, collapses and rate-limits calls to APNs and FCM; manages invalid tokens. See Lesson 17: Notification System.
  • Presence service — online state and subscriptions (Step 13). Entirely in memory; losing it costs a few seconds of wrong green dots.
  • API fleet — sync, history pagination, conversation list, group admin, media upload URLs. Stateless and cacheable.

Step 7 — Walk one 1:1 message, recipient online

sequenceDiagram participant A as Alice phone participant CA as Conn server A participant M as Message service participant D as Store, quorum participant K as Log participant F as Fan-out worker participant R as Session registry participant CB as Conn server B participant B as Bob phone A->>A: write to local outbox, client_msg_id=a-7f1, show clock A->>CA: SEND {a-7f1, conv c:alice:bob, body} CA->>M: forward (user=alice, device=1) M->>D: dedupe lookup (alice, a-7f1) - miss M->>M: seq = next for conv (4183) M->>D: INSERT message + dedupe row, QUORUM D-->>M: ok M->>K: append {conv, seq 4183} M-->>CA: SERVER_ACK {a-7f1, seq 4183} CA-->>A: SERVER_ACK - one tick, drop from outbox K->>F: consume F->>R: where is bob? - cs-0317 dev 1 F->>CB: deliver MSG seq 4183 CB->>B: MSG seq 4183 B->>B: store locally, render, cursor = 4183 B->>CB: DELIV_ACK {conv, up_to 4183} CB->>M: receipt M->>D: user_conversations(bob).delivered_seq = 4183 M->>K: RECEIPT event K->>F: consume F->>CA: RECEIPT delivered up_to 4183 CA->>A: two ticks Note over B: Bob opens the chat B->>CB: READ {conv, up_to 4183} CB->>A: (same path) blue ticks

One tick means "durable on the server". Two ticks means "durable on Bob's device" — proven by Bob's device, not assumed from an open socket.

The order of operations, and why

  1. Local outbox first, on the sender's device. The message is written to the phone's SQLite before any network I/O. If the app is killed, it resends on relaunch with the same client_msg_id. The clock icon means "only my phone has this".
  2. Dedupe before allocating a sequence number. If the dedupe row exists, this is a retry whose ack was lost: return the original SERVER_ACK with the original seq. Allocating first would burn a seq and leave a hole or a duplicate.
  3. Persist at quorum before acking. A write to 2 of 3 replicas in different zones. Only then is the one-tick promise true. Acking before persisting — or, the classic mistake, pushing to Bob first and persisting in the background — means a crash loses a message the sender was told was sent.
  4. Append to the log, then ack the sender. Some designs ack right after the store write and let a change-data-capture stream feed fan-out; either works as long as fan-out is driven from durable state, never from the in-memory request.
  5. Fan-out is asynchronous. Alice's ack latency is store latency (~10–30 ms), not "Bob's phone on 2G" latency.
  6. Delivery is at-least-once; display is exactly-once. If the fan-out worker crashes after writing to Bob's socket but before committing its log offset, it will deliver seq 4183 again. Bob's client drops it because 4183 ≤ its cursor.

Step 8 — Recipient offline, push, and reconnect sync

sequenceDiagram participant A as Alice phone participant M as Message service participant D as Store participant F as Fan-out worker participant R as Session registry participant P as Push service participant O as APNs / FCM participant B as Bob phone participant API as API fleet A->>M: SEND seq assigned 4184, 4185, 4186 M->>D: persist all three M-->>A: SERVER_ACK x3 - one tick F->>R: where is bob? R-->>F: no session (TTL expired) F->>D: user_conversations(bob).latest_seq = 4186, unread += 3 F->>P: notify bob with conv and preview P->>P: collapse per conv, rate-limit per device P->>O: one push with collapse-key conv O->>B: banner "Alice: 3 new messages" Note over B: Bob taps the notification B->>M: HELLO {cursors: c:alice:bob = 4183} M-->>B: WELCOME + SYNC_HINT {c:alice:bob, latest 4186} B->>API: GET /v1/sync?conv=c:alice:bob&after_seq=4183 API->>D: range scan seq 4184..4186 API-->>B: 3 messages B->>M: DELIV_ACK up_to 4186 M->>A: RECEIPT delivered up_to 4186 - two ticks

Nothing is "queued for Bob" in a special place. His messages are already in the conversation log; the only per-user state is a cursor gap. Reconnect is a range scan from his last sequence number.

What to point out

  • The server does not tell Alice "delivery failed". Older designs returned a failure when the recipient was offline and asked the sender to retry. With a durable log, the message is simply sent-but-not-delivered: one tick, and two ticks later when Bob reconnects.
  • Push is a hint, not a transport. APNs and FCM may delay, coalesce or drop notifications, and a muted chat sends none. The message itself always arrives through sync. With E2E encryption the push payload is either encrypted or contentless ("New message") and the app wakes up to fetch.
  • The sync API is the same code path as history. "Give me seq > cursor for this conversation" serves reconnects, new devices, missed pushes and gap repair. One path, heavily tested, rather than a special offline queue that drifts out of step.
  • Cold start with 400 conversations: the client first calls GET /v1/conversations?updated_after=, which reads one user_conversations partition and returns only conversations whose latest_seq moved, then syncs those — newest first, lazily for the rest.

Step 9 — Ordering and dedupe, in code

Why timestamps cannot order a conversation

Alice and Bob both hit send at 20:14:05. Alice's message reaches a server in Frankfurt, Bob's reaches one in Dublin. Each server stamps its own clock, and the two clocks are 40 ms apart. Meanwhile each message is delivered to the other party's phone first, so Alice's screen shows her message then Bob's, and Bob's screen shows the reverse. Nobody is wrong locally, but the two views of the conversation have forked, and a third device syncing later will see a third order. Timestamps from the client are worse: phone clocks are wrong by minutes, and a malicious client can backdate.

The fix is to make one component the sequencer for each conversation. It hands out seq = 1, 2, 3... as messages are accepted. Every device renders by seq. Everyone sees the same order — possibly not the order in which people pressed send, but one consistent order, which is what "consistent history on every device" actually requires.

Where the counter lives

  • Single owner per conversation (preferred). Route every send for conv_id to the message-service instance that owns it (consistent hashing, or the Kafka partition leader). It keeps the counter in memory and persists latest_seq with each write. On owner failover, the new owner reads latest_seq from the store — or fences the old owner with an epoch number — before handing out more.
  • Conditional write in the store. UPDATE conversations SET latest_seq = n+1 IF latest_seq = n (a Cassandra lightweight transaction, or a Redis INCR per conversation). Simple, but a Paxos round per message is expensive, and a hot group serialises on one row.
  • What not to use: a global Snowflake ID as the ordering key. It is time-ordered, but two IDs from different generators in the same millisecond carry no causal meaning; you still need per-conversation density to detect gaps. Snowflake is fine as a globally unique message id alongside seq (Lesson 7: Unique ID Generation).

Dense sequence numbers buy a second property for free: gap detection. If a device holds 1, 2, 4, 5, it knows 3 is missing and asks for it. With sparse IDs it could never tell a gap from silence.

import random
from dataclasses import dataclass

@dataclass(frozen=True)
class Msg:
    conv_id: str
    seq: int              # assigned by the server, dense per conversation
    client_msg_id: str    # assigned by the sender's device, the idempotency key
    sender: str
    body: str

class ConversationLog:
    """Server side: the single writer for one conversation (its partition owner)."""
    def __init__(self, conv_id):
        self.conv_id, self.next_seq = conv_id, 1
        self.rows: list[Msg] = []
        self.by_client_id: dict[tuple[str, str], Msg] = {}   # (sender, client_msg_id)

    def append(self, sender, client_msg_id, body) -> tuple[Msg, bool]:
        key = (sender, client_msg_id)
        if key in self.by_client_id:                  # a retry: same answer, no new row
            return self.by_client_id[key], False
        m = Msg(self.conv_id, self.next_seq, client_msg_id, sender, body)
        self.next_seq += 1
        self.rows.append(m)
        self.by_client_id[key] = m
        return m, True

    def since(self, after_seq):                       # the sync API: seq > cursor
        return [m for m in self.rows if m.seq > after_seq]

class ClientView:
    """Receiver side: render strictly in seq order, buffer gaps, dedupe replays."""
    def __init__(self):
        self.last_contiguous = 0
        self.pending: dict[int, Msg] = {}
        self.rendered: list[str] = []

    def receive(self, m: Msg) -> list[int]:
        if m.seq <= self.last_contiguous or m.seq in self.pending:
            return []                                  # duplicate delivery: ignore
        self.pending[m.seq] = m
        while self.last_contiguous + 1 in self.pending:
            nxt = self.pending.pop(self.last_contiguous + 1)
            self.rendered.append(f"{nxt.seq}:{nxt.sender}:{nxt.body}")
            self.last_contiguous += 1
        # A gap means something is missing: ask the server rather than wait forever.
        return sorted(self.pending)

log = ConversationLog("c:alice:bob")
# 1. Alice's phone sends, the ack is lost on a flaky network, the app retries.
m1, new1 = log.append("alice", "a-7f1", "are you coming?")
m1b, new2 = log.append("alice", "a-7f1", "are you coming?")
print("retry created a new row?", new2, "| same seq:", m1.seq == m1b.seq)
# 2. Bob and Alice type at the same moment: the log, not wall clocks, decides order.
log.append("bob",   "b-001", "leaving now")
log.append("alice", "a-7f2", "bring the charger")
log.append("bob",   "b-002", "ok")
log.append("alice", "a-7f3", "see you")
print("server order:", [(m.seq, m.sender) for m in log.rows])

# 3. Push delivery to Bob's second device arrives shuffled and with a duplicate.
random.seed(4)
wire = log.rows[:] + [log.rows[1]]
random.shuffle(wire)
wire = [m for m in wire if m.seq != 3]           # seq 3 is dropped by a dying socket
print("arrival order:", [m.seq for m in wire])

view = ClientView()
for m in wire:
    view.receive(m)
print("rendered so far:", view.rendered)
print("buffered behind the gap:", sorted(view.pending), "| cursor:", view.last_contiguous)

# 4. Gap detected -> sync from the last contiguous seq, idempotently.
for m in log.since(view.last_contiguous):
    view.receive(m)
print("after sync:", view.rendered)
Output
retry created a new row? False | same seq: True
server order: [(1, 'alice'), (2, 'bob'), (3, 'alice'), (4, 'bob'), (5, 'alice')]
arrival order: [4, 2, 5, 1, 2]
rendered so far: ['1:alice:are you coming?', '2:bob:leaving now']
buffered behind the gap: [4, 5] | cursor: 2
after sync: ['1:alice:are you coming?', '2:bob:leaving now', '3:alice:bring the charger', '4:bob:ok', '5:alice:see you']

The LLD points an interviewer is listening for

  • Two ids, two jobs. client_msg_id belongs to the sender and makes retries idempotent. seq belongs to the conversation and defines order. Mixing them up is the most common bug in whiteboard chat designs.
  • The dedupe key is scoped to (conv, sender, client_msg_id), so two devices generating the same short id cannot collide, and the table can expire after a week.
  • The client buffers, never guesses. Messages 4 and 5 wait behind the gap instead of rendering out of order; after a short timeout (about 2 s) the client repairs the gap by calling sync from its cursor.
  • Sync is idempotent. It replays 3, 4 and 5; the view ignores 4 and 5 as duplicates because they are already buffered. At-least-once plus dedupe equals exactly-once display.
  • The cursor is the only state that must survive a reinstall-free restart — one integer per conversation per device.

Step 10 — Receipts as a state machine

flowchart LR P["PENDING
clock: only on sender device"] -->|SERVER_ACK| S["SENT
one tick: durable on server"] P -->|NACK| X["FAILED
red icon, tap to retry"] P -->|timeout, retry same client_msg_id| P S -->|DELIV_ACK from recipient device| DL["DELIVERED
two ticks"] DL -->|READ from recipient| RD["READ
blue ticks"] S -->|READ arrives first| RD

States only move forward. A late or duplicate receipt can never turn blue ticks back to grey, and a read implies delivered even if the delivered event was lost.

Storing a receipt per message per recipient is the naive design: a 200-member group with 10,000 messages is two million rows, and every receipt is a write. Instead, receipts are watermarks: per member, "delivered up to seq N" and "read up to seq N". Because the log is ordered, a single integer summarises every message below it.

from enum import IntEnum

class Tick(IntEnum):          # ordered: a receipt can only move forward
    PENDING = 0               # clock icon: still in the sender's outbox
    SENT = 1                  # one tick: server persisted it (server_ack)
    DELIVERED = 2             # two ticks: a recipient device acked it
    READ = 3                  # blue ticks: a recipient opened the conversation

class Watermarks:
    """Per conversation member: the highest seq delivered / read.
    One row per (conv, member) replaces one row per (message, member)."""
    def __init__(self, members):
        self.delivered = {u: 0 for u in members}
        self.read = {u: 0 for u in members}

    def on_receipt(self, user, kind, up_to_seq):
        table = self.delivered if kind == "delivered" else self.read
        table[user] = max(table[user], up_to_seq)          # late/duplicate: no regress
        if kind == "read":                                  # read implies delivered
            self.delivered[user] = max(self.delivered[user], up_to_seq)

    def tick_for(self, seq, sender, persisted_up_to):
        others = [u for u in self.read if u != sender]
        if seq > persisted_up_to:
            return Tick.PENDING
        if all(self.read[u] >= seq for u in others):
            return Tick.READ
        if all(self.delivered[u] >= seq for u in others):
            return Tick.DELIVERED
        return Tick.SENT

g = Watermarks(["alice", "bob", "cara"])
persisted = 3                        # alice's messages 1..3 are on the server; 4 is not
events = [("bob", "delivered", 3), ("cara", "delivered", 2),
          ("bob", "read", 3), ("bob", "delivered", 1),        # stale event arrives late
          ("cara", "read", 1)]
for user, kind, seq in events:
    g.on_receipt(user, kind, seq)
    ticks = [g.tick_for(s, "alice", persisted).name for s in (1, 2, 3, 4)]
    print(f"{user:>4} {kind:<9} up to {seq}  ->  msgs 1-4: {ticks}")

# Storage comparison for one busy group
members, msgs = 200, 10_000
print(f"per-message receipts: {members*msgs:,} rows | watermarks: {members*2:,} values")
Output
 bob delivered up to 3  ->  msgs 1-4: ['SENT', 'SENT', 'SENT', 'PENDING']
cara delivered up to 2  ->  msgs 1-4: ['DELIVERED', 'DELIVERED', 'SENT', 'PENDING']
 bob read      up to 3  ->  msgs 1-4: ['DELIVERED', 'DELIVERED', 'SENT', 'PENDING']
 bob delivered up to 1  ->  msgs 1-4: ['DELIVERED', 'DELIVERED', 'SENT', 'PENDING']
cara read      up to 1  ->  msgs 1-4: ['READ', 'DELIVERED', 'SENT', 'PENDING']
per-message receipts: 2,000,000 rows | watermarks: 400 values
  • max() is the whole concurrency story. Watermarks form a monotonic merge (a tiny CRDT), so receipts can arrive late, duplicated or out of order from several devices and still converge.
  • Group ticks are a minimum over members. "Read by all" turns blue when the slowest member's watermark passes the message; the "Info" screen that lists who read it simply compares each member's watermark with seq.
  • Receipts are batched and throttled. A reader scrolling through 300 unread messages emits one READ up_to_seq, not 300 events. In groups above a size threshold (say 256), per-member receipts are not fanned out to every member at all — only to the sender, and only on request.
  • Privacy settings are a filter at fan-out: if Bob disabled read receipts, his read watermark still exists (it drives his own unread badge) but is not sent to others.

Step 11 — Multi-device sync

With one phone per user, "the user's socket" is unambiguous. With a phone, a laptop and a browser tab, every rule has to be restated per device.

  • Fan-out targets devices, not users. The session registry maps user → {device → server}. A message to Bob is delivered to each of his connected devices; each device acks and keeps its own cursor.
  • Your own messages are fanned out to your other devices. Alice sends from her phone; her laptop must show it too. Fan-out includes the sender's other devices, and the laptop dedupes by client_msg_id if it is the device that sent it.
  • Delivered means "at least one device"; read means "any device". Reading on the laptop must clear the unread badge on the phone, so the read watermark is per user, pushed to all of that user's devices as a RECEIPT about themselves.
  • A new device bootstraps from the log. With server-side history, linking a laptop is: authenticate, pull the conversation list, sync the last N messages of each, lazily fetch older ones on scroll. With relay-only E2E history (classic WhatsApp), the server has nothing to give; the primary phone must transfer an encrypted history bundle to the new device.
  • Device limits exist for a reason: every device multiplies fan-out, and with E2E, every message is encrypted once per recipient device.

Step 12 — Group chat versus channels

The design above stores each message once in the conversation partition and fans out notifications. What changes with size is how aggressively you push.

1:1Group (up to ~1k)Channel / broadcast (up to millions)
Who may writeBothEvery memberAdmins only
Message storageOnce, in the conversationOnce, in the conversationOnce, in the channel
Fan-out on writePush to 1 user's devicesPush to every online member device; bump every member's user_conversations rowNo per-member writes. Push only to members currently viewing; badge via a periodic "channel moved" hint
Offline membersPush notificationPush notification (collapsed, respects mute)Usually none, or a rate-limited digest
Read pathSync from cursorSync from cursorFan-out on read: the client pulls seq > cursor when it opens the channel; the tail is served from cache/CDN
ReceiptsFullAggregated watermarksView counters only
Hot spotNoneMembership read per sendOne partition, millions of readers: replicate the tail into cache
  • Routing a group message: the send goes to the owner of g:<id>, which reads the member list (cached, invalidated on membership change), persists once, and emits one log event. Fan-out workers split the member list into batches of ~500 and resolve devices in the registry with pipelined lookups, grouping deliveries by connection server so each box receives one frame listing its recipients.
  • Membership changes are messages in the log. "Cara was added" is a system message at a seq. Its joined_seq is what stops Cara reading history from before she joined, and with E2E it triggers a key rotation.
  • The 50,000-member "group" is a channel. Writing 50,000 inbox rows per message is exactly the push-model fan-out problem of Lesson 11: News Feed, and the answer is the same: switch to pull above a threshold.

Step 13 — Presence and typing at scale

Presence looks trivial — the server knows who has a socket — until you count the fan-out.

online_users      = 125_000_000
transitions_per_h = 6          # app foreground/background flips per online user per hour
contacts          = 150        # average address-book contacts on the platform
online_contacts   = 0.25       # share of those contacts online right now
visible_rows      = 12         # chats on screen that actually render a presence dot

changes = online_users * transitions_per_h / 3600
naive   = changes * contacts * online_contacts          # push to every online contact
lazy    = changes * visible_rows * 0.3                  # only subscribers who are looking

print(f"presence changes: {changes:,.0f}/s")
print(f"broadcast to all online contacts: {naive/1e6:,.1f} M pushes/s")
print(f"subscribe-on-view only:           {lazy/1e6:,.2f} M pushes/s  "
      f"({naive/lazy:.0f}x less)")

# Debounce: a flip that reverts within the window is never published.
window_s, flap_share = 5, 0.40  # 40% of offline->online->offline flaps are < 5 s
print(f"after {window_s}s debounce: {lazy*(1-flap_share)/1e6:,.2f} M pushes/s")
Output
presence changes: 208,333/s
broadcast to all online contacts: 7.8 M pushes/s
subscribe-on-view only:           0.75 M pushes/s  (10x less)
after 5s debounce: 0.45 M pushes/s
  • Online is derived from the socket, with a TTL. The connection server sets presence:u:42 with a 60 s TTL and refreshes it on heartbeats. A crashed server's users go offline automatically when the TTLs lapse — no cleanup job required.
  • Subscribe on view, not on friendship. A client sends SUB_PRES for the dozen chats on screen; the presence service records which connection servers care about which users and pushes changes only there. Scrolling away unsubscribes. This is the 10x in the numbers above.
  • Debounce transitions. Publish "online" only after the user has been connected for a few seconds, and "offline" only after the socket has been gone for ~10 s. Tunnels and app switches would otherwise make the dot flicker and multiply the traffic.
  • Last seen is coarse and lazy. It is written once on disconnect and read when a chat opens. Nobody needs it pushed.
  • Large groups get no member presence — show "3 online" from a sampled count, or nothing.
  • Typing indicators are pure pub/sub. Fire-and-forget through the connection servers, rate-limited to one frame every 3 s per conversation, auto-expired by the client after 5 s. Never persisted, never retried. Losing one is invisible.

Step 14 — Media

  1. The client computes a SHA-256 of the file (and, with E2E, encrypts it with a fresh random AES key first).
  2. POST /v1/media/uploads returns a media_id and a pre-signed URL; the client uploads directly to object storage in resumable chunks. Chat servers never touch the bytes.
  3. The message itself is small: {media_ref, mime, size, sha256, thumbnail (a few KB, inline), key (E2E only)}. It goes through the normal send path with a normal seq.
  4. Recipients download from the CDN on demand; auto-download policies (Wi-Fi only, size caps) live on the client.
  5. Dedupe identical forwarded files by hash on the server when not E2E. With E2E each upload is encrypted with a new key, so a viral video forwarded a million times is a million distinct blobs — a real, known cost of encryption.
  6. Lifecycle: tier media to cold storage after 30–90 days; relay-only systems delete it once all recipients have downloaded it.

This is the same pattern as Lesson 14: Google Drive and Lesson 13: Video Streaming: metadata in the database, bytes in object storage, clients talk to storage directly.

Step 15 — End-to-end encryption, sketched

If the interviewer asks for E2E, the architecture above survives almost unchanged; what changes is what the server may know. A Signal-protocol-style sketch:

  • Keys live on devices. Each device has a long-term identity key pair, a medium-term signed pre-key, and a batch of one-time pre-keys. It uploads only the public halves to a key directory (POST /v1/devices).
  • Session setup works while the recipient is offline. Alice's device fetches Bob's public key bundle and performs an asynchronous key agreement (X3DH) to derive a shared secret without Bob being online. The server hands out each one-time pre-key only once.
  • Every message has a new key. A double ratchet advances the key with each message and each reply, giving forward secrecy (a stolen key cannot decrypt old messages) and post-compromise recovery (future messages become safe again).
  • Multi-device means per-device sessions. A message to Bob, who has three devices, is encrypted three times, plus once for each of Alice's other devices. The envelope carries a ciphertext per device.
  • Groups use sender keys. Each member distributes a group sending key to every other member once, over the pairwise channels. After that a group message is encrypted once and fanned out by the server as usual. Removing a member forces everyone to rotate their sender key.
The server can stillThe server can no longer
Route, store and sequence opaque ciphertext; assign seq; dedupe by client_msg_idRead message bodies, media or group names
Know metadata: who talks to whom, when, message sizes, group membership, IPsSearch messages, generate link previews or run content moderation server-side
Deliver receipts, presence and push (with contentless payloads)Show message previews in push notifications (the app decrypts locally after wake-up)
Serve key bundles and detect key changes (the "security code changed" warning)Restore history to a new device without a client-held backup key
Rate-limit and block on metadata signalsDeduplicate media across users

Say the honest limitation too: the server distributes public keys, so a malicious server could hand out a fake key. The defence is out-of-band verification (safety numbers, QR codes) and, in newer systems, a verifiable key-transparency log.

Step 16 — Scaling each layer

Connection tier (125M sockets)

  • Routing new connections: an L4 load balancer or DNS with anycast picks any healthy box with least connections. There is no need for user-affinity at the load balancer — the older idea of a load balancer that maps user ids to servers breaks every time the fleet resizes. The session registry is the map.
  • Finding a user's box: HGETALL session:u:42 → cs-0317. The registry is sharded Redis, about 125M small hashes (~15 GB), replicated per zone. If it is lost, it rebuilds itself within one heartbeat interval as every connection re-registers.
  • Box-to-box delivery: fan-out workers talk to connection servers over persistent gRPC streams; one frame per box per batch, not one call per recipient.
  • Per-connection memory is the capacity limit. Trim TLS buffers, use one goroutine/epoll slot rather than one thread per socket, and hold only a user id, device id and cursor summary per connection.

Message service and log

  • Stateless apart from per-conversation sequencer ownership, which is spread by consistent hashing over conv_id. A hot group (a company-wide 1,000-member chat) is still only a few messages per second — sequencing one conversation is never the bottleneck; fan-out is.
  • The log is partitioned by conv_id hash into thousands of partitions, so ordering is preserved per conversation while throughput scales out.

Storage partitioning

  • Hash (conv_id, bucket) onto a token ring (Cassandra's native model), replication factor 3 across three zones, writes and reads at QUORUM for the log, ONE for the chat-list index where a moment of staleness is fine.
  • If you build it yourself on MySQL-style shards: start with many logical shards (say 4,096) mapped onto a few physical servers, then move logical shards to new hardware as data grows. shard = hash(conv_id) % 4096 never changes; only the logical-to-physical map does. See Lesson 4: Databases.
  • Five years at 17 PB is mostly cold. Compact older buckets into a cheaper tier (larger SSTables, erasure-coded object storage) — the Reed-Solomon option the storage team will suggest — and keep recent months on fast disks.

Cache

  • Opening a chat reads the most recent page — the last 50 messages of conversations with activity in the past day. Keep those in a Redis list per conversation (recent:<conv_id>) written through by the message service. Most reads are for the last hour, so the hit rate is very high with a small cache.
  • Cache member lists for groups (read on every send) and the user's chat-list page.
  • Do not cache the full history. Scrolling back months is rare and a sequential range scan is already efficient.

Step 17 — Deploying a stateful connection fleet

Restarting a stateless API server loses nothing. Restarting a connection server disconnects 250,000 people at once, and if they all reconnect in the same second, the neighbours fall over. Deploys need their own procedure:

  1. Mark the box draining. The load balancer stops sending it new connections.
  2. Send GOAWAY {reconnect_after_ms} to its sockets in batches, spreading the reconnects over 5–10 minutes, with a random jitter per client.
  3. Clients reconnect elsewhere and resume with their cursors in HELLO. Because the log, not the socket, is the truth, nothing in flight is lost: anything the old box did not deliver shows up as a sync gap.
  4. Close the stragglers after the drain window and terminate the process.
  5. Roll a small percentage of the fleet at a time, and keep connection servers thin so they need deploying rarely — business logic lives in the message service, which deploys like any stateless service.

The same machinery handles an unplanned crash, except that the clients discover it themselves (heartbeat timeout) and must use exponential backoff with full jitter on reconnect, or a single box failure becomes a synchronised reconnect storm.

Step 18 — Fault tolerance and failure modes

TCP connections cannot be migrated to another machine; do not design as if they could. Every failure below is survived the same way: clients reconnect, and the log plus cursors repair whatever was in flight.

What failsWhat the user seesMechanism
Connection server crashesA few seconds of "Connecting..."Clients detect missed heartbeats, back off with jitter, reconnect to another box, resync from cursors. Registry TTLs clean up stale entries.
Sender's ack lost on a flaky networkNothing (message stays on clock briefly)Client retries with the same client_msg_id; dedupe returns the original seq.
Recipient socket is half-openTwo ticks arrive late, not falselyNo DELIV_ACK means no two ticks. The device syncs on reconnect.
Fan-out worker crashes mid-batchNothingLog offset not committed, batch redelivered, devices drop duplicates by seq.
Message store replica or zone downNothingRF 3 across zones; quorum still reachable with one zone lost.
Quorum unavailable for a partitionSends to those chats show the clock and retryRefuse the write rather than ack without durability. The client outbox holds it.
Sequencer owner diesSub-second pause in affected chatsNew owner reads latest_seq, bumps the fencing epoch; writes from the old owner with a stale epoch are rejected.
Session registry lostSome deliveries fall back to push for ~30 sEvery connection re-registers on its next heartbeat; in the meantime messages are durable and synced.
Presence cluster downNo green dots, "last seen" stalePresence is best-effort by design; chat keeps working.
APNs / FCM slow or downLate notifications for backgrounded appsPush queue backs up independently; messages arrive on next app open via sync.
Region lostUsers in that region reconnect to anotherConversations are homed in a region; cross-region async replication means a few seconds of recently acked messages may be delayed until the region returns. Be explicit about that RPO.
Celebrity channel post (millions of readers)Slightly delayed channel updateFan-out on read; the tail is served from cache/CDN.

Step 19 — Observability

  • End-to-end delivery latency, measured from the sender's SEND to the recipient's DELIV_ACK, sampled and reported by clients. Server-side latency looks perfect when the problem is mobile networks.
  • Undelivered age: the oldest message persisted but not yet delivered to an online device. It catches fan-out stalls that no error rate shows.
  • Gap-repair rate: how often clients detect a sequence gap and call sync. A rising rate means the push path is silently dropping frames.
  • Duplicate-send rate (dedupe hits), reconnects per minute per box, sockets per box, heartbeat timeouts, registry misses, push provider error rates, and log consumer lag per partition.
  • Canaries: synthetic accounts in every region that message each other every few seconds and alert when round-trip exceeds the SLO.

The trade-off summary — how to close

  • A per-conversation log with dense sequence numbers, not timestamps. Gives one order on every device and gap detection. We pay with a sequencer per conversation and a failover protocol for it.
  • At-least-once delivery plus idempotent display, not "exactly-once delivery". Exactly-once over an unreliable network does not exist; client_msg_id and seq give the user the same experience.
  • WebSocket for real-time, HTTPS for bulk. The socket carries small frames; sync and history go to a stateless, cacheable fleet.
  • Persist, then ack, then fan out. Sender latency is bounded by one quorum write; recipients are decoupled through the log.
  • Wide-column store partitioned by conversation, with a per-user index. One copy per group message, cheap range scans. We give up cross-conversation queries and server-side joins, which chat does not need.
  • Watermark receipts, subscribe-on-view presence, pull for channels. Three places where the naive design multiplies traffic by 10–1,000x.
  • A thin, rarely deployed connection tier with graceful draining, because it is the only stateful edge in the system.
  • What I would build first: one region, a WebSocket server, a single-node Cassandra or even Postgres with the same schema, 1:1 only — and the sync-from-cursor API on day one, because every later feature leans on it.
// Write your solution here
// Write your solution here
// Write your solution here

Finished reading? Mark this lesson complete to track your progress.

Up next · Lesson 13Video Streaming (YouTube)Upload pipelines, transcoding, adaptive bitrate, CDN strategy and cost.