System Design · Lesson 17 of 18
Notification System
Push, SMS, email fan-out, templates, preferences, retries and dedupe.
- Advanced
- 55 min read
- 6 objectives
Before this lessonLesson 16: Typeahead and Nearby Search
What you will learn
- Size a multi-channel notification platform, including 50M-user campaign blasts
- Isolate OTPs from marketing with per-channel, per-priority queues and reserved quota
- Make sends effectively-once with idempotency keys, dedupe and backoff with jitter
- Respect provider rate limits with token buckets and fail over between SMS vendors
- Enforce preferences, quiet hours and CAN-SPAM, GDPR and TCPA rules
- Track delivery, clean up dead tokens and collapse noisy bursts into digests
Your Progress
0 of 18 lessons 0%
- Lessons0 / 18
- Completed0
- Est. time left~ 16 hours
Progress is saved in this browser. No account needed.
Tip: pressing Next marks this lesson complete automatically.
Design a notification platform: one internal service that every product team calls to reach a user on iOS push, Android push, web push, SMS, email or the in-app inbox. It has to send a login code in under two seconds, tell 50 million people about a sale without melting the SMS vendor, never wake someone at 3 a.m. for a "like", honour every unsubscribe, and not send the same message twice when a worker crashes halfway.
It looks like the easiest prompt in the course ("just call the provider's API"), which is exactly why it is asked. The interesting part is not calling APNs. It is everything around the call:
- Two workloads that share nothing but the pipes. Transactional traffic (OTPs, order updates, security alerts) is low volume, latency critical and legally allowed to reach anyone. Marketing traffic is enormous, bursty, latency tolerant and wrapped in consent law. Putting both in one queue is the classic mistake.
- You do not own the last hop. APNs, FCM, Twilio and SES have their own rate limits, outages, error codes and pricing. Your system is a well-behaved client of several unreliable, metered third parties.
- "Exactly once" is impossible, and the interviewer knows it. The provider call is a side effect outside your transaction. The question is how close you get and what you do about the gap.
If you have done the chat lesson, you have seen the narrow version of this: "the recipient is offline, so hand the message to the phone vendor's push service". This lesson is the whole platform behind that one arrow.
Step 1 — Clarify the requirements
The channels, and why they are not interchangeable
Say this table out loud early. Every later decision (queues, retries, cost, compliance) differs per row.
| Channel | Address | Latency | Cost per send | What "delivered" means | Quirks |
|---|---|---|---|---|---|
| iOS push (APNs) | Device token per app install | Sub-second, usually | Free | Only "Apple accepted it"; no receipt | ~4 KB payload, HTTP/2, token dies on uninstall (HTTP 410) |
| Android push (FCM) | Registration token | Sub-second, usually | Free | "Google accepted it"; optional delivery data in aggregate | ~4 KB payload, per-project quotas, Doze mode delays low-priority messages |
| Web push | Browser subscription (endpoint URL + keys) | Seconds | Free | Push service accepted it | Payload encrypted per subscription (VAPID), endpoint differs per browser vendor |
| SMS | E.164 phone number | 1–10 s | Real money: roughly $0.005–$0.10, far more in some countries | Carrier delivery receipt (DLR), sometimes | 160 GSM-7 chars per segment (70 if any emoji), carrier filtering, sender registration, toll fraud |
| Address | Seconds to minutes | Fractions of a cent | Accepted by the receiving mail server; opens are guesses | Sender reputation, SPF/DKIM/DMARC, bounces, spam complaints, one-click unsubscribe | |
| In-app inbox | User id | Instant if the app is open | Your own storage | You know exactly: it is your database | The only channel you fully control, and the fallback for everything else |
Functional, in scope for v1
- Product services send a notification by naming a template and a recipient (or an audience), not by passing a rendered string.
- Channels: APNs, FCM, web push, SMS, email, in-app inbox. A single request may fan out to several channels, with fallback ("push; if no device, SMS").
- Two priority classes: critical/transactional (OTP, password reset, payment failed, order status) and bulk (marketing, digests, social activity).
- User preferences per category and channel, global unsubscribe, quiet hours in the user's local time zone.
- Templates with versions and localisation; marketing can edit copy without a deploy.
- Campaigns: send to a segment of up to 50M users, schedulable, pausable, cancellable.
- Delivery tracking: per-notification status plus aggregate analytics (sent, delivered, opened, clicked, bounced, unsubscribed).
- Collapse and digest: "Asha and 6 others liked your photo", not seven pushes.
Explicitly out of scope
Segment building and audience analytics (we accept a segment id from a marketing tool), A/B test statistics, the in-app message designer, inbound SMS conversations beyond STOP/HELP keywords, voice calls, and the recommendation model that decides what to send. Naming these shows you know where the boundary is.
Non-functional
- Scale: 100M daily active users, around 1.5 billion notifications a day across channels, plus campaign blasts of 50M recipients.
- Latency: critical lane: accepted by the provider within 2 s p99 of the API call. Bulk lane: minutes are fine, and a campaign completes within 30–60 minutes.
- Isolation: a campaign running at full speed must not add a single second to an OTP. This is the non-functional requirement that shapes the architecture.
- Delivery semantics: at-least-once inside our system, deduplicated so the user sees each notification once in all but rare crash windows. Never silently drop a critical message.
- Availability: 99.99% for the send API. Survive the loss of any single provider.
- Compliance: CAN-SPAM, GDPR and TCPA (details in Step 12). An unsubscribe must be honoured on the very next send, not "eventually".
- Cost: SMS is the one line item that shows up in board meetings. Prefer free channels and never send an SMS we did not need to.
Step 2 — Back-of-the-envelope estimation
DAU = 100_000_000
SECONDS_PER_DAY = 86_400
# --- Transactional + social: triggered by something a user or system did ---
per_user_per_day = {"push": 6, "in_app": 8, "email": 0.5, "sms": 0.05}
for ch, n in per_user_per_day.items():
avg = DAU * n / SECONDS_PER_DAY
print(f"{ch:7s} {DAU * n / 1e6:7,.0f}M/day {avg:9,.0f}/s avg {avg * 5:9,.0f}/s peak")
# --- OTPs: tiny volume, the strictest latency budget in the system ---
otp_per_day = 3_000_000
print(f"OTP {otp_per_day / SECONDS_PER_DAY:,.0f}/s avg -- p99 target: handed to SMS vendor < 2 s")
# --- One marketing campaign blast ---
audience = 50_000_000
devices_each = 1.4 # phones + tablets + browsers
window_minutes = 30 # "everyone within half an hour"
sends = audience * devices_each
print(f"campaign: {sends / 1e6:.0f}M pushes in {window_minutes} min = "
f"{sends / (window_minutes * 60):,.0f}/s")
# --- Storage: the notification log is the big table ---
log_row_bytes = 300 # ids, channel, status, timestamps, provider id
rows_per_day = DAU * sum(per_user_per_day.values())
log_tb_year = rows_per_day * 365 * log_row_bytes / 1e12
print(f"log: {rows_per_day / 1e9:.2f}B rows/day, {log_tb_year:,.0f} TB/year before TTL")
# --- Device tokens: small, fits in memory on a few nodes ---
tokens = DAU * 1.4 * 1.5 # include monthly-active, not just daily
print(f"device tokens: {tokens / 1e6:.0f}M x 250 B = {tokens * 250 / 1e9:.0f} GB")push 600M/day 6,944/s avg 34,722/s peak in_app 800M/day 9,259/s avg 46,296/s peak email 50M/day 579/s avg 2,894/s peak sms 5M/day 58/s avg 289/s peak OTP 35/s avg -- p99 target: handed to SMS vendor < 2 s campaign: 70M pushes in 30 min = 38,889/s log: 1.46B rows/day, 159 TB/year before TTL device tokens: 210M x 250 B = 52 GB
What the numbers actually decide
- Steady-state push is ~7k/s, peak ~35k/s. That is a modest Kafka cluster and a few dozen push workers holding HTTP/2 connections. Not the hard part.
- A single campaign roughly doubles peak push load for half an hour (39k/s on top of 35k/s). If that traffic shares a queue with OTPs, logins stall every time marketing presses "send". Separate lanes, not a priority field.
- OTPs are 35/s. The critical lane is tiny, so it can be over-provisioned cheaply: dedicated workers, dedicated provider connections, a reserved slice of every vendor quota.
- SMS at 289/s peak is a contract number, not a compute number. Vendors and carriers throttle per sender, so the bottleneck is how many sender numbers and vendor accounts you hold, not how many servers you run.
- 1.5 billion log rows a day, 159 TB a year is a write-heavy, append-only, time-ordered, rarely-updated table. That is a wide-column store (Cassandra, ScyllaDB, DynamoDB) with a TTL, not Postgres. Aggregates go to a columnar warehouse.
- Device tokens are only ~52 GB. They fit in a replicated key-value store with a cache in front. Nobody needs to shard this cleverly.
Step 3 — API design
An internal API for product services, plus a small user-facing API for preferences and devices. Auth for internal callers is service identity (mTLS or signed service tokens), and every caller has its own quota so one buggy team cannot flood users.
## Send (internal, called by product services)
POST /v1/notifications
Headers: Idempotency-Key: order-88231-state-PICKED_UP (required)
{
"template": "order_picked_up", # name, not rendered text
"recipient": { "user_id": 42 }, # or { "email": ... } for non-users
"data": { "courier_name": "Ravi", "eta_minutes": 12 },
"channels": ["push", "in_app"], # optional; template has defaults
"fallback": ["sms"], # only if no channel above worked
"priority": "critical", # critical | bulk
"category": "order_updates", # drives preferences + compliance
"collapse_key": "order-88231", # newer replaces older on device
"expires_at": "2026-09-23T14:45:00Z" # stale after this: drop, do not send
}
-> 202 { notification_id: "n_01J9...", status: "QUEUED" }
200 { same body } # replay of the same Idempotency-Key
400 UNKNOWN_TEMPLATE | 400 MISSING_TEMPLATE_VARIABLE
409 IDEMPOTENCY_KEY_REUSED # same key, different body
422 CATEGORY_NOT_ALLOWED_FOR_CALLER
429 + Retry-After # per-caller quota
# 202, not 201 or 200: the notification is accepted, not sent. Sending is async.
GET /v1/notifications/{id}
-> 200 { id, status, attempts: [{ channel, provider, status, at, error? }] }
## Campaigns (internal, called by the marketing tool)
POST /v1/campaigns
{ "template": "autumn_sale", "segment_id": "seg_551",
"channels": ["push", "email"], "send_at": "2026-10-01T09:00:00",
"send_at_mode": "recipient_local_time", # or "absolute"
"max_rate_per_sec": 20000 }
-> 201 { campaign_id, status: "SCHEDULED", estimated_recipients: 50000000 }
POST /v1/campaigns/{id}/pause | /resume | /cancel -> 200 | 409 ALREADY_COMPLETED
GET /v1/campaigns/{id} -> 200 { status, sent, suppressed, failed, progress_pct }
## Templates (internal, versioned, never edited in place)
POST /v1/templates/{name}/versions
{ "channel": "push", "locale": "en", "title": "Your order is on the way",
"body": "{{courier_name}} picked it up. Arriving in {{eta_minutes}} min." }
-> 201 { version: 7 } # publish is a separate step, so drafts are safe
POST /v1/templates/{name}/versions/7/publish -> 200
## User-facing: devices and preferences
POST /v1/me/devices { platform: "ios", token: "...", app_version, locale, tz }
-> 201 | 200 (same token re-registered: update last_seen, do not duplicate)
DELETE /v1/me/devices/{token} -> 204 # called on logout
GET /v1/me/preferences
-> 200 { quiet_hours: { start: "21:00", end: "08:00", tz: "Asia/Kolkata" },
categories: { marketing: { push: false, email: true, sms: false },
social: { push: true, email: false } } }
PATCH /v1/me/preferences { categories: { marketing: { email: false } } } -> 200
## Unsubscribe (public, from the email footer, no login)
GET /u/{signed_token} -> 200 confirmation page
POST /u/{signed_token} -> 200 # RFC 8058 one-click, List-Unsubscribe-Post
# signed_token encodes user_id + category + expiry, HMAC-signed, so it cannot be
# forged to unsubscribe someone else and needs no password.
## Provider webhooks (inbound)
POST /hooks/ses | /hooks/twilio | /hooks/sendgrid
# bounces, complaints, delivery receipts, STOP replies. Verify the signature,
# enqueue, return 200 fast. Never do work inline: providers retry slow hooks.API decisions worth defending
- Callers send a template name and data, never a rendered string. Copy changes, translations and compliance footers happen in one place. It also means the platform knows the category, which is what the law cares about.
Idempotency-Keyis required and chosen by the caller from its own domain (order-88231-state-PICKED_UP), not a random UUID. A domain key survives the caller's own retries and its own crashes; a random UUID generated per attempt dedupes nothing.- 202 Accepted. The API returns once the request is durably queued. Anything slower couples every product service's latency to APNs.
expires_atis on every message. "Your driver is arriving" delivered 40 minutes late after an outage is worse than nothing. Workers drop expired messages instead of sending them.- Priority and category are validated against the caller. The marketing service may not mark a promo as
criticalto skip quiet hours, and the API returns 422 if it tries. Otherwise every team eventually does exactly that.
Step 4 — Data model
Five kinds of data with five different access patterns, so they do not all go in one database.
-- =========================================================
-- DEVICE TOKENS (key-value / Postgres, ~200M rows, read on every push)
-- =========================================================
device_tokens (
token TEXT PRIMARY KEY, -- APNs/FCM token or web-push endpoint
user_id BIGINT NOT NULL,
platform TEXT NOT NULL, -- ios | android | web
app_bundle TEXT NOT NULL, -- prod vs beta apps have different APNs topics
locale TEXT, tz TEXT, -- captured from the device, used as defaults
last_seen_at TIMESTAMPTZ NOT NULL, -- refreshed on app open; drives stale pruning
invalidated_at TIMESTAMPTZ NULL -- set on 410 / UNREGISTERED, deleted later
);
CREATE INDEX ON device_tokens (user_id) WHERE invalidated_at IS NULL;
-- Token is the PK, not (user_id, token): one physical device can be handed to a
-- new user after logout, and the token must move with it, not exist twice.
-- =========================================================
-- PREFERENCES + SUPPRESSION (Postgres source of truth, Redis cache)
-- =========================================================
user_channel_prefs (
user_id BIGINT NOT NULL,
category TEXT NOT NULL, -- marketing | social | order_updates | security
channel TEXT NOT NULL, -- push | email | sms | in_app
enabled BOOLEAN NOT NULL,
source TEXT NOT NULL, -- settings_page | email_footer | sms_stop | import
updated_at TIMESTAMPTZ NOT NULL, -- consent evidence: when and how
PRIMARY KEY (user_id, category, channel)
);
-- Sparse: only rows that differ from the category default are stored.
user_settings (
user_id BIGINT PRIMARY KEY,
quiet_start TIME, quiet_end TIME, tz TEXT NOT NULL DEFAULT 'UTC',
locale TEXT NOT NULL DEFAULT 'en',
sms_marketing_consent_at TIMESTAMPTZ NULL -- TCPA: express written consent record
);
suppression_list ( -- beats every preference, checked last-mile
address TEXT NOT NULL, -- email or E.164 phone, normalised
channel TEXT NOT NULL,
reason TEXT NOT NULL, -- hard_bounce | complaint | sms_stop | legal
created_at TIMESTAMPTZ NOT NULL,
PRIMARY KEY (address, channel)
);
-- Keyed by ADDRESS, not user_id: a bounced address stays suppressed even if a
-- new account signs up with it tomorrow.
-- =========================================================
-- TEMPLATES (Postgres, tiny, cached in every worker)
-- =========================================================
templates (
name TEXT, version INT, channel TEXT, locale TEXT,
category TEXT NOT NULL, -- a template has exactly one category
subject TEXT, title TEXT, body TEXT NOT NULL,
required_vars TEXT[] NOT NULL, -- validated at the API, not at send time
status TEXT NOT NULL, -- draft | published | retired
PRIMARY KEY (name, version, channel, locale)
);
-- Immutable versions: the log records (name, version) so you can always show
-- exactly what a user received, even after the copy changed.
-- =========================================================
-- NOTIFICATION LOG (Cassandra / ScyllaDB, 1.5B rows/day, TTL 90 days)
-- =========================================================
CREATE TABLE notifications_by_user (
user_id bigint,
day date, -- bucket keeps partitions bounded
notification_id timeuuid, -- time-sortable
template text, template_version int,
category text, channel text, provider text,
status text, -- QUEUED SENT DELIVERED OPENED FAILED SUPPRESSED
provider_msg_id text, -- to join webhooks back to this row
error text,
PRIMARY KEY ((user_id, day), notification_id)
) WITH CLUSTERING ORDER BY (notification_id DESC)
AND default_time_to_live = 7776000; -- 90 days; analytics keeps aggregates
-- Second table keyed by provider_msg_id so a Twilio webhook can find its row
-- without scanning. Wide-column stores want one table per query.
-- =========================================================
-- IN-APP INBOX (same store, partitioned by user, like a chat inbox)
-- =========================================================
CREATE TABLE inbox (
user_id bigint, notification_id timeuuid,
title text, body text, deep_link text, read_at timestamp,
PRIMARY KEY (user_id, notification_id)
) WITH CLUSTERING ORDER BY (notification_id DESC);
-- Unread badge count kept as a Redis counter, not COUNT(*).
-- =========================================================
-- DEDUPE + COLLAPSE (Redis, ephemeral)
-- =========================================================
SET dedupe:{event_id}:{user_id}:{channel} 1 NX EX 86400
SET idem:{caller}:{idempotency_key} '{"id":"n_01J9..","hash":".."}' NX EX 172800
HINCRBY digest:{user_id}:like:photo:9 count 1 -- open collapse window
EXPIRE digest:{user_id}:like:photo:9 60
INCR cap:{user_id}:push:2026092314 -- per-user hourly cap
-- =========================================================
-- CAMPAIGNS (Postgres; the audience itself is a file in object storage)
-- =========================================================
campaigns (id, template, segment_id, status, send_at, send_at_mode,
max_rate_per_sec, audience_uri, total, created_by, created_at);
campaign_chunks (campaign_id, chunk_no, status, attempts, -- 1 row per 1,000 users
PRIMARY KEY (campaign_id, chunk_no));Modelling decisions to explain
- The suppression list is keyed by address and checked at the last possible moment, right before the provider call. Preferences can be cached and slightly stale; suppression is the legal backstop and is checked fresh.
- Preferences store
sourceandupdated_at. When a regulator or a lawsuit asks "why did you text this person", you need the consent record: when, how, from which screen. This is a compliance table wearing a settings-table costume. - The log lives in a wide-column store with a TTL, partitioned by
(user_id, day)so no partition grows without bound. Candidates who put 1.5B rows a day in Postgres spend the rest of the interview on vacuum and partition management. - Templates are immutable versions. Editing in place means the log cannot tell you what was actually sent, and a typo "fix" can break a campaign halfway through.
- Dedupe keys have a TTL. They only need to outlive the redelivery window (Kafka retention, retry budget). A day is plenty; forever is a memory leak.
- The campaign audience is a snapshot file, not a live query. Recipients resolved at 09:00 do not change because someone signed up at 09:05, and a resumed campaign sends to exactly the remaining users.
Step 5 — High-level design
auth, validate, idempotency] MKT[Marketing tool] --> CAMP[Campaign service] API --> K1[(Kafka: requests.critical)] API --> K2[(Kafka: requests.bulk)] CAMP --> OBJ[(Audience snapshot
object storage)] CAMP --> K2 K1 --> ROUTER[Router / fan-out workers] K2 --> ROUTER ROUTER --> PREF[(Prefs + settings
Redis cache)] ROUTER --> DEV[(Device tokens)] ROUTER --> TPL[Template renderer] ROUTER --> SCHED[(Delay queue
quiet hours, digests)] SCHED --> ROUTER ROUTER --> QP1[(push.critical)] ROUTER --> QP2[(push.bulk)] ROUTER --> QS1[(sms.critical)] ROUTER --> QS2[(sms.bulk)] ROUTER --> QE[(email.*)] ROUTER --> INBOX[(In-app inbox)] QP1 --> PW[Push workers] QP2 --> PW QS1 --> SW[SMS workers] QS2 --> SW QE --> EW[Email workers] PW --> APNS[APNs] PW --> FCM[FCM] PW --> WP[Web push services] SW --> SV1[SMS vendor A] SW --> SV2[SMS vendor B] EW --> ESP[Email provider] PW --> LOG[(Notification log)] SW --> LOG EW --> LOG HOOK[Webhook ingest] --> LOG SV1 --> HOOK ESP --> HOOK LOG --> WH[(Analytics warehouse)]
Two hops of queues. The first splits by priority so a campaign never sits in front of an OTP. The router resolves who, where and whether, then writes to per-channel, per-priority queues so each provider's workers scale, throttle and fail independently.
What each component owns
- Notification API — authenticates the calling service, enforces per-caller quotas, validates the template and its variables, claims the idempotency key, writes to the right Kafka topic, returns 202. It never talks to a provider.
- Campaign service — resolves a segment to an audience snapshot, splits it into chunks, schedules waves (including per-time-zone waves), paces them into
requests.bulk, and owns pause and cancel. - Router — the brain. For each request: expand the recipient to devices and addresses, apply preferences and suppression, apply quiet hours (park in the delay queue), dedupe, collapse into digests, render the template per channel and locale, choose channels and fallbacks, and enqueue one message per (device or address).
- Delay queue — holds messages until a time: quiet-hours releases, digest window closes, retries with long backoff. A Redis sorted set scored by release time, or a timer service. Not a sleeping thread.
- Channel workers — dumb and fast. Pull a message, check
expires_atand the suppression list, take a token from the provider's bucket, call the provider, classify the response, retry or record. One worker pool per channel, with separate consumer groups for critical and bulk. - Webhook ingest — receives bounces, complaints, delivery receipts and STOP replies, verifies signatures, and turns them into log updates, suppressions and token invalidations.
- Notification log and warehouse — per-message status for support ("did the user get the reset email?") and aggregates for marketing ("what was the open rate?").
Why two queue hops rather than one? The first hop is cheap and uniform (a request), the second is channel-specific (one APNs payload). Fan-out between them can be 1 to 5 messages per request, and for a campaign chunk, 1 to 1,400. Keeping them separate lets the router scale on CPU (rendering, preference lookups) and the workers scale on provider throughput.
Step 6 — Priority lanes: OTP versus marketing
The most common wrong answer is "put a priority field on the message and have workers pick high-priority first". It fails in three ways:
- Kafka partitions are FIFO. There is no "pick the high-priority one" within a partition. An OTP enqueued behind 2 million campaign messages waits for all of them.
- Even with a priority queue, shared workers are a shared bottleneck. If all push workers are blocked on slow APNs calls for the campaign, there is nobody free to take the OTP.
- Shared provider quota. If the campaign spends the entire SMS vendor quota, the OTP gets a 429 from the vendor no matter how fast your queue is.
The fix is isolation at every layer:
| Layer | Critical lane | Bulk lane |
|---|---|---|
| Kafka topics | requests.critical, sms.critical, push.critical | requests.bulk, sms.bulk, push.bulk |
| Workers | Dedicated pool, over-provisioned 5–10x, never autoscaled down to zero | Autoscaled on consumer lag, can be paused |
| Provider quota | Reserved slice (e.g. 20% of SMS throughput) that bulk can never spend | The rest, throttled to protect sender reputation |
| Provider accounts | Often a separate sender id / short code for OTPs | Marketing sender, marketing IP pool for email |
| Quiet hours, digests, caps | Bypass all of them | Subject to all of them |
| Retry budget | Short (a few seconds of total retry, then fail over or fall back); an OTP after 2 minutes is useless | Long (hours), with deep backoff |
| Failure alerting | Pages someone in minutes | A dashboard and a morning ticket |
Separate IP pools and sender ids for transactional email matter more than they look: if a marketing blast gets flagged as spam and damages the marketing IPs' reputation, password resets still land in the inbox.
Step 7 — Walk an OTP end to end
The critical path from API to vendor is two queue hops and one HTTP call, typically well under a second. A slow vendor is abandoned quickly for a second vendor because an OTP that arrives late is a failed login.
The order of operations, and why
- Claim the idempotency key before producing. If the auth service retries its request after a timeout, the second call returns the same
notification_idinstead of sending a second code. If the key claim succeeds but the produce fails, the API releases the key (or marks it failed) so the caller's retry can proceed. - Partition by
user_id. All of one user's notifications land on one partition in order, so "your code is 4412" is never overtaken by an earlier, now-invalid code. - Check
expires_atin the worker, not only at the API. An OTP typically expires in 5–10 minutes. If the queue backed up during an incident, sending codes that are already invalid wastes money and confuses users. - Failover, not blind retry, on a timeout. A timeout is an unknown outcome: vendor A might have sent it. For an OTP a rare duplicate SMS is acceptable and a missing one is not, so we fail over. For a marketing SMS the trade flips: we wait and retry the same vendor, or drop.
- Pass our
notification_idas the vendor's client reference. Delivery receipts come back with it, which is how the webhook finds the log row, and some vendors use it to reject duplicates. - The user can always ask again. The login screen offers "resend code" after 30 seconds and "call me instead" after two tries. Product design is part of the reliability budget.
Step 8 — Idempotency, dedupe and digests
Where duplicates come from
- The caller retries after a timeout on its request to us. Solved by the
Idempotency-Keyat the API. - Kafka redelivers because a worker processed a message but crashed before committing its offset. Solved by a dedupe key checked in the worker.
- The same logical event is produced twice by an upstream bug (two services both notify on "order delivered"). Solved by a dedupe key built from the event, not from the request.
- The provider accepted it, then we crashed before recording it. Not fully solvable. See below.
So there are two keys at two layers. The API key is (caller, Idempotency-Key) and protects against request retries. The worker key is (event_id, user_id, channel) and protects against redelivery. The worker key is set with SET NX before the provider call; if the call then fails with a retryable error, the worker deletes the key (or, better, stores a state: SENDING / SENT) so the retry is allowed.
import hashlib
class DedupeStore:
"""Stands in for Redis: SET key 1 NX EX ttl. Returns True only for the first writer."""
def __init__(self): self.keys = {}
def set_nx(self, key, now, ttl):
exp = self.keys.get(key)
if exp is not None and exp > now:
return False
self.keys[key] = now + ttl
return True
def dedupe_key(event_id, user_id, channel):
# Keyed on the PRODUCER's event id, not on the text. Two different OTPs with
# the same wording must both send; the same OTP delivered twice by Kafka must not.
return hashlib.sha1(f"{event_id}:{user_id}:{channel}".encode()).hexdigest()[:16]
class Digester:
"""Collapse bursts: the first like sends now; later likes inside the window are
folded into one summary sent when the window closes."""
def __init__(self, window):
self.window, self.open = window, {} # (user, type, object) -> state
def offer(self, user, ntype, obj, actor, now):
k = (user, ntype, obj)
st = self.open.get(k)
if st is None or now >= st["closes"]:
self.open[k] = {"closes": now + self.window, "actors": []}
return f"SEND NOW {actor} liked your photo"
st["actors"].append(actor)
return None
def flush(self, now):
out = []
for k, st in list(self.open.items()):
if now >= st["closes"] and st["actors"]:
a = st["actors"]
more = f" and {len(a) - 1} others" if len(a) > 1 else ""
out.append(f"DIGEST {a[0]}{more} liked your photo")
del self.open[k]
return out
store = DedupeStore()
# Kafka redelivers event e-77 after a consumer crash: same id, three times.
for delivery in range(3):
k = dedupe_key("e-77", 42, "push")
print(f"delivery {delivery}: {'SEND' if store.set_nx(k, now=delivery, ttl=86_400) else 'DROP duplicate'}")
d = Digester(window=60)
names = ["Asha", "Ben", "Chen", "Dev", "Eli", "Fay", "Gus", "Hana"]
for i, who in enumerate(names):
r = d.offer(user=42, ntype="like", obj="photo:9", actor=who, now=i * 5)
if r: print(f"t={i*5:3d}s {r}")
for line in d.flush(now=61):
print(f"t= 61s {line}")delivery 0: SEND delivery 1: DROP duplicate delivery 2: DROP duplicate t= 0s SEND NOW Asha liked your photo t= 61s DIGEST Ben and 6 others liked your photo
The gap you should admit
Between "APNs returned 200" and "we wrote SENT", the worker can die. On restart the message is redelivered, the dedupe key says SENDING, and we must choose: send again (risk a duplicate) or skip (risk a loss). The honest answer is at-least-once with dedupe gives effectively-once in practice, with a tiny duplicate window, and you pick the direction per category: critical messages resend, bulk messages skip. You can narrow the window further with provider features: apns-collapse-id and FCM's collapse_key make a second copy replace the first on the device rather than appear twice, and a stable email Message-ID lets some clients thread duplicates together. Claiming true exactly-once across a third-party HTTP API is the wrong answer.
Collapse, digests and per-user caps
- Collapse on device: order status pushes share
collapse_key = order-88231, so the lock screen shows the latest state, not five stale ones. - Digest in the router: the first event in a window sends immediately (feels live); later events inside the window fold into one summary when it closes. The window key is
(user, type, object), stored in Redis with a TTL, and the flush is a delay-queue entry, not a timer in a worker's memory. - Daily or weekly email digests are the same idea with a longer window: accumulate social events into the user's digest row and send once at their local 9 a.m.
- Per-user frequency caps: at most N bulk pushes per user per hour and M per day (
INCR cap:{user}:push:{hour}). Overflow goes to the in-app inbox only. Uninstall rate is the metric that punishes you for skipping this.
Step 9 — Retries: exponential backoff with jitter
A retry policy is three decisions: which errors to retry, how long to wait, and when to stop. Getting the first one wrong is worse than having no retries at all.
- Retry: 429, 500, 502, 503, 504, connection resets, timeouts. The provider might succeed next time.
- Do not retry: 400 (bad payload, will fail forever), 403 (our credentials or certificate are wrong: page an engineer), 410 /
UNREGISTERED(the token is dead: invalidate it), invalid phone number, hard bounce. - Obey
Retry-Afterwhen a provider sends it. It is the provider telling you its recovery plan.
import random
# How each provider response should be treated. This table is the policy.
RETRYABLE = {429, 500, 502, 503, 504, "timeout", "conn_reset"}
PERMANENT = {400, 403, 404, 410} # 410 from APNs / UNREGISTERED from FCM: token is dead
def backoff_full_jitter(attempt, base=0.5, cap=60.0, rng=random):
"""AWS-style 'full jitter': sleep a random amount in [0, min(cap, base * 2^attempt)].
The randomness is the point -- it spreads 1M retrying workers across the window
instead of having them all hit the recovering provider at the same instant."""
return rng.uniform(0, min(cap, base * (2 ** attempt)))
def send_with_retry(send, max_attempts=6, retry_after=None, rng=random, log=print):
for attempt in range(max_attempts):
status = send(attempt)
if status == 200:
log(f" attempt {attempt}: 200 -> delivered")
return "DELIVERED"
if status in PERMANENT:
log(f" attempt {attempt}: {status} -> permanent, do not retry")
return "FAILED_PERMANENT"
if status in RETRYABLE:
if attempt == max_attempts - 1:
log(f" attempt {attempt}: {status} -> retry budget exhausted")
break
wait = backoff_full_jitter(attempt, rng=rng)
if status == 429 and retry_after:
wait = max(wait, retry_after) # the provider told us; obey it
log(f" attempt {attempt}: {status} -> retry in {wait:5.2f}s")
continue
log(f" attempt {attempt}: {status} -> unknown, treat as permanent")
return "FAILED_PERMANENT"
return "DEAD_LETTER" # parked for inspection, never silently dropped
rng = random.Random(7) # seeded so the output is reproducible
print("provider flaps, then recovers:")
script = [503, 503, "timeout", 200]
print(send_with_retry(lambda a: script[a], rng=rng))
print("token was uninstalled:")
print(send_with_retry(lambda a: 410, rng=rng))
print("rate-limited with Retry-After: 5")
script = [429, 200]
print(send_with_retry(lambda a: script[a], retry_after=5, rng=rng))
print("provider down for the whole budget:")
print(send_with_retry(lambda a: 503, rng=rng))
# Why jitter matters: 10,000 workers all failed at t=0 and retry attempt 3.
no_jitter = [0.5 * 2 ** 3] * 10_000
jittered = [backoff_full_jitter(3, rng=rng) for _ in range(10_000)]
def busiest_second(waits):
buckets = {}
for w in waits:
buckets[int(w)] = buckets.get(int(w), 0) + 1
return max(buckets.values())
print(f"peak retries in one second -- no jitter: {busiest_second(no_jitter):,}, "
f"full jitter: {busiest_second(jittered):,}")provider flaps, then recovers: attempt 0: 503 -> retry in 0.16s attempt 1: 503 -> retry in 0.15s attempt 2: timeout -> retry in 1.30s attempt 3: 200 -> delivered DELIVERED token was uninstalled: attempt 0: 410 -> permanent, do not retry FAILED_PERMANENT rate-limited with Retry-After: 5 attempt 0: 429 -> retry in 5.00s attempt 1: 200 -> delivered DELIVERED provider down for the whole budget: attempt 0: 503 -> retry in 0.27s attempt 1: 503 -> retry in 0.37s attempt 2: 503 -> retry in 0.12s attempt 3: 503 -> retry in 2.03s attempt 4: 503 -> retry in 0.30s attempt 5: 503 -> retry budget exhausted DEAD_LETTER peak retries in one second -- no jitter: 10,000, full jitter: 2,517
Why full jitter? Plain exponential backoff means every worker that failed at the same moment retries at the same moment, so a provider recovering from an outage is hit by a synchronised wave and falls over again. Randomising within the window cuts the peak by roughly the width of the window. Also cap the total: cap=60 seconds per wait, a fixed attempt budget, and for critical messages a wall-clock budget (an OTP older than a minute goes to fallback, not to attempt 6).
Implementation detail that matters: do not sleep() inside a worker. A worker sleeping for 30 seconds holds a slot and blocks its partition. Instead, write the message to a retry topic or the delay queue with a not_before timestamp and move on. Many teams use tiered retry topics (push.retry.30s, push.retry.5m, push.retry.1h) and a final dead-letter topic that someone actually looks at.
Step 10 — Provider rate limits and throttling
Every provider meters you, and the limits are per account, per sender, per destination, or all three:
- APNs does not publish a hard number but will throttle abusive senders; the practical ceiling is how many HTTP/2 connections and concurrent streams you open. Keep a pool of long-lived connections per worker rather than reconnecting per message; TLS handshakes are the real cost.
- FCM enforces per-project quotas (on the order of hundreds of thousands of messages per minute by default, raisable on request) and its old multi-message batch endpoint has been retired, so throughput comes from many concurrent HTTP/2 requests.
- SMS throughput is set by the sender type and the country: a single long code may be limited to around one message per second, a short code or a registered high-trust sender to tens or hundreds. US A2P traffic must be registered, and unregistered traffic is filtered.
- Email providers start new accounts at a low sending rate and raise it as reputation builds, and receiving mailbox providers throttle you too: blast 5 million messages at one mailbox provider from a cold IP and most will be deferred or junked. New IPs are "warmed" over weeks.
So each worker pool enforces a token bucket per provider account (and, for email, per receiving domain), shared across all workers through Redis or a central rate-limit service, the same design as the rate limiter lesson. The twist here is priority: a slice of the quota is reserved for the critical lane.
class TokenBucket:
"""rate tokens/second refill, burst = bucket size. Time is passed in, so the
class is deterministic and testable; production reads a monotonic clock."""
def __init__(self, rate, burst):
self.rate, self.burst = rate, burst
self.tokens, self.last = float(burst), 0.0
def try_take(self, now, n=1):
self.tokens = min(self.burst, self.tokens + (now - self.last) * self.rate)
self.last = now
if self.tokens >= n:
self.tokens -= n
return True
return False
class PriorityThrottle:
"""One provider quota shared by two lanes. The OTP lane owns a reserved slice
that marketing can never spend, so a 50M-user campaign cannot starve logins."""
def __init__(self, provider_rate, reserved_for_critical, burst_seconds=0.1):
shared_rate = provider_rate - reserved_for_critical
# Small bursts: vendors meter per second, so a full second of burst on
# top of steady refill would briefly exceed the contract.
self.critical = TokenBucket(reserved_for_critical,
reserved_for_critical * burst_seconds)
self.shared = TokenBucket(shared_rate, shared_rate * burst_seconds)
def admit(self, lane, now):
if lane == "critical":
# critical may use its own slice first, then borrow from shared
return self.critical.try_take(now) or self.shared.try_take(now)
return self.shared.try_take(now)
# SMS vendor contract: 100 msg/s. Reserve 20/s for OTPs.
t = PriorityThrottle(provider_rate=100, reserved_for_critical=20)
sent = {"critical": 0, "bulk": 0}
deferred = {"critical": 0, "bulk": 0}
# Simulate 5 seconds at 10 ms ticks: bulk offers 300/s, OTP offers 15/s.
for tick in range(500):
now = tick / 100
offers = ["bulk"] * 3 + (["critical"] if tick % 7 == 0 else [])
for lane in offers:
if t.admit(lane, now):
sent[lane] += 1
else:
deferred[lane] += 1 # bulk: back on the queue; critical: page someone
for lane in ("critical", "bulk"):
print(f"{lane:8s} sent {sent[lane]:4d} deferred {deferred[lane]:4d}")
print(f"total throughput {sum(sent.values()) / 5:.0f}/s (contract: 100/s)")critical sent 72 deferred 0 bulk sent 407 deferred 1093 total throughput 96/s (contract: 100/s)
Read the result: 300 bulk messages per second were offered against an 80/s share, and 1,093 were deferred (not dropped: they stay on the queue). Every OTP went out. Total throughput stayed just under the contract. The trade-off is that the reserved 20/s slice sits partly idle when there are few OTPs; a refinement lets bulk borrow unused critical tokens down to a floor of, say, 10/s kept permanently free.
When the bucket is empty, a bulk worker should stop pulling, not pull and requeue. Pulling and requeuing burns CPU and reorders messages. Pausing the consumer is how you turn a rate limit into backpressure that propagates up to the campaign pacer.
Step 11 — Quiet hours and time zones
Quiet hours are a per-user rule evaluated in the user's local time, so the router needs the user's IANA time zone (Asia/Kolkata, not "+05:30", because offsets change with daylight saving). Source it from the device (sent at registration), fall back to the account profile, then to the country of the phone number, then to UTC.
from datetime import datetime, timedelta, timezone, time
# Production code uses zoneinfo.ZoneInfo("America/New_York") so DST is handled.
# Fixed offsets keep this demo self-contained.
TZ = {"Mumbai": timezone(timedelta(hours=5, minutes=30)),
"New York": timezone(timedelta(hours=-4)),
"Tokyo": timezone(timedelta(hours=9))}
QUIET_START, QUIET_END = time(21, 0), time(8, 0) # 9 pm to 8 am local
def in_quiet_hours(local_t):
# The window wraps midnight, so it is "after start OR before end".
return local_t >= QUIET_START or local_t < QUIET_END
def release_time(now_utc, tz, category):
"""Return the UTC instant this notification may be sent."""
if category in ("security", "transactional"):
return now_utc # OTPs and order updates never wait
local = now_utc.astimezone(tz)
if not in_quiet_hours(local.time()):
return now_utc
release = local.replace(hour=QUIET_END.hour, minute=0, second=0, microsecond=0)
if local.time() >= QUIET_START: # evening: release tomorrow morning
release += timedelta(days=1)
return release.astimezone(timezone.utc)
campaign_start = datetime(2026, 9, 23, 14, 30, tzinfo=timezone.utc)
for city, tz in TZ.items():
local = campaign_start.astimezone(tz)
r = release_time(campaign_start, tz, "marketing")
wait = (r - campaign_start).total_seconds() / 3600
print(f"{city:9s} local {local:%H:%M} marketing -> send at "
f"{r.astimezone(tz):%a %H:%M} local ({wait:4.1f} h delay)")
otp = release_time(campaign_start, TZ["Tokyo"], "security")
print(f"Tokyo OTP at 23:30 local -> sent immediately: {otp == campaign_start}")Mumbai local 20:00 marketing -> send at Wed 20:00 local ( 0.0 h delay) New York local 10:30 marketing -> send at Wed 10:30 local ( 0.0 h delay) Tokyo local 23:30 marketing -> send at Thu 08:00 local ( 8.5 h delay) Tokyo OTP at 23:30 local -> sent immediately: True
- Deferred messages go to the delay queue with a release timestamp, not back into Kafka to be re-checked every minute.
- Spread the release. If 8 million users in one time zone are all released at exactly 08:00, you have built a thundering herd. Add a random offset of up to 30–60 minutes to each release time.
- Re-check on release. A message parked for 9 hours must pass preferences, suppression and
expires_atagain when it wakes. The user may have unsubscribed overnight, and "flash sale ends at midnight" is useless at 8 a.m. - "Send at 9 a.m. local" for a campaign is the same machinery at audience scale: the campaign service groups the audience by UTC offset and schedules roughly 24–38 waves (there are half-hour and 45-minute offsets), each paced independently.
- DST edge cases: a quiet window of 02:00–03:00 does not exist on spring-forward night, and happens twice in autumn. Using a real time-zone library and storing times as wall-clock plus zone, not as a UTC offset, avoids hand-written bugs.
- Legal quiet hours are not user preferences. US telemarketing rules restrict calls and marketing texts to 8 a.m.–9 p.m. in the recipient's local time, and several states are stricter. That rule applies even if the user never set quiet hours.
Step 12 — Preferences, unsubscribe and compliance
You are not expected to be a lawyer, but a senior candidate knows that these three laws shape the data model, and names them.
| Law | Scope | What it forces in the design |
|---|---|---|
| CAN-SPAM (US) | Commercial email | A working unsubscribe link in every marketing email, honoured within 10 business days (build for immediately); a physical postal address in the footer; no misleading From or subject lines. Transactional messages are exempt from the opt-out requirement, which is why category classification matters. |
| GDPR (EU/UK) and ePrivacy | All personal data of people in the EU; marketing consent | A lawful basis (usually opt-in consent for marketing, recorded with timestamp and source), consent as easy to withdraw as to give, the right to erasure (delete tokens, prefs and logs for a user on request, across the log's TTL too), data minimisation, and keeping EU users' data in-region if you promised to. |
| TCPA (US) | Calls and texts to phones | Prior express written consent before marketing texts, stored as evidence; honour STOP, UNSUBSCRIBE, QUIT and similar replies across every sender number; 8 a.m.–9 p.m. local-time sending window for marketing. Statutory damages are per message, so a buggy blast to 100k numbers is an existential lawsuit, not a bad day. |
Also expected by the big mailbox providers since 2024 for bulk senders: SPF, DKIM and DMARC alignment, a one-click unsubscribe header (List-Unsubscribe with List-Unsubscribe-Post), and a spam complaint rate kept well below 0.3%.
The resolution order in the router
For each (user, channel), evaluate in this order and stop at the first "no":
- Suppression list (hard bounce, spam complaint, SMS STOP, legal hold). Absolute.
- Category class. Security and legally required transactional messages skip steps 3–5. Everything else continues.
- Global unsubscribe for marketing.
- Category × channel preference (sparse rows over category defaults).
- Consent evidence where the law needs it (TCPA SMS consent timestamp, GDPR marketing opt-in).
- Quiet hours and frequency caps. These defer or divert to in-app, they do not drop.
Two failure-mode decisions hide here. If the preference cache is unavailable, fail closed for marketing (do not send) and fail open for critical (send the OTP). And unsubscribe writes go to the source of truth and invalidate the cache synchronously, because "I unsubscribed and got another email five minutes later" is both a legal and a trust failure.
Step 13 — Templating and localisation
template: order_picked_up version: 7 category: order_updates
push / en title: "Your order is on the way"
body: "{{courier_name}} picked it up. Arriving in {{eta_minutes}} min."
push / hi title: "आपका ऑर्डर रास्ते में है"
body: "{{courier_name}} ने ऑर्डर उठा लिया है। {{eta_minutes}} मिनट में पहुँचेगा।"
sms / en body: "Order on the way with {{courier_name}}, ~{{eta_minutes}} min. -Acme"
email/ en subject + HTML body + plain-text part + footer partial (address, unsubscribe)
locale fallback: pt-BR -> pt -> en # never render an empty message
plurals (ICU): "{count, plural, one {# new like} other {# new likes}}"- Render in the router, per recipient, with the user's locale, not in the calling service. A campaign to 50M users in 30 languages is one API call.
- Validate variables at the API against
required_vars, so a missing{{eta_minutes}}is a 400 to the caller now, not "Arriving in min." on a million lock screens later. - Escape per channel. Variables are HTML-escaped for email, left raw for push and SMS. User-provided values (a display name) must never be interpreted as template syntax.
- SMS length is money. A message over 160 GSM-7 characters is billed as multiple segments, and a single emoji or non-Latin character switches the whole message to UCS-2, where a segment is 70 characters. The Hindi SMS above costs two to three times the English one. Show segment count in the template editor.
- Push payloads are capped at around 4 KB, so long content becomes a short teaser plus a deep link into the app.
- Cache compiled templates in every router, keyed by
(name, version, channel, locale). Because versions are immutable, the cache never needs invalidation, only a lookup of which version is published.
Step 14 — Device tokens and invalid-token cleanup
A user has 0 to many tokens; they rotate, they die on uninstall, and dead tokens are not free: each one is a wasted request, and Apple and Google look unfavourably at senders who keep hammering dead tokens.
- Registration is an upsert keyed by token. The app registers on every launch (cheap) and on the OS token-refresh callback, which updates
last_seen_at. If the token already belongs to another user (shared device, logout then login), move it. - Logout deletes the token server-side. Otherwise the next person to log in on that phone gets the previous user's bank alerts.
- Synchronous invalidation from send responses: APNs answers 410
Unregistered(and gives a timestamp; ignore it if the token was re-registered after that time) or 400BadDeviceToken; FCM answersUNREGISTEREDor an invalid-argument error for a malformed token. The worker setsinvalidated_atimmediately, so the next send skips it. - Asynchronous pruning: a daily job removes tokens not seen in, say, 60–90 days. Firebase itself treats tokens inactive for about 270 days as expired; you should prune well before that.
- Environment mismatch is a common production bug: a debug build's token sent to the production APNs endpoint returns
BadDeviceToken. Storeapp_bundleand the APNs environment with the token. - Web push subscriptions answer 404 or 410 from the push service when the user revokes permission or clears site data; same handling.
Step 15 — Delivery tracking and analytics
QUEUED -> (SUPPRESSED | DEFERRED | EXPIRED) # decided by the router
QUEUED -> SENT -> DELIVERED -> OPENED -> CLICKED # happy path
SENT -> FAILED (bounce, invalid token, carrier reject)
SENT -> COMPLAINED (marked as spam) # -> suppression list
any -> UNSUBSCRIBED (from this message) # attribution for marketingWhere each status actually comes from, and how much to trust it:
- SENT is the provider's 2xx. For APNs and FCM that is all you get from the provider. It means "accepted", not "seen".
- DELIVERED for push requires your app to report a receipt (a notification service extension on iOS, a message handler on Android) back to a lightweight endpoint. Batch these on the client; 600M receipts a day is its own ingestion pipeline.
- DELIVERED for SMS and email comes from provider webhooks (carrier delivery receipts, mail server acceptance), minutes later, sometimes never.
- OPENED for email is a tracking pixel and is unreliable: privacy features in some mail clients prefetch images and inflate opens, and many clients block images. Treat clicks as the real engagement signal.
- CLICKED is a redirect through your link-tracking domain with a signed id, which also lets you attribute a conversion to a campaign.
The pipeline: workers and webhook ingest emit status events to Kafka; one consumer updates the per-message row (for support lookups), another aggregates into a columnar warehouse by (campaign, template_version, channel, locale, hour). Webhooks are joined back via provider_msg_id, and they arrive out of order (a DELIVERED can arrive after an OPENED), so status updates only move forward in the state order and never regress.
Two feedback loops close the system: hard bounces and complaints write the suppression list, and invalid-token responses clean device_tokens. A notification system that does not listen to its providers slowly ruins its own deliverability.
Step 16 — Fan-out for a 50M-user campaign
The campaign is decomposed into 50,000 small, independently retryable chunk jobs. Pacing happens at the source, so pause, cancel and backpressure all work by simply not producing more chunks.
Why it is built this way
- Snapshot, then chunk. A chunk of 1,000 users is a unit of work that takes a second or two, can be retried alone, and is recorded in
campaign_chunks. A crashed router loses one chunk's progress, not the campaign's. Resume means "produce the chunks not marked done". - Bulk lookups per chunk. Loading preferences and tokens for 1,000 users in one multi-get is ~1,000x fewer round trips than per-user lookups. This is where most of the campaign's database load goes, so it hits replicas and caches, never the primary.
- Pace at the source. The campaign service produces chunks at
max_rate_per_sec, so Kafka never holds 70M messages at once, and "pause" takes effect in seconds. A design that dumps 70M messages into a queue and then tries to throttle the consumers has no way to cancel a bad campaign. - A kill switch that actually kills. The most expensive incident in this system is a campaign sent to the wrong segment or with a broken template. Cancel must stop new chunks, and workers must check campaign status (cached for a few seconds) before sending, so in-flight messages are discarded too.
- Canary first. Send to 1% of the audience, wait 10 minutes, check bounce, complaint and click rates automatically, then release the rest. Mistakes caught at 500k are much cheaper than at 50M.
- Social fan-out is different from campaigns. "A creator with 20M followers posted" should not become 20M pushes. Use the hybrid approach from the news feed lesson: push only to followers who are recently active and opted in, badge the rest, and let them see it on next open.
Scaling each layer
- API: stateless, behind a load balancer; its only dependencies are Redis (idempotency) and Kafka. Scales horizontally to any request rate we have.
- Kafka: partition by
user_idfor per-user order. Enough partitions (hundreds) that router and worker pools can scale out; one consumer per partition is the ceiling on parallelism. - Router: CPU-bound on rendering and preference evaluation; autoscale on consumer lag of
requests.bulk, keep a fixed floor forrequests.critical. - Workers: I/O-bound. A push worker with async HTTP/2 can keep thousands of requests in flight; a few dozen workers cover 40k/s. SMS and email workers are bounded by provider quotas, not by us.
- Preferences and tokens: read-mostly, cached in Redis with write-through invalidation; ~52 GB of tokens replicates comfortably.
- Log: Cassandra-style cluster sized on write throughput (~17k rows/s average, ~100k/s during campaigns) with TTL-based expiry, so no delete jobs.
Step 17 — Failure modes
| What fails | What users see | Mechanism |
|---|---|---|
| Primary SMS vendor down or slow | OTPs arrive from a different sender number | Circuit breaker per vendor (open after N failures or p99 over budget in a window), automatic failover to vendor B, route per country to whichever vendor has the best recent delivery rate |
| APNs or FCM partially down | Pushes delayed; critical messages fall back to SMS if configured | Retry with backoff to the retry topics; expires_at drops stale ones; fallback channel for critical categories only |
| Email provider degraded | Emails late | Second ESP configured with its own warmed IPs and DKIM keys; switching cold is not an option, so keep a few percent of traffic flowing to it always |
| Kafka consumer lag builds up | Bulk late; critical unaffected | Separate topics and consumer groups per lane; autoscale routers; alert on lag age for critical within a minute |
| Worker crashes mid-send | Rare duplicate | Redelivery plus dedupe key; collapse ids on push reduce visible duplicates |
| Poison message (template renders to an error) | That message not sent | Catch, record FAILED, move to dead-letter after one attempt; never block the partition |
| Bad campaign (wrong segment, broken copy) | Some users get a wrong message | Canary release, automated guardrail metrics, cancel that stops in-flight sends, approval step for audiences above a threshold |
| Preference cache down | Marketing paused, OTPs flow | Fail closed for bulk, fail open for critical; fall back to the database with a concurrency limit |
| Redis dedupe store lost | Possible duplicates for a short window | Critical keeps flowing (duplicates preferred over losses); bulk pauses until the store is back |
| Sender reputation damage | Emails start landing in spam | Separate transactional and marketing IPs and domains; auto-pause campaigns if complaint rate spikes |
| SMS pumping / toll fraud | Huge SMS bill from bots requesting OTPs to premium numbers | Rate limit OTP requests per phone, per IP, per country prefix; block high-risk destinations; CAPTCHA on repeated requests; alert on spend per hour |
| A region lost | Brief delay | API and workers active in two regions; Kafka mirrored; providers are global so any region can send. Dedupe keys are regional, so accept a small duplicate risk during failover. |
SMS vendor failover in detail
Keep two or three vendors live all the time, each carrying some share of traffic, so their routes, sender registrations and credentials are known to work. A router in the SMS worker scores vendors per destination country on recent success rate, latency and cost, and shifts traffic away from a degrading vendor gradually rather than flipping all at once. The circuit breaker opens on errors or timeouts, half-opens after 30 seconds with a trickle of probe traffic, and closes when probes succeed. Failover for OTPs is immediate on timeout; for marketing it is not worth the duplicate risk, so bulk messages simply wait. Sender ids differ per vendor in many countries, so the OTP copy should not say "from 55123"; it should name the brand.
Step 18 — Observability
- The SLO that matters: critical lane, time from API accept to provider accept, p99 under 2 s. Measure it per channel and per vendor, and page on it.
- Business signals: OTP conversion (codes sent versus codes entered) is the best end-to-end health check an SMS pipeline has. A vendor can return 200 for every message while delivering none; only the conversion drop shows it.
- Per-provider: success rate by error class, latency, throttle events, circuit-breaker state, invalid-token rate.
- Queue health: consumer lag in seconds (age of the oldest message), not in message count, per topic.
- Campaign guardrails: bounce rate, complaint rate, unsubscribe rate and uninstall rate per campaign, compared to baseline, with automatic pause thresholds.
- Cost: SMS spend per hour per country, with alerts. It is both a budget control and the fastest fraud detector you have.
- Traceability:
notification_idcarried from API through Kafka headers to the provider's client reference and back through webhooks, so support can answer "why didn't I get my code" in one lookup.
The trade-off summary — how to close
- Separate lanes, not a priority field. Critical and bulk get separate topics, workers and reserved provider quota. We pay with some idle capacity in the critical lane; we get OTP latency that is independent of marketing.
- At-least-once plus dedupe, not exactly-once. Duplicates are possible in a narrow crash window; we choose resend for critical and skip for bulk, and use collapse ids to hide duplicates on devices.
- Async API returning 202. Producers never wait on APNs. We pay with a status endpoint and webhooks instead of a synchronous "sent".
- Throttle at the source and apply backpressure. Campaigns are paced chunk jobs, so pause and cancel work. We pay with a slower, deliberate campaign rather than "as fast as possible".
- Wide-column log with a TTL, warehouse for aggregates. 159 TB/year never touches the relational database; per-message detail expires after 90 days.
- Preferences can be cached; suppression is checked fresh. Stale preferences cost a little annoyance; a stale suppression list costs a lawsuit.
- Multiple vendors kept warm. More integration and contract overhead, in exchange for surviving any single provider's outage.
- What I would build first: API, one Kafka topic per lane, a router, push and email workers, preferences and suppression, the log. Campaign chunking, digests, multi-vendor SMS and time-zone waves are added as volume and regulation demand them.
// Write your solution here
// Write your solution here
// Write your solution here
Finished reading? Mark this lesson complete to track your progress.
