System Design · Lesson 17 of 18

Notification System

Push, SMS, email fan-out, templates, preferences, retries and dedupe.

  • Advanced
  • 55 min read
  • 6 objectives

Before this lessonLesson 16: Typeahead and Nearby Search

What you will learn

  • Size a multi-channel notification platform, including 50M-user campaign blasts
  • Isolate OTPs from marketing with per-channel, per-priority queues and reserved quota
  • Make sends effectively-once with idempotency keys, dedupe and backoff with jitter
  • Respect provider rate limits with token buckets and fail over between SMS vendors
  • Enforce preferences, quiet hours and CAN-SPAM, GDPR and TCPA rules
  • Track delivery, clean up dead tokens and collapse noisy bursts into digests

Your Progress

0 of 18 lessons 0%

  • Lessons0 / 18
  • Completed0
  • Est. time left~ 16 hours

Progress is saved in this browser. No account needed.

Tip: pressing Next marks this lesson complete automatically.

Design a notification platform: one internal service that every product team calls to reach a user on iOS push, Android push, web push, SMS, email or the in-app inbox. It has to send a login code in under two seconds, tell 50 million people about a sale without melting the SMS vendor, never wake someone at 3 a.m. for a "like", honour every unsubscribe, and not send the same message twice when a worker crashes halfway.

It looks like the easiest prompt in the course ("just call the provider's API"), which is exactly why it is asked. The interesting part is not calling APNs. It is everything around the call:

  • Two workloads that share nothing but the pipes. Transactional traffic (OTPs, order updates, security alerts) is low volume, latency critical and legally allowed to reach anyone. Marketing traffic is enormous, bursty, latency tolerant and wrapped in consent law. Putting both in one queue is the classic mistake.
  • You do not own the last hop. APNs, FCM, Twilio and SES have their own rate limits, outages, error codes and pricing. Your system is a well-behaved client of several unreliable, metered third parties.
  • "Exactly once" is impossible, and the interviewer knows it. The provider call is a side effect outside your transaction. The question is how close you get and what you do about the gap.

If you have done the chat lesson, you have seen the narrow version of this: "the recipient is offline, so hand the message to the phone vendor's push service". This lesson is the whole platform behind that one arrow.

Step 1 — Clarify the requirements

The channels, and why they are not interchangeable

Say this table out loud early. Every later decision (queues, retries, cost, compliance) differs per row.

ChannelAddressLatencyCost per sendWhat "delivered" meansQuirks
iOS push (APNs)Device token per app installSub-second, usuallyFreeOnly "Apple accepted it"; no receipt~4 KB payload, HTTP/2, token dies on uninstall (HTTP 410)
Android push (FCM)Registration tokenSub-second, usuallyFree"Google accepted it"; optional delivery data in aggregate~4 KB payload, per-project quotas, Doze mode delays low-priority messages
Web pushBrowser subscription (endpoint URL + keys)SecondsFreePush service accepted itPayload encrypted per subscription (VAPID), endpoint differs per browser vendor
SMSE.164 phone number1–10 sReal money: roughly $0.005–$0.10, far more in some countriesCarrier delivery receipt (DLR), sometimes160 GSM-7 chars per segment (70 if any emoji), carrier filtering, sender registration, toll fraud
EmailAddressSeconds to minutesFractions of a centAccepted by the receiving mail server; opens are guessesSender reputation, SPF/DKIM/DMARC, bounces, spam complaints, one-click unsubscribe
In-app inboxUser idInstant if the app is openYour own storageYou know exactly: it is your databaseThe only channel you fully control, and the fallback for everything else

Functional, in scope for v1

  • Product services send a notification by naming a template and a recipient (or an audience), not by passing a rendered string.
  • Channels: APNs, FCM, web push, SMS, email, in-app inbox. A single request may fan out to several channels, with fallback ("push; if no device, SMS").
  • Two priority classes: critical/transactional (OTP, password reset, payment failed, order status) and bulk (marketing, digests, social activity).
  • User preferences per category and channel, global unsubscribe, quiet hours in the user's local time zone.
  • Templates with versions and localisation; marketing can edit copy without a deploy.
  • Campaigns: send to a segment of up to 50M users, schedulable, pausable, cancellable.
  • Delivery tracking: per-notification status plus aggregate analytics (sent, delivered, opened, clicked, bounced, unsubscribed).
  • Collapse and digest: "Asha and 6 others liked your photo", not seven pushes.

Explicitly out of scope

Segment building and audience analytics (we accept a segment id from a marketing tool), A/B test statistics, the in-app message designer, inbound SMS conversations beyond STOP/HELP keywords, voice calls, and the recommendation model that decides what to send. Naming these shows you know where the boundary is.

Non-functional

  • Scale: 100M daily active users, around 1.5 billion notifications a day across channels, plus campaign blasts of 50M recipients.
  • Latency: critical lane: accepted by the provider within 2 s p99 of the API call. Bulk lane: minutes are fine, and a campaign completes within 30–60 minutes.
  • Isolation: a campaign running at full speed must not add a single second to an OTP. This is the non-functional requirement that shapes the architecture.
  • Delivery semantics: at-least-once inside our system, deduplicated so the user sees each notification once in all but rare crash windows. Never silently drop a critical message.
  • Availability: 99.99% for the send API. Survive the loss of any single provider.
  • Compliance: CAN-SPAM, GDPR and TCPA (details in Step 12). An unsubscribe must be honoured on the very next send, not "eventually".
  • Cost: SMS is the one line item that shows up in board meetings. Prefer free channels and never send an SMS we did not need to.

Step 2 — Back-of-the-envelope estimation

DAU                = 100_000_000
SECONDS_PER_DAY    = 86_400

# --- Transactional + social: triggered by something a user or system did ---
per_user_per_day = {"push": 6, "in_app": 8, "email": 0.5, "sms": 0.05}
for ch, n in per_user_per_day.items():
    avg = DAU * n / SECONDS_PER_DAY
    print(f"{ch:7s} {DAU * n / 1e6:7,.0f}M/day  {avg:9,.0f}/s avg  {avg * 5:9,.0f}/s peak")

# --- OTPs: tiny volume, the strictest latency budget in the system ---
otp_per_day = 3_000_000
print(f"OTP     {otp_per_day / SECONDS_PER_DAY:,.0f}/s avg -- p99 target: handed to SMS vendor < 2 s")

# --- One marketing campaign blast ---
audience       = 50_000_000
devices_each   = 1.4                     # phones + tablets + browsers
window_minutes = 30                      # "everyone within half an hour"
sends = audience * devices_each
print(f"campaign: {sends / 1e6:.0f}M pushes in {window_minutes} min = "
      f"{sends / (window_minutes * 60):,.0f}/s")

# --- Storage: the notification log is the big table ---
log_row_bytes = 300                      # ids, channel, status, timestamps, provider id
rows_per_day  = DAU * sum(per_user_per_day.values())
log_tb_year   = rows_per_day * 365 * log_row_bytes / 1e12
print(f"log: {rows_per_day / 1e9:.2f}B rows/day, {log_tb_year:,.0f} TB/year before TTL")

# --- Device tokens: small, fits in memory on a few nodes ---
tokens = DAU * 1.4 * 1.5                # include monthly-active, not just daily
print(f"device tokens: {tokens / 1e6:.0f}M x 250 B = {tokens * 250 / 1e9:.0f} GB")
Output
push        600M/day      6,944/s avg     34,722/s peak
in_app      800M/day      9,259/s avg     46,296/s peak
email        50M/day        579/s avg      2,894/s peak
sms           5M/day         58/s avg        289/s peak
OTP     35/s avg -- p99 target: handed to SMS vendor < 2 s
campaign: 70M pushes in 30 min = 38,889/s
log: 1.46B rows/day, 159 TB/year before TTL
device tokens: 210M x 250 B = 52 GB

What the numbers actually decide

  • Steady-state push is ~7k/s, peak ~35k/s. That is a modest Kafka cluster and a few dozen push workers holding HTTP/2 connections. Not the hard part.
  • A single campaign roughly doubles peak push load for half an hour (39k/s on top of 35k/s). If that traffic shares a queue with OTPs, logins stall every time marketing presses "send". Separate lanes, not a priority field.
  • OTPs are 35/s. The critical lane is tiny, so it can be over-provisioned cheaply: dedicated workers, dedicated provider connections, a reserved slice of every vendor quota.
  • SMS at 289/s peak is a contract number, not a compute number. Vendors and carriers throttle per sender, so the bottleneck is how many sender numbers and vendor accounts you hold, not how many servers you run.
  • 1.5 billion log rows a day, 159 TB a year is a write-heavy, append-only, time-ordered, rarely-updated table. That is a wide-column store (Cassandra, ScyllaDB, DynamoDB) with a TTL, not Postgres. Aggregates go to a columnar warehouse.
  • Device tokens are only ~52 GB. They fit in a replicated key-value store with a cache in front. Nobody needs to shard this cleverly.

Step 3 — API design

An internal API for product services, plus a small user-facing API for preferences and devices. Auth for internal callers is service identity (mTLS or signed service tokens), and every caller has its own quota so one buggy team cannot flood users.

## Send (internal, called by product services)

POST /v1/notifications
  Headers: Idempotency-Key: order-88231-state-PICKED_UP   (required)
  {
    "template": "order_picked_up",          # name, not rendered text
    "recipient": { "user_id": 42 },         # or { "email": ... } for non-users
    "data": { "courier_name": "Ravi", "eta_minutes": 12 },
    "channels": ["push", "in_app"],          # optional; template has defaults
    "fallback": ["sms"],                     # only if no channel above worked
    "priority": "critical",                  # critical | bulk
    "category": "order_updates",             # drives preferences + compliance
    "collapse_key": "order-88231",           # newer replaces older on device
    "expires_at": "2026-09-23T14:45:00Z"     # stale after this: drop, do not send
  }
  -> 202 { notification_id: "n_01J9...", status: "QUEUED" }
     200 { same body }        # replay of the same Idempotency-Key
     400 UNKNOWN_TEMPLATE | 400 MISSING_TEMPLATE_VARIABLE
     409 IDEMPOTENCY_KEY_REUSED   # same key, different body
     422 CATEGORY_NOT_ALLOWED_FOR_CALLER
     429 + Retry-After            # per-caller quota
  # 202, not 201 or 200: the notification is accepted, not sent. Sending is async.

GET /v1/notifications/{id}
  -> 200 { id, status, attempts: [{ channel, provider, status, at, error? }] }

## Campaigns (internal, called by the marketing tool)

POST /v1/campaigns
  { "template": "autumn_sale", "segment_id": "seg_551",
    "channels": ["push", "email"], "send_at": "2026-10-01T09:00:00",
    "send_at_mode": "recipient_local_time",       # or "absolute"
    "max_rate_per_sec": 20000 }
  -> 201 { campaign_id, status: "SCHEDULED", estimated_recipients: 50000000 }
POST /v1/campaigns/{id}/pause | /resume | /cancel  -> 200 | 409 ALREADY_COMPLETED
GET  /v1/campaigns/{id}   -> 200 { status, sent, suppressed, failed, progress_pct }

## Templates (internal, versioned, never edited in place)

POST /v1/templates/{name}/versions
  { "channel": "push", "locale": "en", "title": "Your order is on the way",
    "body": "{{courier_name}} picked it up. Arriving in {{eta_minutes}} min." }
  -> 201 { version: 7 }       # publish is a separate step, so drafts are safe
POST /v1/templates/{name}/versions/7/publish -> 200

## User-facing: devices and preferences

POST   /v1/me/devices   { platform: "ios", token: "...", app_version, locale, tz }
  -> 201 | 200 (same token re-registered: update last_seen, do not duplicate)
DELETE /v1/me/devices/{token} -> 204           # called on logout
GET    /v1/me/preferences
  -> 200 { quiet_hours: { start: "21:00", end: "08:00", tz: "Asia/Kolkata" },
           categories: { marketing: { push: false, email: true, sms: false },
                         social:    { push: true,  email: false } } }
PATCH  /v1/me/preferences   { categories: { marketing: { email: false } } } -> 200

## Unsubscribe (public, from the email footer, no login)

GET  /u/{signed_token}           -> 200 confirmation page
POST /u/{signed_token}           -> 200    # RFC 8058 one-click, List-Unsubscribe-Post
  # signed_token encodes user_id + category + expiry, HMAC-signed, so it cannot be
  # forged to unsubscribe someone else and needs no password.

## Provider webhooks (inbound)

POST /hooks/ses | /hooks/twilio | /hooks/sendgrid
  # bounces, complaints, delivery receipts, STOP replies. Verify the signature,
  # enqueue, return 200 fast. Never do work inline: providers retry slow hooks.

API decisions worth defending

  • Callers send a template name and data, never a rendered string. Copy changes, translations and compliance footers happen in one place. It also means the platform knows the category, which is what the law cares about.
  • Idempotency-Key is required and chosen by the caller from its own domain (order-88231-state-PICKED_UP), not a random UUID. A domain key survives the caller's own retries and its own crashes; a random UUID generated per attempt dedupes nothing.
  • 202 Accepted. The API returns once the request is durably queued. Anything slower couples every product service's latency to APNs.
  • expires_at is on every message. "Your driver is arriving" delivered 40 minutes late after an outage is worse than nothing. Workers drop expired messages instead of sending them.
  • Priority and category are validated against the caller. The marketing service may not mark a promo as critical to skip quiet hours, and the API returns 422 if it tries. Otherwise every team eventually does exactly that.

Step 4 — Data model

Five kinds of data with five different access patterns, so they do not all go in one database.

-- =========================================================
-- DEVICE TOKENS  (key-value / Postgres, ~200M rows, read on every push)
-- =========================================================
device_tokens (
  token          TEXT PRIMARY KEY,      -- APNs/FCM token or web-push endpoint
  user_id        BIGINT NOT NULL,
  platform       TEXT NOT NULL,         -- ios | android | web
  app_bundle     TEXT NOT NULL,         -- prod vs beta apps have different APNs topics
  locale         TEXT, tz TEXT,         -- captured from the device, used as defaults
  last_seen_at   TIMESTAMPTZ NOT NULL,  -- refreshed on app open; drives stale pruning
  invalidated_at TIMESTAMPTZ NULL       -- set on 410 / UNREGISTERED, deleted later
);
CREATE INDEX ON device_tokens (user_id) WHERE invalidated_at IS NULL;
-- Token is the PK, not (user_id, token): one physical device can be handed to a
-- new user after logout, and the token must move with it, not exist twice.

-- =========================================================
-- PREFERENCES + SUPPRESSION  (Postgres source of truth, Redis cache)
-- =========================================================
user_channel_prefs (
  user_id    BIGINT NOT NULL,
  category   TEXT NOT NULL,             -- marketing | social | order_updates | security
  channel    TEXT NOT NULL,             -- push | email | sms | in_app
  enabled    BOOLEAN NOT NULL,
  source     TEXT NOT NULL,             -- settings_page | email_footer | sms_stop | import
  updated_at TIMESTAMPTZ NOT NULL,      -- consent evidence: when and how
  PRIMARY KEY (user_id, category, channel)
);
-- Sparse: only rows that differ from the category default are stored.

user_settings (
  user_id BIGINT PRIMARY KEY,
  quiet_start TIME, quiet_end TIME, tz TEXT NOT NULL DEFAULT 'UTC',
  locale TEXT NOT NULL DEFAULT 'en',
  sms_marketing_consent_at TIMESTAMPTZ NULL   -- TCPA: express written consent record
);

suppression_list (                      -- beats every preference, checked last-mile
  address    TEXT NOT NULL,             -- email or E.164 phone, normalised
  channel    TEXT NOT NULL,
  reason     TEXT NOT NULL,             -- hard_bounce | complaint | sms_stop | legal
  created_at TIMESTAMPTZ NOT NULL,
  PRIMARY KEY (address, channel)
);
-- Keyed by ADDRESS, not user_id: a bounced address stays suppressed even if a
-- new account signs up with it tomorrow.

-- =========================================================
-- TEMPLATES  (Postgres, tiny, cached in every worker)
-- =========================================================
templates (
  name TEXT, version INT, channel TEXT, locale TEXT,
  category TEXT NOT NULL,               -- a template has exactly one category
  subject TEXT, title TEXT, body TEXT NOT NULL,
  required_vars TEXT[] NOT NULL,        -- validated at the API, not at send time
  status TEXT NOT NULL,                 -- draft | published | retired
  PRIMARY KEY (name, version, channel, locale)
);
-- Immutable versions: the log records (name, version) so you can always show
-- exactly what a user received, even after the copy changed.

-- =========================================================
-- NOTIFICATION LOG  (Cassandra / ScyllaDB, 1.5B rows/day, TTL 90 days)
-- =========================================================
CREATE TABLE notifications_by_user (
  user_id         bigint,
  day             date,                 -- bucket keeps partitions bounded
  notification_id timeuuid,             -- time-sortable
  template        text, template_version int,
  category        text, channel text, provider text,
  status          text,                 -- QUEUED SENT DELIVERED OPENED FAILED SUPPRESSED
  provider_msg_id text,                 -- to join webhooks back to this row
  error           text,
  PRIMARY KEY ((user_id, day), notification_id)
) WITH CLUSTERING ORDER BY (notification_id DESC)
  AND default_time_to_live = 7776000;  -- 90 days; analytics keeps aggregates
-- Second table keyed by provider_msg_id so a Twilio webhook can find its row
-- without scanning. Wide-column stores want one table per query.

-- =========================================================
-- IN-APP INBOX  (same store, partitioned by user, like a chat inbox)
-- =========================================================
CREATE TABLE inbox (
  user_id bigint, notification_id timeuuid,
  title text, body text, deep_link text, read_at timestamp,
  PRIMARY KEY (user_id, notification_id)
) WITH CLUSTERING ORDER BY (notification_id DESC);
-- Unread badge count kept as a Redis counter, not COUNT(*).

-- =========================================================
-- DEDUPE + COLLAPSE  (Redis, ephemeral)
-- =========================================================
SET   dedupe:{event_id}:{user_id}:{channel}  1  NX EX 86400
SET   idem:{caller}:{idempotency_key}  '{"id":"n_01J9..","hash":".."}' NX EX 172800
HINCRBY digest:{user_id}:like:photo:9  count 1        -- open collapse window
EXPIRE  digest:{user_id}:like:photo:9  60
INCR    cap:{user_id}:push:2026092314                  -- per-user hourly cap

-- =========================================================
-- CAMPAIGNS  (Postgres; the audience itself is a file in object storage)
-- =========================================================
campaigns (id, template, segment_id, status, send_at, send_at_mode,
           max_rate_per_sec, audience_uri, total, created_by, created_at);
campaign_chunks (campaign_id, chunk_no, status, attempts,     -- 1 row per 1,000 users
                 PRIMARY KEY (campaign_id, chunk_no));

Modelling decisions to explain

  • The suppression list is keyed by address and checked at the last possible moment, right before the provider call. Preferences can be cached and slightly stale; suppression is the legal backstop and is checked fresh.
  • Preferences store source and updated_at. When a regulator or a lawsuit asks "why did you text this person", you need the consent record: when, how, from which screen. This is a compliance table wearing a settings-table costume.
  • The log lives in a wide-column store with a TTL, partitioned by (user_id, day) so no partition grows without bound. Candidates who put 1.5B rows a day in Postgres spend the rest of the interview on vacuum and partition management.
  • Templates are immutable versions. Editing in place means the log cannot tell you what was actually sent, and a typo "fix" can break a campaign halfway through.
  • Dedupe keys have a TTL. They only need to outlive the redelivery window (Kafka retention, retry budget). A day is plenty; forever is a memory leak.
  • The campaign audience is a snapshot file, not a live query. Recipients resolved at 09:00 do not change because someone signed up at 09:05, and a resumed campaign sends to exactly the remaining users.

Step 5 — High-level design

flowchart LR SVC[Product services] --> API[Notification API
auth, validate, idempotency] MKT[Marketing tool] --> CAMP[Campaign service] API --> K1[(Kafka: requests.critical)] API --> K2[(Kafka: requests.bulk)] CAMP --> OBJ[(Audience snapshot
object storage)] CAMP --> K2 K1 --> ROUTER[Router / fan-out workers] K2 --> ROUTER ROUTER --> PREF[(Prefs + settings
Redis cache)] ROUTER --> DEV[(Device tokens)] ROUTER --> TPL[Template renderer] ROUTER --> SCHED[(Delay queue
quiet hours, digests)] SCHED --> ROUTER ROUTER --> QP1[(push.critical)] ROUTER --> QP2[(push.bulk)] ROUTER --> QS1[(sms.critical)] ROUTER --> QS2[(sms.bulk)] ROUTER --> QE[(email.*)] ROUTER --> INBOX[(In-app inbox)] QP1 --> PW[Push workers] QP2 --> PW QS1 --> SW[SMS workers] QS2 --> SW QE --> EW[Email workers] PW --> APNS[APNs] PW --> FCM[FCM] PW --> WP[Web push services] SW --> SV1[SMS vendor A] SW --> SV2[SMS vendor B] EW --> ESP[Email provider] PW --> LOG[(Notification log)] SW --> LOG EW --> LOG HOOK[Webhook ingest] --> LOG SV1 --> HOOK ESP --> HOOK LOG --> WH[(Analytics warehouse)]

Two hops of queues. The first splits by priority so a campaign never sits in front of an OTP. The router resolves who, where and whether, then writes to per-channel, per-priority queues so each provider's workers scale, throttle and fail independently.

What each component owns

  • Notification API — authenticates the calling service, enforces per-caller quotas, validates the template and its variables, claims the idempotency key, writes to the right Kafka topic, returns 202. It never talks to a provider.
  • Campaign service — resolves a segment to an audience snapshot, splits it into chunks, schedules waves (including per-time-zone waves), paces them into requests.bulk, and owns pause and cancel.
  • Router — the brain. For each request: expand the recipient to devices and addresses, apply preferences and suppression, apply quiet hours (park in the delay queue), dedupe, collapse into digests, render the template per channel and locale, choose channels and fallbacks, and enqueue one message per (device or address).
  • Delay queue — holds messages until a time: quiet-hours releases, digest window closes, retries with long backoff. A Redis sorted set scored by release time, or a timer service. Not a sleeping thread.
  • Channel workers — dumb and fast. Pull a message, check expires_at and the suppression list, take a token from the provider's bucket, call the provider, classify the response, retry or record. One worker pool per channel, with separate consumer groups for critical and bulk.
  • Webhook ingest — receives bounces, complaints, delivery receipts and STOP replies, verifies signatures, and turns them into log updates, suppressions and token invalidations.
  • Notification log and warehouse — per-message status for support ("did the user get the reset email?") and aggregates for marketing ("what was the open rate?").

Why two queue hops rather than one? The first hop is cheap and uniform (a request), the second is channel-specific (one APNs payload). Fan-out between them can be 1 to 5 messages per request, and for a campaign chunk, 1 to 1,400. Keeping them separate lets the router scale on CPU (rendering, preference lookups) and the workers scale on provider throughput.

Step 6 — Priority lanes: OTP versus marketing

The most common wrong answer is "put a priority field on the message and have workers pick high-priority first". It fails in three ways:

  1. Kafka partitions are FIFO. There is no "pick the high-priority one" within a partition. An OTP enqueued behind 2 million campaign messages waits for all of them.
  2. Even with a priority queue, shared workers are a shared bottleneck. If all push workers are blocked on slow APNs calls for the campaign, there is nobody free to take the OTP.
  3. Shared provider quota. If the campaign spends the entire SMS vendor quota, the OTP gets a 429 from the vendor no matter how fast your queue is.

The fix is isolation at every layer:

LayerCritical laneBulk lane
Kafka topicsrequests.critical, sms.critical, push.criticalrequests.bulk, sms.bulk, push.bulk
WorkersDedicated pool, over-provisioned 5–10x, never autoscaled down to zeroAutoscaled on consumer lag, can be paused
Provider quotaReserved slice (e.g. 20% of SMS throughput) that bulk can never spendThe rest, throttled to protect sender reputation
Provider accountsOften a separate sender id / short code for OTPsMarketing sender, marketing IP pool for email
Quiet hours, digests, capsBypass all of themSubject to all of them
Retry budgetShort (a few seconds of total retry, then fail over or fall back); an OTP after 2 minutes is uselessLong (hours), with deep backoff
Failure alertingPages someone in minutesA dashboard and a morning ticket

Separate IP pools and sender ids for transactional email matter more than they look: if a marketing blast gets flagged as spam and damages the marketing IPs' reputation, password resets still land in the inbox.

Step 7 — Walk an OTP end to end

sequenceDiagram participant A as Auth service participant N as Notification API participant R as Redis participant K as Kafka critical participant W as SMS worker participant V as SMS vendor A participant V2 as SMS vendor B participant H as Webhook ingest A->>N: POST /notifications template=login_otp, Idempotency-Key=otp-req-5521 N->>R: SET idem key NX (claim) N->>K: produce to requests.critical, key=user_id N-->>A: 202 notification_id K->>W: router has resolved phone, rendered text, enqueued sms.critical W->>W: expires_at ok, suppression ok, take token from reserved bucket W->>V: send(to, body, client_ref=notification_id) V--xW: timeout after 1.5 s W->>W: vendor A circuit counts a failure W->>V2: failover send(to, body, client_ref=notification_id) V2-->>W: 202 accepted, vendor_msg_id W->>R: log SENT via vendor B V2->>H: delivery receipt DELIVERED H->>R: update log row by vendor_msg_id

The critical path from API to vendor is two queue hops and one HTTP call, typically well under a second. A slow vendor is abandoned quickly for a second vendor because an OTP that arrives late is a failed login.

The order of operations, and why

  1. Claim the idempotency key before producing. If the auth service retries its request after a timeout, the second call returns the same notification_id instead of sending a second code. If the key claim succeeds but the produce fails, the API releases the key (or marks it failed) so the caller's retry can proceed.
  2. Partition by user_id. All of one user's notifications land on one partition in order, so "your code is 4412" is never overtaken by an earlier, now-invalid code.
  3. Check expires_at in the worker, not only at the API. An OTP typically expires in 5–10 minutes. If the queue backed up during an incident, sending codes that are already invalid wastes money and confuses users.
  4. Failover, not blind retry, on a timeout. A timeout is an unknown outcome: vendor A might have sent it. For an OTP a rare duplicate SMS is acceptable and a missing one is not, so we fail over. For a marketing SMS the trade flips: we wait and retry the same vendor, or drop.
  5. Pass our notification_id as the vendor's client reference. Delivery receipts come back with it, which is how the webhook finds the log row, and some vendors use it to reject duplicates.
  6. The user can always ask again. The login screen offers "resend code" after 30 seconds and "call me instead" after two tries. Product design is part of the reliability budget.

Step 8 — Idempotency, dedupe and digests

Where duplicates come from

  • The caller retries after a timeout on its request to us. Solved by the Idempotency-Key at the API.
  • Kafka redelivers because a worker processed a message but crashed before committing its offset. Solved by a dedupe key checked in the worker.
  • The same logical event is produced twice by an upstream bug (two services both notify on "order delivered"). Solved by a dedupe key built from the event, not from the request.
  • The provider accepted it, then we crashed before recording it. Not fully solvable. See below.

So there are two keys at two layers. The API key is (caller, Idempotency-Key) and protects against request retries. The worker key is (event_id, user_id, channel) and protects against redelivery. The worker key is set with SET NX before the provider call; if the call then fails with a retryable error, the worker deletes the key (or, better, stores a state: SENDING / SENT) so the retry is allowed.

import hashlib

class DedupeStore:
    """Stands in for Redis: SET key 1 NX EX ttl. Returns True only for the first writer."""
    def __init__(self): self.keys = {}
    def set_nx(self, key, now, ttl):
        exp = self.keys.get(key)
        if exp is not None and exp > now:
            return False
        self.keys[key] = now + ttl
        return True

def dedupe_key(event_id, user_id, channel):
    # Keyed on the PRODUCER's event id, not on the text. Two different OTPs with
    # the same wording must both send; the same OTP delivered twice by Kafka must not.
    return hashlib.sha1(f"{event_id}:{user_id}:{channel}".encode()).hexdigest()[:16]

class Digester:
    """Collapse bursts: the first like sends now; later likes inside the window are
    folded into one summary sent when the window closes."""
    def __init__(self, window):
        self.window, self.open = window, {}      # (user, type, object) -> state
    def offer(self, user, ntype, obj, actor, now):
        k = (user, ntype, obj)
        st = self.open.get(k)
        if st is None or now >= st["closes"]:
            self.open[k] = {"closes": now + self.window, "actors": []}
            return f"SEND NOW  {actor} liked your photo"
        st["actors"].append(actor)
        return None
    def flush(self, now):
        out = []
        for k, st in list(self.open.items()):
            if now >= st["closes"] and st["actors"]:
                a = st["actors"]
                more = f" and {len(a) - 1} others" if len(a) > 1 else ""
                out.append(f"DIGEST    {a[0]}{more} liked your photo")
                del self.open[k]
        return out

store = DedupeStore()
# Kafka redelivers event e-77 after a consumer crash: same id, three times.
for delivery in range(3):
    k = dedupe_key("e-77", 42, "push")
    print(f"delivery {delivery}: {'SEND' if store.set_nx(k, now=delivery, ttl=86_400) else 'DROP duplicate'}")

d = Digester(window=60)
names = ["Asha", "Ben", "Chen", "Dev", "Eli", "Fay", "Gus", "Hana"]
for i, who in enumerate(names):
    r = d.offer(user=42, ntype="like", obj="photo:9", actor=who, now=i * 5)
    if r: print(f"t={i*5:3d}s {r}")
for line in d.flush(now=61):
    print(f"t= 61s {line}")
Output
delivery 0: SEND
delivery 1: DROP duplicate
delivery 2: DROP duplicate
t=  0s SEND NOW  Asha liked your photo
t= 61s DIGEST    Ben and 6 others liked your photo

The gap you should admit

Between "APNs returned 200" and "we wrote SENT", the worker can die. On restart the message is redelivered, the dedupe key says SENDING, and we must choose: send again (risk a duplicate) or skip (risk a loss). The honest answer is at-least-once with dedupe gives effectively-once in practice, with a tiny duplicate window, and you pick the direction per category: critical messages resend, bulk messages skip. You can narrow the window further with provider features: apns-collapse-id and FCM's collapse_key make a second copy replace the first on the device rather than appear twice, and a stable email Message-ID lets some clients thread duplicates together. Claiming true exactly-once across a third-party HTTP API is the wrong answer.

Collapse, digests and per-user caps

  • Collapse on device: order status pushes share collapse_key = order-88231, so the lock screen shows the latest state, not five stale ones.
  • Digest in the router: the first event in a window sends immediately (feels live); later events inside the window fold into one summary when it closes. The window key is (user, type, object), stored in Redis with a TTL, and the flush is a delay-queue entry, not a timer in a worker's memory.
  • Daily or weekly email digests are the same idea with a longer window: accumulate social events into the user's digest row and send once at their local 9 a.m.
  • Per-user frequency caps: at most N bulk pushes per user per hour and M per day (INCR cap:{user}:push:{hour}). Overflow goes to the in-app inbox only. Uninstall rate is the metric that punishes you for skipping this.

Step 9 — Retries: exponential backoff with jitter

A retry policy is three decisions: which errors to retry, how long to wait, and when to stop. Getting the first one wrong is worse than having no retries at all.

  • Retry: 429, 500, 502, 503, 504, connection resets, timeouts. The provider might succeed next time.
  • Do not retry: 400 (bad payload, will fail forever), 403 (our credentials or certificate are wrong: page an engineer), 410 / UNREGISTERED (the token is dead: invalidate it), invalid phone number, hard bounce.
  • Obey Retry-After when a provider sends it. It is the provider telling you its recovery plan.
import random

# How each provider response should be treated. This table is the policy.
RETRYABLE = {429, 500, 502, 503, 504, "timeout", "conn_reset"}
PERMANENT = {400, 403, 404, 410}      # 410 from APNs / UNREGISTERED from FCM: token is dead

def backoff_full_jitter(attempt, base=0.5, cap=60.0, rng=random):
    """AWS-style 'full jitter': sleep a random amount in [0, min(cap, base * 2^attempt)].
    The randomness is the point -- it spreads 1M retrying workers across the window
    instead of having them all hit the recovering provider at the same instant."""
    return rng.uniform(0, min(cap, base * (2 ** attempt)))

def send_with_retry(send, max_attempts=6, retry_after=None, rng=random, log=print):
    for attempt in range(max_attempts):
        status = send(attempt)
        if status == 200:
            log(f"  attempt {attempt}: 200 -> delivered")
            return "DELIVERED"
        if status in PERMANENT:
            log(f"  attempt {attempt}: {status} -> permanent, do not retry")
            return "FAILED_PERMANENT"
        if status in RETRYABLE:
            if attempt == max_attempts - 1:
                log(f"  attempt {attempt}: {status} -> retry budget exhausted")
                break
            wait = backoff_full_jitter(attempt, rng=rng)
            if status == 429 and retry_after:
                wait = max(wait, retry_after)     # the provider told us; obey it
            log(f"  attempt {attempt}: {status} -> retry in {wait:5.2f}s")
            continue
        log(f"  attempt {attempt}: {status} -> unknown, treat as permanent")
        return "FAILED_PERMANENT"
    return "DEAD_LETTER"          # parked for inspection, never silently dropped

rng = random.Random(7)            # seeded so the output is reproducible

print("provider flaps, then recovers:")
script = [503, 503, "timeout", 200]
print(send_with_retry(lambda a: script[a], rng=rng))

print("token was uninstalled:")
print(send_with_retry(lambda a: 410, rng=rng))

print("rate-limited with Retry-After: 5")
script = [429, 200]
print(send_with_retry(lambda a: script[a], retry_after=5, rng=rng))

print("provider down for the whole budget:")
print(send_with_retry(lambda a: 503, rng=rng))

# Why jitter matters: 10,000 workers all failed at t=0 and retry attempt 3.
no_jitter = [0.5 * 2 ** 3] * 10_000
jittered  = [backoff_full_jitter(3, rng=rng) for _ in range(10_000)]
def busiest_second(waits):
    buckets = {}
    for w in waits:
        buckets[int(w)] = buckets.get(int(w), 0) + 1
    return max(buckets.values())
print(f"peak retries in one second -- no jitter: {busiest_second(no_jitter):,}, "
      f"full jitter: {busiest_second(jittered):,}")
Output
provider flaps, then recovers:
  attempt 0: 503 -> retry in  0.16s
  attempt 1: 503 -> retry in  0.15s
  attempt 2: timeout -> retry in  1.30s
  attempt 3: 200 -> delivered
DELIVERED
token was uninstalled:
  attempt 0: 410 -> permanent, do not retry
FAILED_PERMANENT
rate-limited with Retry-After: 5
  attempt 0: 429 -> retry in  5.00s
  attempt 1: 200 -> delivered
DELIVERED
provider down for the whole budget:
  attempt 0: 503 -> retry in  0.27s
  attempt 1: 503 -> retry in  0.37s
  attempt 2: 503 -> retry in  0.12s
  attempt 3: 503 -> retry in  2.03s
  attempt 4: 503 -> retry in  0.30s
  attempt 5: 503 -> retry budget exhausted
DEAD_LETTER
peak retries in one second -- no jitter: 10,000, full jitter: 2,517

Why full jitter? Plain exponential backoff means every worker that failed at the same moment retries at the same moment, so a provider recovering from an outage is hit by a synchronised wave and falls over again. Randomising within the window cuts the peak by roughly the width of the window. Also cap the total: cap=60 seconds per wait, a fixed attempt budget, and for critical messages a wall-clock budget (an OTP older than a minute goes to fallback, not to attempt 6).

Implementation detail that matters: do not sleep() inside a worker. A worker sleeping for 30 seconds holds a slot and blocks its partition. Instead, write the message to a retry topic or the delay queue with a not_before timestamp and move on. Many teams use tiered retry topics (push.retry.30s, push.retry.5m, push.retry.1h) and a final dead-letter topic that someone actually looks at.

Step 10 — Provider rate limits and throttling

Every provider meters you, and the limits are per account, per sender, per destination, or all three:

  • APNs does not publish a hard number but will throttle abusive senders; the practical ceiling is how many HTTP/2 connections and concurrent streams you open. Keep a pool of long-lived connections per worker rather than reconnecting per message; TLS handshakes are the real cost.
  • FCM enforces per-project quotas (on the order of hundreds of thousands of messages per minute by default, raisable on request) and its old multi-message batch endpoint has been retired, so throughput comes from many concurrent HTTP/2 requests.
  • SMS throughput is set by the sender type and the country: a single long code may be limited to around one message per second, a short code or a registered high-trust sender to tens or hundreds. US A2P traffic must be registered, and unregistered traffic is filtered.
  • Email providers start new accounts at a low sending rate and raise it as reputation builds, and receiving mailbox providers throttle you too: blast 5 million messages at one mailbox provider from a cold IP and most will be deferred or junked. New IPs are "warmed" over weeks.

So each worker pool enforces a token bucket per provider account (and, for email, per receiving domain), shared across all workers through Redis or a central rate-limit service, the same design as the rate limiter lesson. The twist here is priority: a slice of the quota is reserved for the critical lane.

class TokenBucket:
    """rate tokens/second refill, burst = bucket size. Time is passed in, so the
    class is deterministic and testable; production reads a monotonic clock."""
    def __init__(self, rate, burst):
        self.rate, self.burst = rate, burst
        self.tokens, self.last = float(burst), 0.0

    def try_take(self, now, n=1):
        self.tokens = min(self.burst, self.tokens + (now - self.last) * self.rate)
        self.last = now
        if self.tokens >= n:
            self.tokens -= n
            return True
        return False

class PriorityThrottle:
    """One provider quota shared by two lanes. The OTP lane owns a reserved slice
    that marketing can never spend, so a 50M-user campaign cannot starve logins."""
    def __init__(self, provider_rate, reserved_for_critical, burst_seconds=0.1):
        shared_rate = provider_rate - reserved_for_critical
        # Small bursts: vendors meter per second, so a full second of burst on
        # top of steady refill would briefly exceed the contract.
        self.critical = TokenBucket(reserved_for_critical,
                                    reserved_for_critical * burst_seconds)
        self.shared   = TokenBucket(shared_rate, shared_rate * burst_seconds)

    def admit(self, lane, now):
        if lane == "critical":
            # critical may use its own slice first, then borrow from shared
            return self.critical.try_take(now) or self.shared.try_take(now)
        return self.shared.try_take(now)

# SMS vendor contract: 100 msg/s. Reserve 20/s for OTPs.
t = PriorityThrottle(provider_rate=100, reserved_for_critical=20)
sent = {"critical": 0, "bulk": 0}
deferred = {"critical": 0, "bulk": 0}

# Simulate 5 seconds at 10 ms ticks: bulk offers 300/s, OTP offers 15/s.
for tick in range(500):
    now = tick / 100
    offers = ["bulk"] * 3 + (["critical"] if tick % 7 == 0 else [])
    for lane in offers:
        if t.admit(lane, now):
            sent[lane] += 1
        else:
            deferred[lane] += 1   # bulk: back on the queue; critical: page someone

for lane in ("critical", "bulk"):
    print(f"{lane:8s} sent {sent[lane]:4d}  deferred {deferred[lane]:4d}")
print(f"total throughput {sum(sent.values()) / 5:.0f}/s (contract: 100/s)")
Output
critical sent   72  deferred    0
bulk     sent  407  deferred 1093
total throughput 96/s (contract: 100/s)

Read the result: 300 bulk messages per second were offered against an 80/s share, and 1,093 were deferred (not dropped: they stay on the queue). Every OTP went out. Total throughput stayed just under the contract. The trade-off is that the reserved 20/s slice sits partly idle when there are few OTPs; a refinement lets bulk borrow unused critical tokens down to a floor of, say, 10/s kept permanently free.

When the bucket is empty, a bulk worker should stop pulling, not pull and requeue. Pulling and requeuing burns CPU and reorders messages. Pausing the consumer is how you turn a rate limit into backpressure that propagates up to the campaign pacer.

Step 11 — Quiet hours and time zones

Quiet hours are a per-user rule evaluated in the user's local time, so the router needs the user's IANA time zone (Asia/Kolkata, not "+05:30", because offsets change with daylight saving). Source it from the device (sent at registration), fall back to the account profile, then to the country of the phone number, then to UTC.

from datetime import datetime, timedelta, timezone, time

# Production code uses zoneinfo.ZoneInfo("America/New_York") so DST is handled.
# Fixed offsets keep this demo self-contained.
TZ = {"Mumbai": timezone(timedelta(hours=5, minutes=30)),
      "New York": timezone(timedelta(hours=-4)),
      "Tokyo": timezone(timedelta(hours=9))}

QUIET_START, QUIET_END = time(21, 0), time(8, 0)   # 9 pm to 8 am local

def in_quiet_hours(local_t):
    # The window wraps midnight, so it is "after start OR before end".
    return local_t >= QUIET_START or local_t < QUIET_END

def release_time(now_utc, tz, category):
    """Return the UTC instant this notification may be sent."""
    if category in ("security", "transactional"):
        return now_utc                        # OTPs and order updates never wait
    local = now_utc.astimezone(tz)
    if not in_quiet_hours(local.time()):
        return now_utc
    release = local.replace(hour=QUIET_END.hour, minute=0, second=0, microsecond=0)
    if local.time() >= QUIET_START:           # evening: release tomorrow morning
        release += timedelta(days=1)
    return release.astimezone(timezone.utc)

campaign_start = datetime(2026, 9, 23, 14, 30, tzinfo=timezone.utc)
for city, tz in TZ.items():
    local = campaign_start.astimezone(tz)
    r = release_time(campaign_start, tz, "marketing")
    wait = (r - campaign_start).total_seconds() / 3600
    print(f"{city:9s} local {local:%H:%M}  marketing -> send at "
          f"{r.astimezone(tz):%a %H:%M} local ({wait:4.1f} h delay)")

otp = release_time(campaign_start, TZ["Tokyo"], "security")
print(f"Tokyo OTP at 23:30 local -> sent immediately: {otp == campaign_start}")
Output
Mumbai    local 20:00  marketing -> send at Wed 20:00 local ( 0.0 h delay)
New York  local 10:30  marketing -> send at Wed 10:30 local ( 0.0 h delay)
Tokyo     local 23:30  marketing -> send at Thu 08:00 local ( 8.5 h delay)
Tokyo OTP at 23:30 local -> sent immediately: True
  • Deferred messages go to the delay queue with a release timestamp, not back into Kafka to be re-checked every minute.
  • Spread the release. If 8 million users in one time zone are all released at exactly 08:00, you have built a thundering herd. Add a random offset of up to 30–60 minutes to each release time.
  • Re-check on release. A message parked for 9 hours must pass preferences, suppression and expires_at again when it wakes. The user may have unsubscribed overnight, and "flash sale ends at midnight" is useless at 8 a.m.
  • "Send at 9 a.m. local" for a campaign is the same machinery at audience scale: the campaign service groups the audience by UTC offset and schedules roughly 24–38 waves (there are half-hour and 45-minute offsets), each paced independently.
  • DST edge cases: a quiet window of 02:00–03:00 does not exist on spring-forward night, and happens twice in autumn. Using a real time-zone library and storing times as wall-clock plus zone, not as a UTC offset, avoids hand-written bugs.
  • Legal quiet hours are not user preferences. US telemarketing rules restrict calls and marketing texts to 8 a.m.–9 p.m. in the recipient's local time, and several states are stricter. That rule applies even if the user never set quiet hours.

Step 12 — Preferences, unsubscribe and compliance

You are not expected to be a lawyer, but a senior candidate knows that these three laws shape the data model, and names them.

LawScopeWhat it forces in the design
CAN-SPAM (US)Commercial emailA working unsubscribe link in every marketing email, honoured within 10 business days (build for immediately); a physical postal address in the footer; no misleading From or subject lines. Transactional messages are exempt from the opt-out requirement, which is why category classification matters.
GDPR (EU/UK) and ePrivacyAll personal data of people in the EU; marketing consentA lawful basis (usually opt-in consent for marketing, recorded with timestamp and source), consent as easy to withdraw as to give, the right to erasure (delete tokens, prefs and logs for a user on request, across the log's TTL too), data minimisation, and keeping EU users' data in-region if you promised to.
TCPA (US)Calls and texts to phonesPrior express written consent before marketing texts, stored as evidence; honour STOP, UNSUBSCRIBE, QUIT and similar replies across every sender number; 8 a.m.–9 p.m. local-time sending window for marketing. Statutory damages are per message, so a buggy blast to 100k numbers is an existential lawsuit, not a bad day.

Also expected by the big mailbox providers since 2024 for bulk senders: SPF, DKIM and DMARC alignment, a one-click unsubscribe header (List-Unsubscribe with List-Unsubscribe-Post), and a spam complaint rate kept well below 0.3%.

The resolution order in the router

For each (user, channel), evaluate in this order and stop at the first "no":

  1. Suppression list (hard bounce, spam complaint, SMS STOP, legal hold). Absolute.
  2. Category class. Security and legally required transactional messages skip steps 3–5. Everything else continues.
  3. Global unsubscribe for marketing.
  4. Category × channel preference (sparse rows over category defaults).
  5. Consent evidence where the law needs it (TCPA SMS consent timestamp, GDPR marketing opt-in).
  6. Quiet hours and frequency caps. These defer or divert to in-app, they do not drop.

Two failure-mode decisions hide here. If the preference cache is unavailable, fail closed for marketing (do not send) and fail open for critical (send the OTP). And unsubscribe writes go to the source of truth and invalidate the cache synchronously, because "I unsubscribed and got another email five minutes later" is both a legal and a trust failure.

Step 13 — Templating and localisation

template: order_picked_up   version: 7   category: order_updates

push / en      title: "Your order is on the way"
               body:  "{{courier_name}} picked it up. Arriving in {{eta_minutes}} min."
push / hi      title: "आपका ऑर्डर रास्ते में है"
               body:  "{{courier_name}} ने ऑर्डर उठा लिया है। {{eta_minutes}} मिनट में पहुँचेगा।"
sms  / en      body:  "Order on the way with {{courier_name}}, ~{{eta_minutes}} min. -Acme"
email/ en      subject + HTML body + plain-text part + footer partial (address, unsubscribe)

locale fallback: pt-BR -> pt -> en        # never render an empty message
plurals (ICU):   "{count, plural, one {# new like} other {# new likes}}"
  • Render in the router, per recipient, with the user's locale, not in the calling service. A campaign to 50M users in 30 languages is one API call.
  • Validate variables at the API against required_vars, so a missing {{eta_minutes}} is a 400 to the caller now, not "Arriving in min." on a million lock screens later.
  • Escape per channel. Variables are HTML-escaped for email, left raw for push and SMS. User-provided values (a display name) must never be interpreted as template syntax.
  • SMS length is money. A message over 160 GSM-7 characters is billed as multiple segments, and a single emoji or non-Latin character switches the whole message to UCS-2, where a segment is 70 characters. The Hindi SMS above costs two to three times the English one. Show segment count in the template editor.
  • Push payloads are capped at around 4 KB, so long content becomes a short teaser plus a deep link into the app.
  • Cache compiled templates in every router, keyed by (name, version, channel, locale). Because versions are immutable, the cache never needs invalidation, only a lookup of which version is published.

Step 14 — Device tokens and invalid-token cleanup

A user has 0 to many tokens; they rotate, they die on uninstall, and dead tokens are not free: each one is a wasted request, and Apple and Google look unfavourably at senders who keep hammering dead tokens.

  • Registration is an upsert keyed by token. The app registers on every launch (cheap) and on the OS token-refresh callback, which updates last_seen_at. If the token already belongs to another user (shared device, logout then login), move it.
  • Logout deletes the token server-side. Otherwise the next person to log in on that phone gets the previous user's bank alerts.
  • Synchronous invalidation from send responses: APNs answers 410 Unregistered (and gives a timestamp; ignore it if the token was re-registered after that time) or 400 BadDeviceToken; FCM answers UNREGISTERED or an invalid-argument error for a malformed token. The worker sets invalidated_at immediately, so the next send skips it.
  • Asynchronous pruning: a daily job removes tokens not seen in, say, 60–90 days. Firebase itself treats tokens inactive for about 270 days as expired; you should prune well before that.
  • Environment mismatch is a common production bug: a debug build's token sent to the production APNs endpoint returns BadDeviceToken. Store app_bundle and the APNs environment with the token.
  • Web push subscriptions answer 404 or 410 from the push service when the user revokes permission or clears site data; same handling.

Step 15 — Delivery tracking and analytics

QUEUED -> (SUPPRESSED | DEFERRED | EXPIRED)            # decided by the router
QUEUED -> SENT -> DELIVERED -> OPENED -> CLICKED       # happy path
          SENT -> FAILED (bounce, invalid token, carrier reject)
          SENT -> COMPLAINED (marked as spam)          # -> suppression list
          any  -> UNSUBSCRIBED (from this message)     # attribution for marketing

Where each status actually comes from, and how much to trust it:

  • SENT is the provider's 2xx. For APNs and FCM that is all you get from the provider. It means "accepted", not "seen".
  • DELIVERED for push requires your app to report a receipt (a notification service extension on iOS, a message handler on Android) back to a lightweight endpoint. Batch these on the client; 600M receipts a day is its own ingestion pipeline.
  • DELIVERED for SMS and email comes from provider webhooks (carrier delivery receipts, mail server acceptance), minutes later, sometimes never.
  • OPENED for email is a tracking pixel and is unreliable: privacy features in some mail clients prefetch images and inflate opens, and many clients block images. Treat clicks as the real engagement signal.
  • CLICKED is a redirect through your link-tracking domain with a signed id, which also lets you attribute a conversion to a campaign.

The pipeline: workers and webhook ingest emit status events to Kafka; one consumer updates the per-message row (for support lookups), another aggregates into a columnar warehouse by (campaign, template_version, channel, locale, hour). Webhooks are joined back via provider_msg_id, and they arrive out of order (a DELIVERED can arrive after an OPENED), so status updates only move forward in the state order and never regress.

Two feedback loops close the system: hard bounces and complaints write the suppression list, and invalid-token responses clean device_tokens. A notification system that does not listen to its providers slowly ruins its own deliverability.

Step 16 — Fan-out for a 50M-user campaign

sequenceDiagram participant M as Marketing tool participant C as Campaign service participant S as Segment store participant O as Object storage participant K as Kafka bulk participant R as Router workers participant Q as push.bulk and email.bulk participant W as Channel workers participant P as APNs, FCM, ESP M->>C: POST /campaigns segment=seg_551, send_at 09:00 local C->>S: export segment snapshot S-->>C: 50M user ids C->>O: write audience file, 50,000 chunks of 1,000 C->>C: group chunks into time-zone waves loop each wave, paced at max_rate_per_sec C->>K: produce chunk jobs (campaign_id, chunk_no) end K->>R: chunk job R->>O: read 1,000 user ids R->>R: bulk-load prefs, suppression, tokens. Drop opted-out users R->>Q: enqueue ~1,400 per-device messages R->>C: chunk done, counts sent and suppressed Q->>W: message W->>W: token bucket per provider, pause consumer if empty W->>P: send over pooled HTTP/2 connections P-->>W: 200, or 410 invalid token M->>C: POST /campaigns/id/pause C->>K: stop producing new chunks, workers drain in-flight only

The campaign is decomposed into 50,000 small, independently retryable chunk jobs. Pacing happens at the source, so pause, cancel and backpressure all work by simply not producing more chunks.

Why it is built this way

  • Snapshot, then chunk. A chunk of 1,000 users is a unit of work that takes a second or two, can be retried alone, and is recorded in campaign_chunks. A crashed router loses one chunk's progress, not the campaign's. Resume means "produce the chunks not marked done".
  • Bulk lookups per chunk. Loading preferences and tokens for 1,000 users in one multi-get is ~1,000x fewer round trips than per-user lookups. This is where most of the campaign's database load goes, so it hits replicas and caches, never the primary.
  • Pace at the source. The campaign service produces chunks at max_rate_per_sec, so Kafka never holds 70M messages at once, and "pause" takes effect in seconds. A design that dumps 70M messages into a queue and then tries to throttle the consumers has no way to cancel a bad campaign.
  • A kill switch that actually kills. The most expensive incident in this system is a campaign sent to the wrong segment or with a broken template. Cancel must stop new chunks, and workers must check campaign status (cached for a few seconds) before sending, so in-flight messages are discarded too.
  • Canary first. Send to 1% of the audience, wait 10 minutes, check bounce, complaint and click rates automatically, then release the rest. Mistakes caught at 500k are much cheaper than at 50M.
  • Social fan-out is different from campaigns. "A creator with 20M followers posted" should not become 20M pushes. Use the hybrid approach from the news feed lesson: push only to followers who are recently active and opted in, badge the rest, and let them see it on next open.

Scaling each layer

  • API: stateless, behind a load balancer; its only dependencies are Redis (idempotency) and Kafka. Scales horizontally to any request rate we have.
  • Kafka: partition by user_id for per-user order. Enough partitions (hundreds) that router and worker pools can scale out; one consumer per partition is the ceiling on parallelism.
  • Router: CPU-bound on rendering and preference evaluation; autoscale on consumer lag of requests.bulk, keep a fixed floor for requests.critical.
  • Workers: I/O-bound. A push worker with async HTTP/2 can keep thousands of requests in flight; a few dozen workers cover 40k/s. SMS and email workers are bounded by provider quotas, not by us.
  • Preferences and tokens: read-mostly, cached in Redis with write-through invalidation; ~52 GB of tokens replicates comfortably.
  • Log: Cassandra-style cluster sized on write throughput (~17k rows/s average, ~100k/s during campaigns) with TTL-based expiry, so no delete jobs.

Step 17 — Failure modes

What failsWhat users seeMechanism
Primary SMS vendor down or slowOTPs arrive from a different sender numberCircuit breaker per vendor (open after N failures or p99 over budget in a window), automatic failover to vendor B, route per country to whichever vendor has the best recent delivery rate
APNs or FCM partially downPushes delayed; critical messages fall back to SMS if configuredRetry with backoff to the retry topics; expires_at drops stale ones; fallback channel for critical categories only
Email provider degradedEmails lateSecond ESP configured with its own warmed IPs and DKIM keys; switching cold is not an option, so keep a few percent of traffic flowing to it always
Kafka consumer lag builds upBulk late; critical unaffectedSeparate topics and consumer groups per lane; autoscale routers; alert on lag age for critical within a minute
Worker crashes mid-sendRare duplicateRedelivery plus dedupe key; collapse ids on push reduce visible duplicates
Poison message (template renders to an error)That message not sentCatch, record FAILED, move to dead-letter after one attempt; never block the partition
Bad campaign (wrong segment, broken copy)Some users get a wrong messageCanary release, automated guardrail metrics, cancel that stops in-flight sends, approval step for audiences above a threshold
Preference cache downMarketing paused, OTPs flowFail closed for bulk, fail open for critical; fall back to the database with a concurrency limit
Redis dedupe store lostPossible duplicates for a short windowCritical keeps flowing (duplicates preferred over losses); bulk pauses until the store is back
Sender reputation damageEmails start landing in spamSeparate transactional and marketing IPs and domains; auto-pause campaigns if complaint rate spikes
SMS pumping / toll fraudHuge SMS bill from bots requesting OTPs to premium numbersRate limit OTP requests per phone, per IP, per country prefix; block high-risk destinations; CAPTCHA on repeated requests; alert on spend per hour
A region lostBrief delayAPI and workers active in two regions; Kafka mirrored; providers are global so any region can send. Dedupe keys are regional, so accept a small duplicate risk during failover.

SMS vendor failover in detail

Keep two or three vendors live all the time, each carrying some share of traffic, so their routes, sender registrations and credentials are known to work. A router in the SMS worker scores vendors per destination country on recent success rate, latency and cost, and shifts traffic away from a degrading vendor gradually rather than flipping all at once. The circuit breaker opens on errors or timeouts, half-opens after 30 seconds with a trickle of probe traffic, and closes when probes succeed. Failover for OTPs is immediate on timeout; for marketing it is not worth the duplicate risk, so bulk messages simply wait. Sender ids differ per vendor in many countries, so the OTP copy should not say "from 55123"; it should name the brand.

Step 18 — Observability

  • The SLO that matters: critical lane, time from API accept to provider accept, p99 under 2 s. Measure it per channel and per vendor, and page on it.
  • Business signals: OTP conversion (codes sent versus codes entered) is the best end-to-end health check an SMS pipeline has. A vendor can return 200 for every message while delivering none; only the conversion drop shows it.
  • Per-provider: success rate by error class, latency, throttle events, circuit-breaker state, invalid-token rate.
  • Queue health: consumer lag in seconds (age of the oldest message), not in message count, per topic.
  • Campaign guardrails: bounce rate, complaint rate, unsubscribe rate and uninstall rate per campaign, compared to baseline, with automatic pause thresholds.
  • Cost: SMS spend per hour per country, with alerts. It is both a budget control and the fastest fraud detector you have.
  • Traceability: notification_id carried from API through Kafka headers to the provider's client reference and back through webhooks, so support can answer "why didn't I get my code" in one lookup.

The trade-off summary — how to close

  • Separate lanes, not a priority field. Critical and bulk get separate topics, workers and reserved provider quota. We pay with some idle capacity in the critical lane; we get OTP latency that is independent of marketing.
  • At-least-once plus dedupe, not exactly-once. Duplicates are possible in a narrow crash window; we choose resend for critical and skip for bulk, and use collapse ids to hide duplicates on devices.
  • Async API returning 202. Producers never wait on APNs. We pay with a status endpoint and webhooks instead of a synchronous "sent".
  • Throttle at the source and apply backpressure. Campaigns are paced chunk jobs, so pause and cancel work. We pay with a slower, deliberate campaign rather than "as fast as possible".
  • Wide-column log with a TTL, warehouse for aggregates. 159 TB/year never touches the relational database; per-message detail expires after 90 days.
  • Preferences can be cached; suppression is checked fresh. Stale preferences cost a little annoyance; a stale suppression list costs a lawsuit.
  • Multiple vendors kept warm. More integration and contract overhead, in exchange for surviving any single provider's outage.
  • What I would build first: API, one Kafka topic per lane, a router, push and email workers, preferences and suppression, the log. Campaign chunking, digests, multi-vendor SMS and time-zone waves are added as volume and regulation demand them.
// Write your solution here
// Write your solution here
// Write your solution here

Finished reading? Mark this lesson complete to track your progress.

Up next · Lesson 18Capstone: Food Delivery (Swiggy / DoorDash)One real-world ask, every layer: requirements, HLD, LLD, API, schema, sharding, caching and failure.