System Design · Lesson 13 of 18

Video Streaming (YouTube)

Upload pipelines, transcoding, adaptive bitrate, CDN strategy and cost.

  • Advanced
  • 60 min read
  • 6 objectives

Before this lessonLesson 12: Chat System (WhatsApp / Messenger)

What you will learn

  • Size a video platform from egress, storage and transcode numbers
  • Design resumable chunked uploads with pre-signed URLs and deduplication
  • Build a parallel transcoding DAG into a bitrate ladder and HLS/DASH manifests
  • Explain and simulate adaptive bitrate in the player
  • Tier a CDN for popularity, the long tail and viral spikes
  • Count views, protect content with DRM and reason about cost

Your Progress

0 of 18 lessons 0%

  • Lessons0 / 18
  • Completed0
  • Est. time left~ 16 hours

Progress is saved in this browser. No account needed.

Tip: pressing Next marks this lesson complete automatically.

Design a video platform in the mould of YouTube or Netflix: creators upload files, the platform turns each one into something every device on every network can play, and hundreds of millions of viewers press play and expect the first frame in under two seconds, with no spinner for the rest of the video. Around that core sit search, view counts, comments, recommendations and content protection.

The single most useful framing for this interview is that you are designing two systems that barely touch, plus a small database between them:

  • An ingest and processing pipeline — resumable uploads, deduplication, transcoding into a bitrate ladder, packaging, thumbnails, captions. Low request rate (tens of uploads per second), enormous compute, latency measured in minutes, and correctness means "never lose a creator's upload".
  • A delivery system — manifests, segments, a multi-tier CDN, adaptive bitrate in the player. Millions of requests per second and tens of terabits per second of egress, latency measured in milliseconds, and correctness means "the spinner never appears".
  • A metadata and engagement layer in between — video rows, view counts, likes, comments, watch history, search and recommendations. Ordinary web-scale problems, except for the hot-key problem when one video goes viral.

Candidates who draw one box labelled "video service" that both accepts uploads and streams bytes have already lost the thread. The API servers in this design never touch a video byte in either direction.

Step 1 — Clarify the requirements

Functional, in scope

  • Upload: a creator uploads a file of up to 12 hours or 256 GB, from a phone on a flaky network or a studio on fibre. The upload survives disconnects and resumes where it stopped.
  • Process: the platform transcodes the file into several resolutions and codecs, generates thumbnails and auto-captions, and tells the creator when it is watchable.
  • Watch: viewers play any public video on web, mobile and TV, seek anywhere, and quality adapts to their connection. Playback resumes where they left off on another device.
  • Discover: search by title, description and tags; a home feed and "up next" recommendations (sketch only).
  • Engage: view counts, likes and dislikes, comments.
  • Protect: private and unlisted videos, and DRM for premium (Netflix-style) content.

Explicitly out of scope

Live streaming (a different latency model — mention it as a follow-up), ads insertion, creator monetisation and payouts, the copyright-claims workflow, channels and subscriptions beyond what the feed needs, the ML models themselves, and moderation tooling. Name each one so the interviewer knows you know it exists.

Non-functional, with numbers

  • Scale: 500M daily active viewers, 6 plays each per day; 5M uploads per day averaging 7 minutes.
  • Playback latency: time to first frame p50 under 1 s and p95 under 2 s; rebuffering under 0.5% of watch time. These are the metrics that correlate with people leaving.
  • Processing latency: a short video is watchable at low resolution within a minute or two of upload completing; full quality within tens of minutes even for long 4K files.
  • Durability: an accepted upload is never lost — eleven nines on the source file. This is stronger than for any rendition, because renditions can always be regenerated from the source.
  • Availability: playback 99.99%. Uploads and processing can tolerate a delayed queue; playback cannot tolerate an outage.
  • Consistency: eventual almost everywhere. A new video appearing in search a minute late is fine; view counts lagging by seconds is fine. The exceptions are privacy (a video set to private must stop playing quickly) and the creator seeing their own upload status accurately.

Step 2 — Back-of-the-envelope estimation

Do the delivery side first, because it dominates every cost and every architectural decision.

SEC_DAY = 86_400

# ---------- Assumptions (state them out loud) ----------
DAU                = 500_000_000
views_per_user     = 6            # plays started per user per day
avg_watch_s        = 6 * 60       # people abandon; watched time, not video length
avg_delivered_mbps = 2.5          # blend of phones at 480p and TVs at 1080p/4K
uploads_per_day    = 5_000_000
avg_upload_min     = 7
segment_s          = 4            # HLS/DASH segment duration

# ---------- Traffic ----------
views_day   = DAU * views_per_user
play_qps    = views_day / SEC_DAY
seg_rps     = views_day * (avg_watch_s / segment_s) / SEC_DAY
upload_rps  = uploads_per_day / SEC_DAY
print(f"play starts: {play_qps:,.0f}/s avg, {play_qps * 3:,.0f}/s peak")
print(f"segment GETs at the CDN: {seg_rps / 1e6:,.1f}M/s avg")
print(f"uploads: {upload_rps:,.0f}/s   read:write = {play_qps / upload_rps:,.0f}:1")

# ---------- Egress: the number that dominates everything ----------
egress_tbps = views_day * avg_watch_s * avg_delivered_mbps * 1e6 / SEC_DAY / 1e12
egress_pb_day = views_day * avg_watch_s * avg_delivered_mbps * 1e6 / 8 / 1e15
print(f"egress: {egress_tbps:,.0f} Tbps avg, ~{egress_tbps * 2:,.0f} Tbps at evening peak,"
      f" {egress_pb_day:,.0f} PB/day")

# ---------- Ingest ----------
upload_min_day = uploads_per_day * avg_upload_min
print(f"hours uploaded per minute: {upload_min_day / 60 / 1440:,.0f}")
source_mbps = 8                                           # typical phone/camera upload
ingest_gbps = upload_min_day * 60 * source_mbps / SEC_DAY / 1e3
print(f"ingest bandwidth: {ingest_gbps:,.0f} Gbps  (~{egress_tbps * 1e3 / ingest_gbps:,.0f}x less than egress)")

# ---------- Storage per day ----------
ladder_h264 = [0.3, 0.6, 1.2, 2.5, 5.0]                   # 240p..1080p, Mbps
def mb_per_min(mbps): return mbps * 60 / 8
per_min_source = mb_per_min(source_mbps)
per_min_h264   = sum(mb_per_min(b) for b in ladder_h264)
per_min_av1    = per_min_h264 * 0.6                        # ~40% smaller, same quality
tb = lambda mb: upload_min_day * mb / 1e6
print(f"per source minute: source {per_min_source:.0f} MB, H.264 ladder {per_min_h264:.0f} MB,"
      f" AV1 ladder {per_min_av1:.0f} MB")
print(f"new storage/day: source {tb(per_min_source):,.0f} TB + H.264 {tb(per_min_h264):,.0f} TB"
      f" (+ AV1 {tb(per_min_av1):,.0f} TB if encoded for everything)")
total_year_pb = (tb(per_min_source) + tb(per_min_h264)) * 365 / 1e3
print(f"one copy, one year: {total_year_pb:,.0f} PB  -> x replication/erasure overhead ~1.5")

# ---------- Transcode compute ----------
core_min_per_src_min_h264 = 4                              # full ladder, fast presets
cores = upload_min_day * core_min_per_src_min_h264 / 1440
print(f"transcode fleet (H.264 only): ~{cores:,.0f} cores busy around the clock")
Output
play starts: 34,722/s avg, 104,167/s peak
segment GETs at the CDN: 3.1M/s avg
uploads: 58/s   read:write = 600:1
egress: 31 Tbps avg, ~62 Tbps at evening peak, 338 PB/day
hours uploaded per minute: 405
ingest bandwidth: 194 Gbps  (~161x less than egress)
per source minute: source 60 MB, H.264 ladder 72 MB, AV1 ladder 43 MB
new storage/day: source 2,100 TB + H.264 2,520 TB (+ AV1 1,512 TB if encoded for everything)
one copy, one year: 1,686 PB  -> x replication/erasure overhead ~1.5
transcode fleet (H.264 only): ~97,222 cores busy around the clock

What the numbers actually decide

  • 31–62 Tbps of egress is the entire problem. No data centre serves that from one place, and no transit contract makes it affordable. This number alone forces a multi-tier CDN with caches embedded inside ISP networks, and makes egress price per GB the biggest line in the budget.
  • 3.1M segment requests per second are all static-file GETs. They must be served by caches that never call an application server — which is why the player fetches plain files named in a manifest, rather than calling a streamVideo(offset) API that returns bytes.
  • 35k–104k play starts per second is the only per-play load on the application tier: fetch metadata, authorise, return a manifest URL. That is a normal, cacheable web workload.
  • 58 uploads per second is small, but each one is gigabytes. The upload API handles metadata only; bytes go straight from the client to object storage using pre-signed URLs.
  • 4.6 PB of new storage per day and 1.7 EB per year means storage tiering is mandatory. Most of the catalogue is almost never watched, and should not sit on the same storage class as this week's hits.
  • ~100k cores of transcoding is a batch compute fleet that runs on cheap, pre-emptible capacity, which means every transcode task must be small, idempotent and restartable.
  • Read:write is 600:1 for plays, and far higher for segment reads. Every read path gets a cache; the write paths get queues.

Step 3 — API design

Three families of endpoint: the upload control plane, the playback control plane, and engagement. None of them carry video bytes.

## Upload control plane (bytes go to object storage, never through us)

POST /v1/uploads
  Headers: Idempotency-Key: <uuid>
  { filename, size_bytes, content_type, sha256?, title, description?, tags[],
    visibility: "private" | "unlisted" | "public", default_language }
  -> 201 { upload_id, video_id, part_size_bytes, part_count,
           expires_at }                  # upload session valid for 7 days
     409 DUPLICATE_SOURCE { existing_video_id }   # exact-hash dedupe hit
     413 FILE_TOO_LARGE | 429 UPLOAD_QUOTA_EXCEEDED

POST /v1/uploads/{upload_id}/parts:sign  { part_numbers: [1..50] }
  -> 200 { urls: { "1": "https://store/...&X-Signature=...", ... } }
     # pre-signed PUT URLs, 1 hour each; the client asks for more as it goes

PUT <pre-signed part URL>                 # straight to object storage
  body: bytes of that part   -> 200, ETag: "9b2cf..."

GET  /v1/uploads/{upload_id}              # used after a crash or app restart
  -> 200 { parts_received: [{number, etag, size}], state: "UPLOADING" }

POST /v1/uploads/{upload_id}/complete     { parts: [{number, etag}] }
  -> 202 { video_id, state: "PROCESSING" }   # accepted, not yet watchable
     400 PART_MISSING | 409 ALREADY_COMPLETED (returns the same body)

GET  /v1/videos/{video_id}/status
  -> 200 { state: "PROCESSING", playable_renditions: ["360p", "240p"],
           progress: 0.42 }             # plus a push notification when READY

## Playback control plane (small, cacheable JSON)

GET /v1/videos/{video_id}
  -> 200 { title, channel, duration_s, view_count, like_count, published_at,
           thumbnails: {...}, captions: [{lang, url}] }
     Cache-Control: public, max-age=30     # counts may be 30 s stale; fine
     404 | 410 REMOVED | 451 UNAVAILABLE_IN_REGION

POST /v1/videos/{video_id}/play           { device: {type, codecs[], max_height, drm[]} }
  -> 200 { manifest_url: "https://edge.../v/abc/master.m3u8?token=...",
           license_url?, resume_position_s: 734, session_id }
     403 PRIVATE_VIDEO | 403 NOT_ENTITLED   # the one place auth is enforced

PUT /v1/me/progress/{video_id}           { position_s, session_id }  -> 204
  # sent every ~10 s and on pause; this is how resume-on-another-device works

## Engagement and discovery

POST /v1/events/playback                  # batched QoE + view heartbeats
  { session_id, events: [{type: "heartbeat", position_s, rendition, buffer_s,
                          rebuffer_ms, ts}, ...] }   -> 204 (fire-and-forget)
PUT    /v1/videos/{video_id}/rating       { value: "like" | "dislike" | "none" } -> 204
GET    /v1/videos/{video_id}/comments?sort=top&cursor=  -> 200 { items, next_cursor }
POST   /v1/videos/{video_id}/comments     { text, parent_id? }  -> 201
GET    /v1/search?q=&lang=&duration=&cursor=  -> 200 { results: [VideoCard], next_cursor }
GET    /v1/feed/home?cursor=              -> 200 { items: [VideoCard], next_cursor }

API decisions worth defending

  • The server never proxies bytes. A pre-signed URL is a time-limited, single-object permission signed with the storage service's key. The client talks to object storage directly, which scales without us and keeps gigabytes out of API server memory and sockets.
  • Upload is a session, not a request. A session id, numbered parts and a "what have you got" endpoint make resume trivial. A single POST with a 6 GB body fails at 97% on a train and starts again from zero.
  • complete returns 202, not 201. The video exists but is not watchable; the state field and a push notification tell the creator when it is.
  • There is no streamVideo(offset, codec, resolution) call. Older designs sketch an API that returns a byte stream from an offset. In practice the client declares its capabilities once, receives a manifest, and then fetches static segments from the CDN, choosing codec and resolution itself; seeking to an offset is just requesting a different segment number. The server-side part of "resume from offset" is the progress endpoint.
  • Playback authorisation happens once, at /play, and is carried to the CDN as a signed token. The edge validates a signature locally in microseconds; it never calls back to an auth service per segment.
  • View and QoE events are batched and fire-and-forget. The player must never block playback on analytics.

Step 4 — Data model

Five kinds of data, five storage choices. The video metadata is the smallest and the one people spend the most time on; the bytes are the biggest and the simplest.

-- =========================================================
-- VIDEO METADATA  (sharded MySQL/Vitess or Spanner, key = video_id)
-- =========================================================
videos (
  video_id        BIGINT PRIMARY KEY,     -- Snowflake-style (lesson 7); public id is
                                          -- an 11-char base64url encoding of it
  channel_id      BIGINT NOT NULL,        -- owner; secondary index, see sharding
  title           VARCHAR(100) NOT NULL,
  description     TEXT,
  duration_ms     INT,                    -- known only after probing the source
  state           TINYINT NOT NULL,       -- UPLOADING, PROCESSING, READY, FAILED, REMOVED
  visibility      TINYINT NOT NULL,       -- PRIVATE, UNLISTED, PUBLIC
  source_key      VARCHAR(255),           -- object-store key of the original upload
  source_sha256   CHAR(64),               -- exact-duplicate detection
  published_at    DATETIME,
  geo_block       VARBINARY(64),          -- region bitmap for licensing
  manifest_version INT NOT NULL DEFAULT 0 -- bump when renditions are added -> cache key
);

renditions (                              -- one row per playable variant
  video_id   BIGINT, codec VARCHAR(8),    -- h264 | vp9 | av1 ; audio in its own rows
  height     SMALLINT, bitrate_kbps INT,
  segment_count INT, storage_class TINYINT,   -- HOT | WARM | COLD
  PRIMARY KEY (video_id, codec, height)
);

transcode_tasks (                         -- the DAG's durable state
  video_id BIGINT, task_id VARCHAR(64),   -- e.g. "encode:h264:720p:chunk-0042"
  state TINYINT, attempts TINYINT, worker_id VARCHAR(64), lease_until DATETIME,
  output_key VARCHAR(255),
  PRIMARY KEY (video_id, task_id)         -- the task id IS the idempotency key
);

-- =========================================================
-- CHANNEL INDEX  (sharded by channel_id: "all videos by this creator")
-- =========================================================
channel_videos (channel_id BIGINT, published_at DATETIME, video_id BIGINT,
                PRIMARY KEY (channel_id, published_at, video_id))

-- =========================================================
-- ENGAGEMENT  (wide-column: Bigtable / Cassandra)
-- =========================================================
comments      row key: video_id#bucket#reverse_ts#comment_id -> user_id, text, likes
ratings       row key: user_id#video_id  -> like | dislike    (per-user truth)
counters      row key: video_id -> views, likes, dislikes    (aggregated, see Step 12)
watch_history row key: user_id#reverse_ts -> video_id, position_s, device

-- =========================================================
-- BYTES  (object storage, erasure-coded across zones)
-- =========================================================
src/{video_id}/original                          -- never deleted while video exists
v/{video_id}/{manifest_version}/master.m3u8
v/{video_id}/{manifest_version}/h264_720p/playlist.m3u8
v/{video_id}/{manifest_version}/h264_720p/seg_00042.m4s
v/{video_id}/{manifest_version}/audio_en_128k/seg_00042.m4s
thumbs/{video_id}/{variant}.webp                  -- small, huge read volume
captions/{video_id}/{lang}.vtt

Modelling decisions to explain

  • Metadata and bytes are separate systems. A row in videos is about 1 KB; a video is gigabytes. The database stores keys, never blobs. Putting video files in a database (or in a distributed filesystem that the metadata service manages directly) couples two very different scaling curves.
  • The source file is kept forever, the renditions are disposable. Every rendition can be regenerated from src/ when a better codec appears, which is exactly what happens every few years. That is why the source gets the strongest durability and the renditions can live on cheaper tiers or be deleted for cold videos.
  • manifest_version is in the object path. When AV1 renditions are added to an old video, a new version directory is written and the pointer bumped; CDN caches of the old manifest simply age out instead of needing a global purge.
  • Thumbnails are a separate, small-object problem. A page shows 20–40 thumbnails and one video, so thumbnail reads outnumber video plays by an order of magnitude. Billions of 10–50 KB files are inefficient as individual objects on disk (a seek each); store them packed (a Bigtable-like store or a haystack-style packed file with an in-memory offset index) and serve them from the CDN like everything else.
  • Counters are a separate table from videos. The video row is read-mostly and cached; the counter row is write-heavy. Mixing them means every view invalidates the cached video row.
  • Comments are keyed by video, with a bucket. A viral video gets millions of comments; bucketing by time (video_id#2026-09) keeps any one partition bounded.

Step 5 — Sharding the metadata

At billions of video rows and 100k+ reads per second, the metadata has to be partitioned. There are two obvious keys, and the interviewer wants to hear the trade-off.

  • Shard by creator (user_id / channel_id). "All videos by this channel" is a single-shard query, which is nice. But creators are wildly uneven — one channel has 40,000 uploads and a billion views, most have three — so both storage and traffic pile onto whichever shards hold the big channels, and rebalancing means moving an entire channel at once. Anything that is not per-creator (search, "get video by id" when you do not know the owner) scatters to every shard.
  • Shard by video_id. Rows spread evenly because ids are effectively random. "Get video by id" — the call made on every play — hits exactly one shard. The per-channel listing now needs a secondary index, which you maintain as its own table sharded by channel_id. The remaining problem is a viral video: one row, one shard, a million reads a second. That is not a sharding problem; it is a caching problem.
import random, hashlib
from collections import Counter

rng = random.Random(7)
SHARDS, N_VIDEOS, N_CREATORS = 16, 200_000, 20_000

def shard(key: str) -> int:
    return int(hashlib.md5(key.encode()).hexdigest(), 16) % SHARDS

# Creators are skewed (a few upload thousands); so are views (a few videos go viral).
creator_w = [1 / (i + 1) ** 1.1 for i in range(N_CREATORS)]
owners = rng.choices(range(N_CREATORS), weights=creator_w, k=N_VIDEOS)
view_w = [1 / (i + 1) ** 1.1 for i in range(N_VIDEOS)]
rng.shuffle(view_w)                       # popularity is unrelated to video id

def skew(counter: Counter) -> float:      # hottest shard vs a perfectly even split
    return max(counter.values()) / (sum(counter.values()) / SHARDS)

for name, key in (("user_id ", lambda v: f"u{owners[v]}"), ("video_id", lambda v: f"v{v}")):
    rows, reads = Counter(), Counter()
    for v in range(N_VIDEOS):
        s = shard(key(v))
        rows[s] += 1
        reads[s] += view_w[v]
    print(f"shard by {name}: rows skew {skew(rows):.2f}x   read skew {skew(reads):.2f}x")
Output
shard by user_id : rows skew 3.07x   read skew 3.97x
shard by video_id: rows skew 1.02x   read skew 2.73x

The simulation shows the point precisely: sharding by video id fixes storage skew completely (1.02x), but a handful of viral videos still make one shard nearly three times hotter than average. No choice of shard key fixes a single hot row. The answer is a cache in front of the database — the hot video's metadata served from memory in every app server and replicated across cache nodes — so the database only sees the cold tail.

  • Routing: use consistent hashing (or a directory of key ranges, as Vitess and Spanner do) so adding a shard moves only a fraction of the keys, and a dead node's ranges can be taken over by replicas.
  • Replication: each shard is a primary with two or more replicas across zones. Replica lag means a creator who just edited a title might read the old one; route that creator's own reads to the primary for a few seconds after a write (read-your-writes), and let everyone else read replicas.
  • Search does not scatter to shards at all. It goes to a separate search index fed by change-data-capture, so the "query all shards and merge" problem that appears in naive designs never arises.

Step 6 — High-level design

flowchart TB CR[Creator app] --> GW[API gateway
auth, quotas, routing] CR -->|pre-signed part PUTs| RAW[(Object storage
source uploads)] GW --> UP[Upload service] UP --> MDB[(Metadata DB
sharded by video_id)] UP -->|upload.completed| K[(Kafka)] K --> ORCH[Transcode orchestrator
builds the DAG] ORCH --> TQ[[Task queue
by priority]] TQ --> WK[Encode workers
pre-emptible fleet] WK --> RAW WK --> OUT[(Object storage
renditions, thumbs, captions)] ORCH --> MDB ORCH -->|video.ready| K VW[Viewer player] --> GW GW --> PB[Playback service
authorise, sign manifest URL] PB --> MC[(Metadata cache)] MC --> MDB VW -->|manifest + segments| EDGE[CDN edge / ISP appliance] EDGE -->|miss| REG[Regional cache] REG -->|miss| SH[Origin shield] SH -->|miss| OUT VW -->|heartbeats| EV[Event ingest] EV --> K K --> VC[View counter
stream aggregator] VC --> CNT[(Counters store)] K --> IDX[Search indexer] IDX --> ES[(Search index)] K --> REC[Recommendations
offline + online] PB --> LIC[DRM license server]

The left half is a batch pipeline driven by events; the right half is a read path that ends in caches. They share only the object store and the metadata database, and the API tier never touches video bytes.

What each component owns

  • Upload service — creates upload sessions, signs part URLs, checks quotas, verifies the multipart completion, runs exact-hash dedupe, and emits upload.completed. Stateless apart from the metadata row.
  • Transcode orchestrator — turns one upload into a DAG of tasks, tracks their state durably in transcode_tasks, retries failures, and publishes video.ready (and later rendition.added) events. It schedules work; it never encodes.
  • Encode workers — pull one task at a time (split, encode one chunk at one rung, stitch, package, thumbnail), read inputs from and write outputs to object storage. Dumb, horizontally scalable, pre-emptible.
  • Playback service — the only per-play application call. Reads cached metadata, enforces visibility, region and entitlement rules, filters the manifest to what the device can decode, and returns a signed manifest URL plus a DRM license URL.
  • CDN tiers — serve manifests and segments. Edge caches validate the URL token locally and never call the application.
  • Event ingest and view counter — absorb player heartbeats, dedupe, aggregate and flush counts in batches.
  • Search indexer and recommendations — consumers of the event log. If they are down, uploads and playback are unaffected.

Step 7 — The upload path

Resumable, chunked, direct to storage

The upload is split into numbered parts, each sent as its own HTTP request to a pre-signed URL. Object stores support this natively — S3 multipart upload, GCS resumable uploads, Azure block blobs — and the open tus protocol standardises the same idea over plain HTTP for when you run your own upload servers. The two differ in who tracks progress:

S3-style multipart with pre-signed URLstus (resumable upload protocol)
Bytes go toThe object store directlyYour tus servers, which then write to storage
Progress tracked byPart numbers and ETags; the client or your service lists partsA single byte offset (HEAD returns Upload-Offset)
ParallelismParts upload in parallel, in any orderSequential by default; parallel via the concatenation extension
ConstraintsParts at least 5 MiB (except the last), at most 10,000 partsAny chunk size
Good forLarge files, fat pipes, zero upload servers to operateBrowser and mobile SDKs, custom auth, inline processing

Part size is a real decision. Too small and a 400 GB studio master exceeds the part limit; too large and every dropped connection throws away a large part. The planner below starts at 16 MiB and grows only when the file needs it.

import math

MiB, GiB = 1024 ** 2, 1024 ** 3
MIN_PART, MAX_PARTS = 5 * MiB, 10_000       # S3 multipart limits (last part may be smaller)

def plan_parts(size_bytes: int, target_part=16 * MiB) -> tuple[int, int]:
    """Pick a part size: at least 5 MiB, and big enough that we stay under 10,000 parts."""
    part = max(MIN_PART, target_part, math.ceil(size_bytes / MAX_PARTS))
    part = math.ceil(part / MiB) * MiB        # round up to a whole MiB
    return part, math.ceil(size_bytes / part)

for label, size in (("phone clip", 180 * MiB), ("1h 1080p", 6 * GiB),
                    ("2h 4K ProRes", 400 * GiB)):
    part, n = plan_parts(size)
    print(f"{label:>13}: {n:>6,} parts of {part // MiB} MiB")

# ---- Resume after a dropped connection: ask the server what it has ----
class UploadSession:
    def __init__(self, size, part):
        self.size, self.part = size, part
        self.n = math.ceil(size / part)
        self.etags: dict[int, str] = {}      # part_number -> ETag (server's record)

    def missing(self) -> list[int]:
        return [p for p in range(1, self.n + 1) if p not in self.etags]

    def put_part(self, number: int):         # the client PUTs to a pre-signed URL
        self.etags[number] = f"etag-{number}"

s = UploadSession(6 * GiB, 16 * MiB)
for p in range(1, 301):                       # 300 parts in, the train enters a tunnel
    s.put_part(p)
todo = s.missing()
print(f"after reconnect: {len(s.etags)} parts done, resume at part {todo[0]},"
      f" {len(todo)} to go ({len(todo) * 16 / 1024:.1f} GiB), not from zero")
Output
   phone clip:     12 parts of 16 MiB
     1h 1080p:    384 parts of 16 MiB
 2h 4K ProRes:  9,991 parts of 41 MiB
after reconnect: 300 parts done, resume at part 301, 84 to go (1.3 GiB), not from zero

Two more details make uploads robust. The client computes a checksum per part and sends it with the PUT, so corruption is caught per part rather than after 6 GB. And sessions that never complete are garbage-collected by a lifecycle rule (abort incomplete multipart uploads after 7 days), otherwise orphaned parts quietly cost money forever.

Upload to watchable, end to end

sequenceDiagram participant C as Creator app participant U as Upload service participant S as Object storage participant M as Metadata DB participant K as Kafka participant O as Orchestrator participant W as Workers C->>U: POST /uploads (size, sha256, title) U->>M: INSERT video state=UPLOADING U->>S: create multipart upload U-->>C: 201 upload_id, part size C->>U: sign parts 1..50 U-->>C: pre-signed URLs C->>S: PUT part 1..N in parallel Note over C,S: connection drops, app restarts C->>U: GET /uploads/id U->>S: list parts U-->>C: parts 1..300 received C->>S: PUT parts 301..N C->>U: POST complete (part ETags) U->>S: complete multipart U->>M: state=PROCESSING, write outbox row U-->>C: 202 PROCESSING M->>K: upload.completed K->>O: build DAG for video O->>W: probe, split, then encode tasks W->>S: read chunks, write segments W-->>O: task done O->>M: 360p ready, state=READY O->>K: video.ready K-->>C: push notification: your video is live

The creator's device only ever talks to the upload service for small JSON calls. The object store holds the bytes from the first part onwards, and the pipeline starts from a durable event rather than a synchronous call.

The order of operations, and why

  1. Create the metadata row before any bytes move. The video id exists from the start, so every part, log line and retry is attributable. A row stuck in UPLOADING for days is swept.
  2. Dedupe on the declared hash before accepting bytes. If the client sends a SHA-256 up front and it matches an existing source, the upload can be skipped entirely — seconds instead of an hour of upload and 4 core-minutes per minute of transcoding.
  3. Verify on complete. The completion call lists every part's ETag; the store assembles them atomically. Missing parts return 400 PART_MISSING and the client resumes.
  4. Publish upload.completed through an outbox. The state change and the event commit together, so a crash cannot leave a video in PROCESSING that no worker ever hears about.
  5. Mark READY as soon as the first rendition is packaged, not when the whole ladder is done. Higher rungs are added to the manifest as they finish.

Deduplication, at three levels

A large share of uploads are re-uploads: the same file, the same clip re-encoded, a clip with a watermark or borders, a short excerpt of a longer film. Duplicates waste storage, transcode compute, CDN cache space and network, and they pollute search results. Detect them in layers, cheapest first:

  • Exact bytes — SHA-256 of the file, checked before upload. Catches the identical re-upload. Free, but trivially defeated by a single changed byte.
  • Perceptual fingerprint — decode keyframes, downscale to a tiny greyscale grid, and hash the pattern of brightness differences; plus an audio fingerprint of spectral peaks. Re-encoding, resizing, borders and small overlays leave these fingerprints almost unchanged, so near-duplicates match within a small Hamming distance. Run it on the first minute as soon as those parts land, so a duplicate can be flagged while the upload is still in progress.
  • Segment matching — index fingerprints per few seconds of content, so an excerpt matches the middle of a longer original. This is the basis of copyright-match systems and is expensive, so it runs asynchronously after the video is live.

What to do on a match is a product decision, not a storage one. A creator's own identical re-upload can reuse the existing renditions (store a pointer, not a copy). A higher-quality version of an existing video can replace the old source. Someone else's copyrighted film gets routed to the claims workflow. Silently discarding an upload because it looks like another one is never the answer.

Step 8 — The transcoding pipeline

A phone uploads 8 Mbps H.264 at 1080p; a studio uploads 400 GB of ProRes. Neither can be sent to a viewer on a 2 Mbps phone connection. Transcoding produces a bitrate ladder: the same content at several resolutions and bitrates, in several codecs, cut into short segments so the player can switch between them.

The bitrate ladder

RungResolutionH.264 (Mbps)AV1 (Mbps, similar quality)Who gets it
240p426 x 2400.30.18Emergency rung on a failing connection
360p640 x 3600.60.36Poor mobile networks; the fast-start rung
480p854 x 4801.20.7Typical phone on cellular
720p1280 x 7202.51.5Phones on Wi-Fi, laptops
1080p1920 x 10805.03.0Laptops and TVs
2160p3840 x 216016104K TVs, only if the source is 4K

Two refinements matter at scale. Per-title encoding picks the ladder per video: a static lecture looks perfect at 1080p and 1.5 Mbps, while a football match needs 6 Mbps, so a fixed ladder either wastes bytes on the lecture or starves the match. Analysing complexity first and choosing bitrates per title saves 20–40% of egress on typical catalogues. Audio is separate: two or three audio rungs (64, 128, 256 kbps) per language, packaged independently so that one set of video segments serves every audio track.

Codecs: the compute-versus-bandwidth trade

  • H.264/AVC — decodes in hardware on essentially every device ever made. Cheap to encode. The mandatory baseline.
  • VP9 — roughly 30–40% smaller than H.264 at equal quality, royalty-free, broadly supported in browsers and Android. Several times more expensive to encode.
  • HEVC/H.265 — similar efficiency to VP9, strong on Apple devices and TVs, with patent-licensing complexity.
  • AV1 — roughly another 30% smaller than VP9, royalty-free, hardware decode on recent phones and TVs. Encoding costs an order of magnitude more than H.264.

The design consequence: encode H.264 for everything, and spend AV1 or VP9 compute only on videos with enough views to pay it back in egress savings (Step 15 does the arithmetic). The playback service filters the manifest by the codecs the device declares, so old devices never see an AV1 rendition they cannot decode.

Split, encode in parallel, stitch

Encoding a 2-hour 4K film on one machine takes days. The trick is that video is already divided into independently decodable units: a GOP (group of pictures) starts with a keyframe that carries a full image, followed by frames that only store differences. Cut the source at keyframe boundaries into 5–10 second chunks and each chunk can be encoded on a different machine; the outputs join seamlessly because each still starts on a keyframe. Force keyframes at fixed intervals (for example every 2 seconds) in every rung, so that segment boundaries line up across the whole ladder — that alignment is what lets a player switch rungs between any two segments.

flowchart LR SRC[(Source file)] --> PR[Probe + validate
codec, duration, corrupt?] PR --> SPL[Split at keyframes
into N chunks] PR --> AUD[Extract audio] PR --> FP[Fingerprint
dedupe + claims] SPL --> E1[Encode chunks
H.264 360p] SPL --> E2[Encode chunks
H.264 720p] SPL --> E3[Encode chunks
H.264 1080p] SPL --> E4[Encode chunks
AV1 ladder, if popular] AUD --> AE[Encode audio
64 / 128 / 256 kbps] AUD --> ASR[Speech to text
auto captions] SPL --> TH[Thumbnails
+ seek sprite sheet] E1 --> PK[Package: CMAF segments
HLS + DASH manifests] E2 --> PK E3 --> PK E4 --> PK AE --> PK ASR --> PK PK --> RDY[Publish manifest version
mark READY]

Each box is a task in the DAG, and each encode box is really N parallel chunk tasks. The orchestrator releases a task when its inputs exist; the 360p branch runs at the highest priority so the video becomes watchable first.

The simulation schedules the chunk tasks of a 2-hour 4K upload across a worker pool, lowest rung first, and reports when each rung is complete and packaged.

import heapq

# Encoding cost in worker-seconds per second of video, per rung (illustrative, H.264).
RUNGS = [("360p", 0.5), ("240p", 0.3), ("480p", 0.9), ("720p", 1.8),
         ("1080p", 4.0), ("2160p", 16.0)]          # listed in PRIORITY order
PRIORITY = {name: i for i, (name, _) in enumerate(RUNGS)}

def build_dag(duration_s: int, chunk_s: int):
    """split -> encode(chunk, rung) x N -> per-rung package. Returns encode tasks."""
    chunks = range(0, duration_s, chunk_s)          # GOP-aligned cut points
    tasks = []
    for name, cost in RUNGS:
        for start in chunks:
            length = min(chunk_s, duration_s - start)
            tasks.append((PRIORITY[name], start, name, cost * length))
    return sorted(tasks)                            # low-res first, in timeline order

def schedule(tasks, workers: int, split_s=30, package_s=20):
    free = [split_s] * workers                      # every worker waits for the split step
    heapq.heapify(free)
    rung_done: dict[str, float] = {}
    for _, _, rung, cost in tasks:
        t = heapq.heappop(free) + cost
        heapq.heappush(free, t)
        rung_done[rung] = max(rung_done.get(rung, 0), t)
    return {r: t + package_s for r, t in rung_done.items()}   # + write playlist

duration = 2 * 3600                                  # a 2-hour 4K upload
serial = sum(cost for *_, cost in build_dag(duration, 10))
print(f"one machine, serial: {serial / 3600:,.1f} hours")
for workers in (50, 400):
    done = schedule(build_dag(duration, chunk_s=10), workers)
    line = "  ".join(f"{r} {done[r] / 60:5.1f}m" for r, _ in RUNGS)
    print(f"{workers:>3} workers -> {line}")
Output
one machine, serial: 47.0 hours
 50 workers -> 360p   2.1m  240p   2.8m  480p   5.0m  720p   9.3m  1080p  19.2m  2160p  58.7m
400 workers -> 360p   1.0m  240p   1.1m  480p   1.4m  720p   2.0m  1080p   3.3m  2160p   8.6m

47 hours of serial encoding becomes about a minute to first playable and under nine minutes to 4K with 400 workers. With a smaller pool, the priority order still delivers a watchable 360p in two minutes while 4K takes an hour — which is the behaviour creators actually want. Smaller chunks give more parallelism but cost a little compression efficiency and more task overhead; 5–10 seconds is the usual compromise.

Making the pipeline reliable

  • Every task is idempotent and addressed by its output. The task id encode:h264:720p:chunk-0042 determines the output key. Running it twice overwrites the same object with the same bytes, so retries are always safe.
  • Leases, not locks. A worker claims a task with a lease (say 5 minutes) and heartbeats while it works. A pre-empted spot instance simply stops heartbeating, the lease expires, and another worker picks the task up. Losing a worker costs one chunk, never the whole video.
  • Poison inputs are quarantined. A corrupt file that crashes the encoder three times goes to a dead-letter queue and the video is marked FAILED with a reason the creator can see, instead of looping forever.
  • Priority lanes. Short videos and the first rung of every video go in a fast lane; 4K and AV1 go in a bulk lane that soaks up spare capacity overnight. A 12-hour livestream archive must not delay a thousand 30-second clips.
  • Thumbnails and captions are part of the DAG, not an afterthought: candidate thumbnails from frames with high sharpness and faces, a sprite sheet of tiny frames every few seconds for the seek bar preview, and speech-to-text captions exported as WebVTT per language.

Step 9 — Packaging: HLS and DASH manifests

The player does not download "a video". It downloads a manifest — a small text file listing the available renditions — then a per-rendition media playlist listing segments, then the segments themselves, one at a time.

  • HLS (HTTP Live Streaming, from Apple) uses .m3u8 text playlists. Required for Safari and iOS.
  • DASH (MPEG-DASH) uses an XML manifest, the .mpd. Common on Android, smart TVs and in browsers via Media Source Extensions.
  • CMAF — a common fragmented-MP4 segment format both can reference. Encode and store segments once, and write two small manifests pointing at the same files. Before CMAF, platforms stored every video twice, once per protocol.

A master playlist lists the variants and what each costs; the player picks one using the BANDWIDTH and CODECS attributes:

#EXTM3U
#EXT-X-VERSION:7
#EXT-X-INDEPENDENT-SEGMENTS

#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="aud",LANGUAGE="en",NAME="English",DEFAULT=YES,URI="audio_en_128k/playlist.m3u8"
#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="aud",LANGUAGE="hi",NAME="Hindi",DEFAULT=NO,URI="audio_hi_128k/playlist.m3u8"
#EXT-X-MEDIA:TYPE=SUBTITLES,GROUP-ID="subs",LANGUAGE="en",NAME="English",URI="subs_en/playlist.m3u8"

#EXT-X-STREAM-INF:BANDWIDTH=450000,AVERAGE-BANDWIDTH=380000,RESOLUTION=426x240,CODECS="avc1.42c015,mp4a.40.2",AUDIO="aud",SUBTITLES="subs"
h264_240p/playlist.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=900000,AVERAGE-BANDWIDTH=750000,RESOLUTION=640x360,CODECS="avc1.4d401e,mp4a.40.2",AUDIO="aud",SUBTITLES="subs"
h264_360p/playlist.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=3000000,AVERAGE-BANDWIDTH=2600000,RESOLUTION=1280x720,CODECS="avc1.4d401f,mp4a.40.2",AUDIO="aud",SUBTITLES="subs"
h264_720p/playlist.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=5800000,AVERAGE-BANDWIDTH=5100000,RESOLUTION=1920x1080,CODECS="avc1.640028,mp4a.40.2",AUDIO="aud",SUBTITLES="subs"
h264_1080p/playlist.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=3500000,AVERAGE-BANDWIDTH=3100000,RESOLUTION=1920x1080,CODECS="av01.0.08M.08,mp4a.40.2",AUDIO="aud",SUBTITLES="subs"
av1_1080p/playlist.m3u8

Each variant's media playlist lists the segments in order. For video on demand it is complete and ends with #EXT-X-ENDLIST; for live streaming the same file is re-fetched and grows.

#EXTM3U
#EXT-X-VERSION:7
#EXT-X-TARGETDURATION:4
#EXT-X-PLAYLIST-TYPE:VOD
#EXT-X-MAP:URI="init.mp4"
#EXT-X-KEY:METHOD=SAMPLE-AES,URI="skd://key/abc123",KEYFORMAT="com.apple.streamingkeydelivery"
#EXTINF:4.000,
seg_00000.m4s
#EXTINF:4.000,
seg_00001.m4s
#EXTINF:4.000,
seg_00002.m4s
# ... one entry per 4 s ...
#EXTINF:2.480,
seg_00104.m4s
#EXT-X-ENDLIST

The DASH equivalent expresses the same thing as XML, usually with a template instead of listing every segment:

<MPD type="static" mediaPresentationDuration="PT6M58.48S" minBufferTime="PT2S">
  <Period>
    <AdaptationSet contentType="video" segmentAlignment="true">
      <ContentProtection schemeIdUri="urn:uuid:edef8ba9-79d6-4ace-a3c8-27dcd51d21ed"/>
      <SegmentTemplate timescale="1000" duration="4000" startNumber="0"
          initialization="$RepresentationID$/init.mp4"
          media="$RepresentationID$/seg_$Number%05d$.m4s"/>
      <Representation id="h264_360p"  bandwidth="900000"  width="640"  height="360"  codecs="avc1.4d401e"/>
      <Representation id="h264_720p"  bandwidth="3000000" width="1280" height="720"  codecs="avc1.4d401f"/>
      <Representation id="h264_1080p" bandwidth="5800000" width="1920" height="1080" codecs="avc1.640028"/>
    </AdaptationSet>
    <AdaptationSet contentType="audio" lang="en"> ... </AdaptationSet>
  </Period>
</MPD>

Why segments, and why 2–6 seconds

  • Segments are plain static files, so any HTTP cache on earth can serve them. That property is the whole reason HTTP streaming replaced stateful streaming servers.
  • Short segments let the player switch quality quickly and start faster (it needs one segment before first frame). Long segments compress slightly better and mean fewer requests. 4 seconds for VOD and 1–2 seconds (or partial segments) for low-latency live are typical.
  • Seeking is arithmetic. Seek to 11:23 is segment 683 / 4 = 170; the player fetches that segment directly. No server-side offset handling needed.

Step 10 — Playback and adaptive bitrate

From tap to first frame, the budget is about a second: /play call (≈100 ms, cached metadata) → master manifest from the edge (≈30 ms) → media playlist (≈30 ms) → init segment and first media segment (a few hundred ms at the lowest sensible rung). Players cut this further by prefetching the manifest for the thumbnail you are hovering over, starting at a conservative rung, and embedding the first playlist inside the /play response.

After that, the player runs a loop: before each segment, pick a rung. That choice — adaptive bitrate (ABR) — is the entire viewer experience. Pick too high and the buffer drains and the spinner appears; pick too low and a 4K TV shows mush; switch too often and quality visibly pumps. There are two classic families:

  • Throughput-based: measure how fast recent segments downloaded, pick the highest rung under a safety fraction of that estimate. Intuitive, but throughput estimates are noisy on mobile, and the estimate is always about the past.
  • Buffer-based (BBA, and BOLA in dash.js): ignore throughput and pick the rung from how many seconds of video are buffered. A low buffer means play it safe; a full buffer means you can afford a big segment. It uses the buffer as a shock absorber and naturally reaches high quality, but it starts slowly (the buffer is empty at startup) and can be caught by a sudden cliff.

The simulation runs all three strategies — the two classic ones and a hybrid — over the same five-minute commute with a noisy connection and a near-dead tunnel. The digits show the rung chosen for each 4-second segment (0 = 240p, 4 = 1080p).

import random

LADDER = [0.4, 0.8, 1.4, 2.8, 5.0]          # Mbps: 240p, 360p, 480p, 720p, 1080p
SEG_S, MAX_BUF, N_SEGS, DT = 4.0, 30.0, 75, 0.05   # 4 s segments, 5 min of video

# A commute: Wi-Fi, a weak-signal stretch, 4G, a near-dead tunnel, home Wi-Fi.
# Every second is also multiplied by random jitter, as real cellular links are.
PLAN = ((40, 6.0), (70, 1.2), (130, 4.0), (160, 0.5), (1e9, 7.0))
rng = random.Random(2)
JITTER = [rng.uniform(0.3, 1.7) for _ in range(2000)]

def bandwidth(t: float) -> float:
    for until, mbps in PLAN:
        if t < until:
            return mbps * JITTER[int(t)]

def estimate(history):                       # harmonic mean resists one lucky burst
    recent = history[-3:]
    return len(recent) / sum(1 / x for x in recent)

def throughput_rule(buffer_s, history):
    """Classic rate-based: highest rung under 80% of the estimated throughput."""
    if not history:
        return 0
    est = 0.8 * estimate(history)
    return max([i for i, b in enumerate(LADDER) if b <= est], default=0)

def buffer_rule(buffer_s, history, reservoir=6.0, cushion=20.0):
    """BBA-style: the buffer level alone picks the rung. Below the reservoir take
    the lowest rung, above reservoir + cushion the highest, linear in between."""
    if buffer_s <= reservoir:
        return 0
    if buffer_s >= reservoir + cushion:
        return len(LADDER) - 1
    return int((buffer_s - reservoir) / cushion * (len(LADDER) - 1))

def hybrid_rule(buffer_s, history):
    """What real players do: rate-based while the buffer is thin (startup, after a
    seek), buffer-based once it is healthy, and never pick a segment that the
    current throughput could not finish before the buffer hits the reservoir."""
    if buffer_s < 10.0:
        return throughput_rule(buffer_s, history)
    rung, est = buffer_rule(buffer_s, history), estimate(history)
    while rung > 0 and LADDER[rung] * SEG_S / est > buffer_s - 6.0:
        rung -= 1
    return rung

def simulate(rule):
    t = buffer = rebuffer = 0.0
    history, chosen, playing = [], [], False
    for _ in range(N_SEGS):
        rung = rule(buffer, history)
        bits, got, start = LADDER[rung] * SEG_S, 0.0, t
        while got < bits:                        # download in 50 ms steps
            got += bandwidth(t) * DT
            t += DT
            if playing:                          # playback drains the buffer meanwhile
                if buffer > DT:
                    buffer -= DT
                else:
                    rebuffer += DT - buffer      # spinner on screen
                    buffer = 0.0
        history.append(bits / (t - start))       # measured Mbps for this segment
        chosen.append(rung)
        buffer += SEG_S
        playing = True
        if buffer > MAX_BUF:                     # buffer full: pause fetching
            t += buffer - MAX_BUF
            buffer = MAX_BUF
    switches = sum(1 for a, b in zip(chosen, chosen[1:]) if a != b)
    avg = sum(LADDER[r] for r in chosen) / len(chosen)
    return avg, rebuffer, switches, "".join(map(str, chosen))

for name, rule in (("throughput", throughput_rule), ("buffer    ", buffer_rule),
                   ("hybrid    ", hybrid_rule)):
    avg, rb, sw, trace = simulate(rule)
    print(f"{name} avg {avg:.2f} Mbps  rebuffer {rb:4.2f}s  switches {sw}")
    print(f"  {trace}")
Output
throughput avg 3.17 Mbps  rebuffer 0.00s  switches 26
  044443344333333344321111223333333322333231002334443333344444433443334444443
buffer     avg 3.74 Mbps  rebuffer 4.65s  switches 20
  000112334444444444422223333443343433343400112334444444444444444444444444444
hybrid     avg 3.68 Mbps  rebuffer 0.00s  switches 23
  044443311233344444422223333434343334333411101223444444444444444444444444444

Reading the result

  • Throughput-based never stalled but left quality on the table (3.17 Mbps average) and switched 26 times, because every noisy measurement moved the choice.
  • Buffer-based reached the best quality but spent 4.65 seconds on a spinner: it had a full buffer, chose a 1080p segment just as the tunnel started, and that single segment took longer to download than the buffer could cover. Notice also its slow start (000112…): with an empty buffer it will not leave the bottom rung.
  • The hybrid gets both: nearly the buffer-based quality, zero rebuffering. It uses throughput during startup, uses the buffer once it is healthy, and refuses any segment whose estimated download time would eat into the reservoir.

This is roughly how production players behave. dash.js switches between throughput and BOLA modes based on buffer level; Netflix's work on buffer-based ABR uses capacity estimates during startup. Real players add more: abandoning a segment mid-download when it is clearly too slow and refetching a lower rung, capping resolution at the screen size (a phone never needs 4K), and much longer buffers on TVs (minutes, not 30 seconds), which makes the buffer-based approach far safer.

Step 11 — CDN: tiers, placement and the long tail

Tiering

flowchart LR P[Player] -->|DNS / anycast
nearest healthy| E1[Edge: ISP-embedded
appliance, ~50 TB] P -.-> E2[Edge: IXP PoP
~200 TB] E1 -->|miss| R[Regional cache
~1 PB, per country] E2 -->|miss| R R -->|miss, collapsed| SH[Origin shield
~10 PB, per continent] SH -->|miss| O[(Object storage
full catalogue)] F[Nightly fill job
popularity forecast] -.->|push| E1 F -.->|push| R

Each tier absorbs most of the requests that reach it. The origin sees only the long tail, and the fill job pre-positions tomorrow's popular titles during off-peak hours.

  • Edge — servers placed inside ISP networks (Netflix Open Connect and Google Global Cache work this way) or at internet exchange points. Bytes from an ISP-embedded box never cross a transit link, which is why ISPs host them for free: it saves them money too.
  • Regional — larger caches per country or metro that serve edge misses and small ISPs without their own appliances.
  • Origin shield — one designated cache layer in front of the object store. Without it, a cold video that becomes popular causes every regional cache to miss at once and hit origin simultaneously. The shield collapses those into one origin fetch per segment (request coalescing: the first miss fetches, concurrent misses for the same key wait for it).
  • Origin — object storage holding everything, on tiers from hot to archive.

How much each tier absorbs

Video popularity roughly follows a power law: rank k gets views proportional to 1/ks. The exponent s matters enormously, so the simulation computes hit ratios for two plausible values.

def zipf_mass(k: int, s: float, exact_upto: int = 100_000) -> float:
    """Sum of i^-s for i = 1..k: exact for the head, an integral for the long tail."""
    m = min(k, exact_upto)
    head = sum(i ** -s for i in range(1, m + 1))
    if k <= m:
        return head
    a, b = m + 0.5, k + 0.5
    return head + (b ** (1 - s) - a ** (1 - s)) / (1 - s)

CATALOG = 800_000_000          # videos ever uploaded
MB_PER_TITLE = 130             # ~7 min of the 1-2 renditions people actually watch
TIERS = (("edge PoP (one rack)", 50), ("regional cache", 1_000), ("origin shield", 10_000))

print(f"{'tier':<22}{'capacity':>10}{'titles':>13}{'hit s=0.9':>11}{'hit s=1.1':>11}")
for tier, tb in TIERS:
    titles = int(tb * 1e6 / MB_PER_TITLE)
    hits = [zipf_mass(titles, s) / zipf_mass(CATALOG, s) for s in (0.9, 1.1)]
    print(f"{tier:<22}{tb:>7,} TB{titles:>13,}{hits[0]:>11.1%}{hits[1]:>11.1%}")

for s in (0.9, 1.1):
    share = zipf_mass(CATALOG // 100, s) / zipf_mass(CATALOG, s)
    print(f"s={s}: top 1% of videos draw {share:.0%} of views")
Output
tier                    capacity       titles  hit s=0.9  hit s=1.1
edge PoP (one rack)        50 TB      384,615      39.2%      84.1%
regional cache          1,000 TB    7,692,307      57.7%      91.8%
origin shield          10,000 TB   76,923,076      76.2%      96.3%
s=0.9: top 1% of videos draw 58% of views
s=1.1: top 1% of videos draw 92% of views

Two lessons. First, the skew exponent swings the edge hit ratio from 39% to 84% — so measure it on your own traffic before sizing hardware. Real edges do better than either column, because popularity is also local and temporal: a regional cache in Brazil holds what Brazil watches this week, not the global top list. Second, even at the favourable skew, about 4% of requests reach past the shield. Those are the long tail: hundreds of millions of videos watched a few times a day each. They are cheap individually and collectively expensive, and they are served from the regional tier or directly from origin with a slower start — which is acceptable because nobody is waiting for them at scale.

Popularity-based placement

  • Pull-through for the unpredictable: caches fill on demand with LRU or, better, a frequency-aware policy (LFU with ageing, or admission filters that ignore one-hit wonders so a crawler scanning old videos does not flush the hot set).
  • Push for the predictable: a nightly job forecasts what each region will watch tomorrow — a new season of a series, a creator with 50M subscribers who just uploaded — and pre-positions it in off-peak hours when links are idle. This is the heart of Netflix-style delivery, where the catalogue is small and predictable.
  • Rendition-aware placement: cache the rungs people actually watch in that region. An edge serving mostly phones on 4G does not need to hold 4K.
  • Hot-object replication: within one PoP, requests are spread across servers by consistent hashing on the segment URL, so each server holds a distinct slice (no duplicate caching). A viral video would overload its one server, so objects above a request-rate threshold are replicated to several servers in the PoP and requests are spread among them.

Load balancing and steering

The player is steered to an edge by DNS (geo and latency-aware resolution) or anycast, or explicitly: the /play response can name the edge hostname, chosen by a steering service that knows each PoP's load and health. Overloaded servers can redirect a client to a sibling with HTTP 302, but each redirect adds a round trip before the first frame, and redirects across tiers can send a viewer to a distant cache, so steering at the start of the session is preferred to redirecting mid-stream. The player also keeps a list of fallback hosts and fails over itself when a segment request times out.

Step 12 — View counting at scale

A view count looks like UPDATE videos SET views = views + 1. At 35k plays per second, with one launch video drawing a large share of them, that single row becomes the hottest lock in the company. It is also wrong on product grounds: a view should mean someone actually watched (not that a bot requested the manifest), and refreshing the page should not count twice.

import random
from collections import Counter

rng = random.Random(42)
HOT, FLUSH_EVERY_S, MIN_WATCH_S, DEDUPE_WINDOW_S = "v_launch", 10, 30, 3600

def heartbeats(second):
    """5,000 player heartbeats per second: a launch going viral plus 2,000 others."""
    for _ in range(5_000):
        vid = HOT if rng.random() < 0.7 else f"v{rng.randrange(2_000)}"
        viewer = f"u{rng.randrange(200_000)}"               # some viewers repeat
        watched = rng.choice((5, 12, 31, 45, 90, 240))      # seconds watched so far
        yield second, vid, viewer, watched

class ViewAggregator:
    """One instance per stream partition (partitioned by video_id, so no locks)."""
    def __init__(self):
        self.first_seen: dict[tuple, int] = {}   # (video, viewer) -> t; Redis + TTL in prod
        self.pending = Counter()                  # views since the last flush
        self.db = Counter()                       # stands in for the counter store
        self.events = self.db_writes = self.short = self.dups = 0

    def on_event(self, t, vid, viewer, watched):
        self.events += 1
        if watched < MIN_WATCH_S:
            self.short += 1; return               # not a "view" yet
        first = self.first_seen.get((vid, viewer))
        if first is not None and t - first < DEDUPE_WINDOW_S:
            self.dups += 1; return                # refresh spam, replays
        self.first_seen[(vid, viewer)] = t
        self.pending[vid] += 1

    def flush(self):                              # one write per video per window
        for vid, n in self.pending.items():
            self.db[vid] += n                     # UPDATE ... SET views = views + n
            self.db_writes += 1
        self.pending.clear()

agg = ViewAggregator()
for second in range(60):
    for event in heartbeats(second):
        agg.on_event(*event)
    if (second + 1) % FLUSH_EVERY_S == 0:
        agg.flush()

print(f"heartbeats: {agg.events:,}  (naive design = {agg.events:,} row updates)")
print(f"not views: {agg.short:,} under {MIN_WATCH_S}s, {agg.dups:,} already counted")
print(f"views counted for {HOT}: {agg.db[HOT]:,}")
print(f"counter-store writes: {agg.db_writes:,} total, {60 // FLUSH_EVERY_S} on the hot row")
Output
heartbeats: 300,000  (naive design = 300,000 row updates)
not views: 100,263 under 30s, 39,209 already counted
views counted for v_launch: 100,777
counter-store writes: 11,905 total, 6 on the hot row

The design, as the code does it:

  • The player sends heartbeats, not "a view happened". The server decides what counts: a minimum watch time, one view per viewer per video per window, and later, fraud filtering on patterns such as one IP generating thousands of views.
  • Partition the stream by video_id. All events for one video land on one consumer, so dedupe and aggregation need no distributed locks. A single hot video may need its key split across several partitions (video_id#0..7) with a final sum.
  • Aggregate in memory and flush every few seconds, one write per video per window. 300k heartbeats became 11,905 writes, and the viral video's row was touched 6 times instead of 100,777.
  • Two counts, deliberately. A fast approximate count shown publicly within seconds, and a slower, audited count computed from the event log in batch, used for monetisation and trending. They may disagree briefly; the batch one wins. Exactly-once processing (Kafka transactions or idempotent writes keyed by window) keeps the fast one from double-counting on consumer restarts.
  • Likes use the same pattern, with one difference: the per-user record (user_id#video_id → like) is the source of truth, so toggling like, dislike, none is idempotent and the aggregate can always be rebuilt from it.

Step 13 — Search and recommendations (sketch)

Search

An inverted index (Elasticsearch-style) over title, description, tags, channel name and auto-caption text, fed by change-data-capture from the metadata database and by video.ready events. Rank by text relevance blended with engagement signals — views, watch time, freshness, channel quality — and apply visibility and region filters in the query. Shard the index by video id with replicas for read throughput; cache the results of the few thousand most common queries for a minute. The index is allowed to be seconds to minutes behind the database.

Recommendations

The home feed and "up next" drive most watch time on large platforms, so name the architecture even if you do not design the models. It is a funnel:

  1. Candidate generation (offline and nearline) — narrow hundreds of millions of videos to a few hundred per user from several sources: embeddings learnt from co-watch patterns ("people who finished this also finished…"), subscriptions, trending in the user's region, and recent searches. Nearest-neighbour search over video embeddings does the heavy lifting.
  2. Ranking (online, per request) — a model scores each candidate using user features, video features and context (device, time of day), optimising for expected watch time or satisfaction rather than clicks, which would reward clickbait.
  3. Re-ranking and policy — diversity (not ten videos from one channel), freshness, removing already-watched items, and safety filters.

Precompute each active user's candidates in batch and store them keyed by user; run the ranker online within a ~100 ms budget; and if the recommender is down, fall back to a cached trending list for the region. Recommendations are always a degradable feature.

Step 14 — Access control and DRM

  • Signed URLs or cookies for every video. The playback service issues a token containing the video id, an expiry of a few hours, optionally the viewer's IP range or session, and an HMAC signature. The edge checks the signature with a shared key, locally. Without it, a segment URL copied into a forum is a free, permanent download link.
  • Private and unlisted videos are enforced at /play. Making a video private bumps its manifest_version, so outstanding tokens point at a path that no longer resolves, and cached copies expire. This is the one place where eventual consistency must be short and bounded.
  • DRM for premium content. Segments are encrypted once with AES under Common Encryption (CENC, with the cbcs scheme now supported across the major systems), so a single set of encrypted CMAF files serves Widevine (Chrome, Android), FairPlay (Apple) and PlayReady (Windows, many TVs). The player asks the license server for the content key; the license server checks entitlement, issues a key bound to the device's secure hardware, with rules such as expiry, maximum resolution and offline allowance.
  • Key design: separate keys per video and often per resolution tier, so a leaked key for a weakly protected 480p device does not unlock 4K, which is only licensed to devices with hardware-backed security. Rotate keys for long content, and keep the key store behind the license server only.
  • Honest limits: DRM raises the cost of copying; it does not prevent screen recording. Forensic watermarking (per-session invisible marks) is how premium platforms trace leaks.

Step 15 — Cost math

At this scale, architecture decisions are cost decisions. Put rough prices on the three big lines — egress, storage and compute — and several design choices justify themselves.

MONTH_DAYS = 30

# From the capacity estimate
egress_pb_day      = 338
new_storage_tb_day = 2_100 + 2_520          # source + H.264 ladder, one copy
h264_cores         = 97_222

# Illustrative unit prices (list prices vary; the ratios are what matter)
cdn_per_gb      = {"commercial CDN": 0.005, "own ISP-embedded edge": 0.0012}
storage_tb_mo   = {"hot": 21.0, "cold": 4.0}     # object storage, per TB-month
core_hour       = 0.02
storage_overhead = 1.5                           # erasure coding across zones

egress_gb_mo = egress_pb_day * 1e6 * MONTH_DAYS
for name, price in cdn_per_gb.items():
    print(f"egress via {name:<22} ${egress_gb_mo * price / 1e6:>6,.1f}M / month")

year_tb = new_storage_tb_day * 365 * storage_overhead
all_hot = year_tb * storage_tb_mo["hot"]
tiered = year_tb * (0.1 * storage_tb_mo["hot"] + 0.9 * storage_tb_mo["cold"])
print(f"storing one year of uploads: ${all_hot / 1e6:,.1f}M/month all-hot,"
      f" ${tiered / 1e6:,.1f}M/month with 90% in a cold tier")

h264 = h264_cores * 24 * MONTH_DAYS * core_hour
print(f"H.264 transcode: ${h264 / 1e6:,.1f}M / month")

# Should we spend 10x the compute on AV1 (40% smaller files)? Only where views are.
av1_cost_multiplier, av1_saving = 10, 0.40
for share_of_uploads, share_of_views in ((1.00, 1.00), (0.01, 0.58)):
    extra = h264 * av1_cost_multiplier * share_of_uploads
    saved = egress_gb_mo * cdn_per_gb["own ISP-embedded edge"] * av1_saving * share_of_views
    print(f"AV1 for top {share_of_uploads:>4.0%} of uploads: +${extra / 1e6:5.1f}M compute,"
          f" -${saved / 1e6:4.1f}M egress -> net ${(saved - extra) / 1e6:+.1f}M/month")
Output
egress via commercial CDN         $  50.7M / month
egress via own ISP-embedded edge  $  12.2M / month
storing one year of uploads: $53.1M/month all-hot, $14.4M/month with 90% in a cold tier
H.264 transcode: $1.4M / month
AV1 for top 100% of uploads: +$ 14.0M compute, -$ 4.9M egress -> net $-9.1M/month
AV1 for top   1% of uploads: +$  0.1M compute, -$ 2.8M egress -> net $+2.7M/month
  • Egress dominates. Building (or partnering for) ISP-embedded edges cuts the biggest bill roughly by a factor of four, which is why every large video platform eventually runs its own CDN.
  • Storage tiering is not optional. The same year of uploads costs almost four times more if it all sits on hot storage. Move renditions of videos with no views in 90 days to a cold tier; for very cold videos, delete renditions entirely and keep only the source plus a 360p rung, regenerating on demand.
  • AV1 for everything loses money; AV1 for the popular 1% makes money. The top 1% of uploads carries most of the egress, so encoding just those with the expensive codec saves more bandwidth than it costs compute. Make the trigger dynamic: re-encode when a video crosses a view threshold.
  • Transcoding is the smallest line, which is why it is fine to spend compute generously on popular videos, and why pre-emptible capacity is the right fleet.

Step 16 — Consistency, decided per feature

DataGuaranteeWhy
Accepted source uploadDurable before 202 is returnedLosing a creator's upload is the one unforgivable failure
Upload and processing stateStrong for the creator (read from primary)The creator refreshes the page and must see the truth
Visibility changes, takedownsBounded: effective within ~1 minuteLegal and privacy obligations; versioned paths make it fast
Video metadata (title, description)Eventual, secondsCached everywhere; nobody notices a stale title
Manifests and segmentsImmutable per versionImmutable objects never need invalidation, only new versions
View and like countsEventual, seconds (public); exact in batchApproximate and fast for display; audited for money
CommentsEventual, with read-your-writes for the authorThe author must see their own comment immediately
Search index, recommendationsEventual, minutesDerived data, rebuilt from the log
Watch progressLast writer wins, per userThe latest position from any device is the right one

Step 17 — Failure modes and graceful degradation

What failsWhat users seeMechanism
Upload connection dropsUpload pauses, then resumesMultipart session; client lists received parts and continues
Upload service downNew uploads cannot start; in-flight part PUTs continueBytes go to object storage directly; only signing and completion are affected
Transcode worker pre-emptedNothingLease expires, the chunk task reruns; idempotent output keys
Corrupt or malicious source file"Processing failed" with a reasonProbe first, sandboxed decoders, retry cap, dead-letter queue
Transcode backlog after a spikeLow resolution first, HD laterPriority lanes: first rung of every video before 4K of any video
Edge server or PoP downA brief quality dip, rarely a stallPlayer retries a fallback host; steering removes the PoP from DNS
Viral video, cold cachesSlightly slower start for the first viewersOrigin shield with request coalescing; hot-object replication in PoPs
Object storage region degradedCached videos unaffected; long tail slowerCross-region replicated source and hot renditions; shield serves from another region
Metadata DB shard downPopular videos still play; some cold ones fail to startReplica promotion; metadata cache with stale-if-error
DRM license server downPremium content fails to start; free content fineMulti-region license servers; licenses with a validity period so playing sessions continue
Counters, search or recommendations downStale counts, trending fallback feedThey are log consumers: they catch up from Kafka on recovery
A leaked segment URLNothing, after expiryShort-lived signed tokens; version bump for takedowns

Step 18 — Observability

  • Quality-of-experience metrics, from the player: video start time, rebuffering ratio (stalled time over watch time), video start failures, average delivered bitrate, and switches per minute. Slice them by ISP, region, device, CDN PoP and app version — most real incidents appear as "rebuffering up 3x on one ISP in one city".
  • Delivery metrics: cache hit ratio per tier, origin egress, shield request collapse ratio, edge error rates and time to first byte.
  • Pipeline metrics: upload success rate and resume rate, time from upload complete to first playable and to full ladder (p50 and p99, by duration bucket), queue depth per priority lane, task retry and dead-letter rates.
  • Business metrics: watch time per session, abandonment during startup. A regression that adds 500 ms to start time shows up here before anywhere else.
  • Alert on symptoms: page on "video start failures above 1% in a region" or "time to first playable p99 above 10 minutes", not on individual server CPU.

The trade-off summary — how to close

  • Bytes never go through the application tier. Pre-signed URLs for upload, CDN for playback. We pay with a slightly more complex client (multipart, manifests) and gain a design where the application only handles JSON.
  • Segmented HTTP streaming, not a streaming API. Static, immutable, cacheable segments let any cache serve them. We pay with a small startup overhead (manifest fetches) and gain every CDN on earth.
  • Chunked, parallel transcoding with priority. Watchable in about a minute, 4K in minutes. We pay with a DAG orchestrator and a tiny compression loss at chunk boundaries.
  • H.264 for all, expensive codecs for the popular few. Compute is spent where views make it pay back.
  • Metadata sharded by video id, hot rows cached. Sharding balances storage; caching handles virality. The per-channel listing becomes a secondary index.
  • A multi-tier CDN with ISP-embedded edges and push placement. The single largest cost lever in the system.
  • Counts via stream aggregation, not row updates. Fast approximate numbers for display, exact numbers in batch.
  • What I would build first: a single-region pipeline with managed transcoding and a commercial CDN, H.264 only, one metadata database. Build your own edge and per-title AV1 only when the egress bill justifies it — which, at this scale, it quickly does.

For related depth: the object-storage and chunking ideas reappear in File Storage (Dropbox / Drive), sharding and consistent hashing are covered in Choosing and Scaling Databases and Key-Value Store, caching tiers in Scaling and Caching, and the event-log backbone in Queues, APIs and Reliability.

// Write your solution here
// Write your solution here
// Write your solution here

Finished reading? Mark this lesson complete to track your progress.

Up next · Lesson 14File Storage (Dropbox / Drive)Chunking, dedupe, delta sync, metadata service and conflict resolution.