Video Streaming (YouTube)

Upload, transcode, CDN playback, recommendations and comments.

• Advanced
• 40 min read
• 3 objectives

Before this lessonLesson 10: Chat System (WhatsApp / Messenger)

Design a video platform: a creator uploads a file, you process it, and viewers around the world can play it with low startup time. Comments, thumbs, and a simple 'next video' hook sit around the core pipeline — they are not the first bottleneck.

Step 1 — Clarify

• Max length and resolution? Live streaming in v1, or VOD only?
• Who can upload? Must the video be playable before 1080p finishes?
• Recommendations and search in scope?
• Comments, likes, subscriptions?

v1: VOD only. Upload, transcode to several bitrates, playback with adaptive streaming, title/thumb, view count, comments. Live, full search, and a production recommender are follow-ups you can name without designing the ML.

Step 2 — Scale

uploads_per_day = 1_000_000
avg_min = 8
mb_per_min = 10                 # one rendition
renditions = 3                  # 360p, 720p, 1080p
storage_tb_day = uploads_per_day * avg_min * mb_per_min * renditions / 1e6
print("new video TB/day", round(storage_tb_day))
print("watch is orders of magnitude above upload — CDN bound")

new video TB/day 240
watch is orders of magnitude above upload — CDN bound

Hundreds of TB a day of video means object storage and a CDN. The metadata database stores ids, titles, duration, and pointers — never the bytes. Playback QPS is enormous compared with upload QPS; the two paths must not share a bottleneck.

Step 3 — API and metadata

POST /v1/videos            { title } -> { video_id, upload_url }
PUT  <pre-signed S3 URL>   raw bytes (multipart / resumable)
POST /v1/videos/{id}/complete
GET  /v1/videos/{id}       metadata + playback playlist URL
GET  /v1/videos/{id}/comments?cursor=

videos(video_id, user_id, title, status, duration, created_at)
renditions(video_id, height, bitrate, playlist_key)
thumbs(video_id, offset_s, image_key)
comments(video_id, comment_id, user_id, text, ts)  -- shard by video_id
-- status: uploading | processing | ready | failed

Step 4 — Upload and transcode

flowchart TB U[Creator] -->|pre-signed PUT| S3[(Object storage)] U --> API[API: create row] API --> Q[Transcode queue] Q --> W[FFmpeg workers] W --> S3 W --> Meta[(Metadata DB)] P[Viewer] --> CDN[CDN] CDN -->|miss| S3 P --> API2[Playback API] API2 --> Meta

Bytes never stream through the API. Workers write renditions back to object storage. Viewers hit the CDN; origin is S3.

1. API creates a row status=uploading and returns a pre-signed (or multipart/resumable) URL.
2. The client PUTs straight to object storage. On complete, status=processing and a message lands on the transcode queue.
3. Workers produce HLS/DASH renditions (360p first, then 720p/1080p) plus a thumbnail strip. Each job is idempotent on (video_id, rendition).
4. When the first playable rendition exists, status=ready so the creator can share while 1080p still cooks. Failures retry; poison files go to a DLQ.

Never proxy gigabytes through the app server — you will exhaust RAM and sockets.

Step 5 — Playback

• The player asks the API for a short-lived signed playlist URL.
• The CDN caches segments; origin is object storage. Viral videos stay at the edge; cold videos pay one origin fetch.
• Adaptive bitrate: the player switches renditions as bandwidth changes (HLS/DASH).
• View counts: do not UPDATE videos SET views = views+1 per play. Emit events to a stream; aggregate in Redis and flush every few seconds.
• Comments: shard by video_id, cursor pagination, cache the first page of a hot video.

Step 6 — Discovery (keep it short unless asked)

Search: an inverted index on title/tags, not on the video bytes. Recommendations: offline jobs build candidate sets; an online ranker personalizes. Name the split (offline candidates / online rank) and stop unless they want ML.

// Write your solution here