File Storage (Dropbox / Drive)

Chunked upload, sync, sharing and metadata vs blob storage.

• Advanced
• 38 min read
• 3 objectives

Before this lessonLesson 11: Video Streaming (YouTube)

Design cloud file storage: a user drops files on one device and they appear on others, with sharing, versions, and a sane answer when two devices edit at once. Photo apps are the same split (metadata vs bytes) without the desktop sync protocol.

Step 1 — Clarify

• Personal drive, or team shared folders?
• Max file size? Millions of tiny files?
• Version history length? Conflict behaviour?
• Sharing by link vs ACL? Offline clients?

v1: upload/download, sync a folder across devices, share by link or user ACL, keep versions, detect identical blocks so a second copy of a 2 GB video does not upload twice.

Step 2 — Split metadata from bytes

flowchart LR Client -->|tree / rev| Meta[Metadata service] Meta --> MDB[(Postgres shards by user)] Client -->|blocks| BS[Block / object store] BS --> S3[(S3 or block cluster)] Meta --> N[Notification service] N --> Other[Other devices]

Names, folders, permissions and versions live in a database. File bytes are content-hashed blocks in object storage. Notifications wake other devices to pull.

If you store files as blobs on the same database that holds the folder tree, you will lose. Metadata must answer 'what is in /Projects?' in milliseconds. Bytes can be slow, resumable, and deduplicated.

files(file_id, user_id, parent_id, name, is_dir, head_rev)
revisions(file_id, rev, size, created_at)
blocks(hash, size)                       -- global chunk table
file_blocks(file_id, rev, seq, hash)     -- ordered chunks of a revision
acls(file_id, principal, perm)

Step 3 — Upload and sync

1. Client splits the file into ~4 MB chunks and hashes each (SHA-256).
2. It asks metadata which hashes are new. Only missing chunks are uploaded (resumable, parallel).
3. Client commits a revision: an ordered list of hashes. Metadata bumps head_rev.
4. A notification (WebSocket or long-poll) tells other devices to pull the tree delta and download missing blocks.

import hashlib

def chunk_hashes(data: bytes, size=4 * 1024 * 1024):
    out = []
    for i in range(0, len(data), size):
        piece = data[i:i + size]
        out.append(hashlib.sha256(piece).hexdigest()[:16])
    return out

sample = b"hello world" * 100
print(chunk_hashes(sample, size=16)[:3], "n=", len(chunk_hashes(sample, size=16)))

Step 4 — Sharing, versions, conflicts

• Share: ACL rows, or a capability link. A shared folder may need its own workspace shard — mention that moving a folder between users is a reshard, not a rename.
• Versions: every commit is a new rev pointing at blocks. Old revs stay until a retention job drops them. Identical files share blocks (copy-on-write).
• Conflicts: two devices commit from the same parent → keep both (conflict copy) or last-write-wins for binaries. Collaborative docs (OT/CRDT) are a different design; do not pretend a drive sync is Google Docs.
• Notifications: do not poll metadata every second. A notification service (later lesson) wakes the desktop client.

// Write your solution here