Scaling and Caching

Load balancers, horizontal scaling, caches and CDNs.

When one server is no longer enough you have two options: make it bigger (vertical scaling) or add more of them (horizontal scaling). Vertical is simple but has a ceiling and a single point of failure. Horizontal scales much further and tolerates failures, but requires your design to spread work across machines.

Stateless services and load balancers

To scale out, application servers should be stateless: no user data kept only in a server's memory. Sessions and files live in shared stores (a database, Redis, object storage), so any server can handle any request. A load balancer sits in front and distributes traffic.

• Algorithms: round robin, least connections, hash by client.
• Health checks remove failed servers automatically.
• Layer 4 balances by connection; layer 7 (HTTP) can route by path or header.
• Run at least two balancers, or use a managed one, to avoid a new single point of failure.

Caching

A cache stores results of expensive work close to where they are needed. It is the single most effective performance tool: memory is roughly a thousand times faster than a database query. Caches exist at every layer:

• Browser cache and CDN: static files and even API responses served from a location near the user.
• Application cache (Redis, Memcached): computed results, sessions, hot database rows.
• Database cache: query plans and pages.

Cache-aside pattern

The most common approach: the app checks the cache first, falls back to the database on a miss, then stores the result.

import json, redis
r = redis.Redis()

def get_user(user_id):
    key = f"user:{user_id}"
    cached = r.get(key)
    if cached:                              # hit
        return json.loads(cached)
    user = db.query_user(user_id)           # miss: go to the database
    r.setex(key, 300, json.dumps(user))     # store with a 5-minute TTL
    return user

def update_user(user_id, data):
    db.update_user(user_id, data)
    r.delete(f"user:{user_id}")             # invalidate so the next read refreshes

Other write strategies

• Write-through: write to cache and database together. Reads are always fresh; writes are slower.
• Write-back: write to cache and flush to the database later. Fast, but data can be lost if the cache fails.

Invalidation and eviction

"There are only two hard things in computer science: cache invalidation and naming things." Stale data is the price of caching. Manage it with a TTL (expire after a time), explicit invalidation on writes, and versioned keys. When memory fills, an eviction policy such as LRU (least recently used) decides what to drop.

Cache stampede

When a popular key expires, thousands of requests can hit the database at once. Mitigate with locks so only one request recomputes, random jitter added to TTLs, and refreshing entries before they expire.

Cache the product JSON in Redis with cache-aside and a TTL of ~10 minutes,
plus a CDN for images. When the price changes, the update code deletes the
key (explicit invalidation) so users see the new price immediately; the
TTL is only a safety net. Add jitter to TTLs to avoid a stampede.