System Design · Lesson 4 of 5

Queues, APIs and Reliability

Async processing, idempotency, rate limiting and failure handling.

• Advanced
• 16 min read
• 3 objectives

Before this lessonLesson 3: Choosing and Scaling Databases

Not everything should happen while the user waits. Sending an email, resizing a video or updating a search index can be done later. Message queues decouple the part of the system that requests work from the part that does it, which improves speed, resilience and scalability.

Queues

A producer puts a message on a queue; one or more consumers take messages off and process them. Examples: RabbitMQ, Amazon SQS, Kafka, Redis streams.

• Smooths spikes: a burst of 10,000 signups is absorbed by the queue and processed at a steady pace.
• Isolates failure: if the email service is down, messages wait instead of failing user requests.
• Scales independently: add more consumers when the queue grows.

# producer (in the web request)
queue.publish("send_welcome_email", {"user_id": 42})
return {"status": "created"}          # respond immediately

# consumer (background worker)
def handle(message):
    user = db.get_user(message["user_id"])
    email.send(user.email, "Welcome!")

Publish/subscribe (pub/sub) delivers one event to many subscribers, such as "order placed" going to billing, shipping and analytics. Kafka is a popular log that lets consumers replay history.

Delivery guarantees and idempotency

Most queues guarantee at-least-once delivery: a message may be delivered twice (for example after a crash before acknowledgment). So consumers must be idempotent: processing the same message twice has the same effect as once.

def charge(message):
    key = message["idempotency_key"]
    if db.exists("processed", key):       # already handled
        return
    with db.transaction():
        payments.charge(message["user_id"], message["amount"])
        db.insert("processed", key)

The same idea protects APIs. A client sends an Idempotency-Key header on a payment request; if the network drops and the client retries, the server recognizes the key and does not charge twice.

Retries, backoff and dead-letter queues

• Retry transient failures with exponential backoff plus random jitter (wait 1s, 2s, 4s...) to avoid hammering a struggling service.
• After a limited number of attempts, move the message to a dead-letter queue for inspection instead of retrying forever.
• Use timeouts everywhere; a call without one can hang a whole system.
• A circuit breaker stops calling a failing dependency for a while, so failures do not cascade.

Rate limiting

Protect your service from abuse and overload by limiting requests per client. A token bucket allows bursts while enforcing an average rate.

import time

class TokenBucket:
    def __init__(self, rate, capacity):
        self.rate, self.capacity = rate, capacity
        self.tokens, self.last = capacity, time.monotonic()

    def allow(self):
        now = time.monotonic()
        self.tokens = min(self.capacity, self.tokens + (now - self.last) * self.rate)
        self.last = now
        if self.tokens >= 1:
            self.tokens -= 1
            return True
        return False          # respond with HTTP 429 Too Many Requests

bucket = TokenBucket(rate=5, capacity=10)   # 5 req/s, bursts up to 10

API design basics

• Version your API (/v1/) so you can change it without breaking clients.
• Paginate lists, ideally with cursors instead of large offsets.
• Return consistent errors and correct status codes.

// Write your solution here