APIs, SSE, WebSockets and Streaming

How apps talk to models over HTTP, and how streaming works: polling, SSE, WebSockets, with a working parser.

• Intermediate
• 26 min read
• 3 objectives

Before this lessonLesson 4: Hallucinations: Why and How to Reduce Them

So far we treated the model as a magic box. Now let us open the plumbing. How does your app actually talk to an LLM? Why does ChatGPT type out its answer word by word instead of making you wait? And what are SSE and WebSockets, the two words everyone throws around when discussing real-time AI apps? This lesson answers all of it, with a working streaming parser you can run.

APIs in one minute

An API (application programming interface) is a doorway that lets one program use another program's abilities. The web mostly speaks HTTP: your program sends a request (a URL, a method like POST, headers, a body), and gets back a response (a status code, headers, a body). The body is usually JSON, a simple text format for structured data.

sequenceDiagram participant App as Your app participant LLM as LLM provider App->>LLM: POST /v1/chat/completions Note right of App: Bearer key plus messages JSON LLM-->>App: 200 OK with assistant message

• Method: GET reads, POST sends data (LLM calls are POSTs).
• Status codes: 200 ok, 400 you sent something wrong, 401 bad API key, 429 rate limited (slow down!), 500/503 the server had a problem.
• API key: a secret that identifies you and bills you. Keep it on your server. Never put it in browser JavaScript or commit it to Git.
• Rate limits: providers cap requests and tokens per minute. Handle 429 with retry and exponential backoff (wait 1s, 2s, 4s, with a little randomness).

A minimal call with Python's requests library looks like this (shown for reading; it needs a real key so it has no Run button):

import os, requests

response = requests.post(
    "https://api.example.com/v1/chat/completions",
    headers={"Authorization": f"Bearer {os.environ['API_KEY']}"},
    json={
        "model": "some-model",
        "messages": [{"role": "user", "content": "Explain RAG in one sentence."}],
        "max_tokens": 100,
    },
    timeout=30,
)
response.raise_for_status()
print(response.json()["choices"][0]["message"]["content"])

The waiting problem

A 500-token answer takes ten seconds to generate. If your app waits for the whole answer before showing anything, users stare at a spinner for ten seconds and assume it is broken. The fix is streaming: the server sends each token (or small group of tokens) the moment it is produced, and the UI paints them as they arrive. Total time is the same, but the first words appear in half a second. Perceived speed is what matters.

Four ways to move data in real time

How can a server push data to a browser as it becomes available? There are four classic answers:

• Polling. The client asks again and again: "anything new? anything new?" Simple, wasteful, laggy.
• Long polling. The client asks; the server holds the request open until it has something, then replies, and the client immediately asks again. Better, but clunky.
• Server-Sent Events (SSE). One normal HTTP request that the server keeps open, streaming a text feed of events down it, one direction only (server to client).
• WebSockets. A permanent, two-way connection upgraded from HTTP. Both sides can send messages at any time.

SSE in detail: the format of LLM streaming

SSE is what most LLM APIs use for streaming, so it is worth understanding at the byte level. The response has the header Content-Type: text/event-stream and the connection stays open. The body is plain text made of events, each a few lines, separated by a blank line:

event: token
data: {"text": "Hel"}

event: token
data: {"text": "lo"}

data: {"text": " world"}

data: [DONE]

• A line starting with data: carries the payload (often JSON).
• event: optionally names the event type.
• id: optionally sets an ID, so a reconnecting client can say "resume after event 42" using the Last-Event-ID header.
• A blank line ends the event.
• Lines starting with : are comments, often used as keep-alive heartbeats.

Let us write a real SSE parser and feed it a simulated LLM stream, chopped at awkward places, the way real network chunks arrive:

import json

def sse_events(chunks):
    """Turn arbitrary network chunks into complete SSE events."""
    buffer = ""
    for chunk in chunks:
        buffer += chunk
        # an event is finished when we see a blank line
        while "\n\n" in buffer:
            raw, buffer = buffer.split("\n\n", 1)
            event = {"event": "message", "data": []}
            for line in raw.split("\n"):
                if line.startswith(":"):          # comment / heartbeat
                    continue
                field, _, value = line.partition(":")
                value = value.lstrip(" ")
                if field == "data":
                    event["data"].append(value)
                elif field == "event":
                    event["event"] = value
            if event["data"]:
                event["data"] = "\n".join(event["data"])
                yield event

# A stream that arrives in badly cut pieces (mid-line, mid-event)
network_chunks = [
    'event: token\ndata: {"te',
    'xt": "Hel"}\n\nevent: token\nda',
    'ta: {"text": "lo"}\n\n: heartbeat\n\n',
    'data: {"text": " world"}\n\ndata: [DONE]\n\n',
]

answer = ""
for event in sse_events(network_chunks):
    if event["data"] == "[DONE]":
        print("stream finished")
        break
    token = json.loads(event["data"])["text"]
    answer += token
    print(f"got {event['event']!r:8} -> {token!r:9} so far: {answer!r}")

got 'token'  -> 'Hel'     so far: 'Hel'
got 'token'  -> 'lo'      so far: 'Hello'
got 'message' -> ' world'  so far: 'Hello world'
stream finished

Notice how the parser copes with network chunks that cut an event in half: it buffers until it sees the blank line. Getting this right is the classic bug in hand-written streaming clients.

The server side: streaming with SSE

On the server, streaming means returning a generator that yields events. Here is the shape in FastAPI (read-only, needs the framework):

from fastapi import FastAPI
from fastapi.responses import StreamingResponse
import json, asyncio

app = FastAPI()

async def token_stream(prompt: str):
    # in real life: async for chunk in llm.stream(prompt): ...
    for word in ["RAG", " retrieves", " then", " generates."]:
        yield f"data: {json.dumps({'text': word})}\n\n"
        await asyncio.sleep(0.1)
    yield "data: [DONE]\n\n"

@app.post("/chat")
async def chat(body: dict):
    return StreamingResponse(token_stream(body["prompt"]), media_type="text/event-stream")

And in the browser. Note that EventSource only supports GET and cannot set headers, so for POST-based chat APIs people use fetch() and read the response body as a stream:

const response = await fetch("/chat", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({ prompt: "Explain SSE" }),
});

const reader = response.body.getReader();
const decoder = new TextDecoder();
let buffer = "";

while (true) {
  const { value, done } = await reader.read();
  if (done) break;
  buffer += decoder.decode(value, { stream: true });

  let idx;
  while ((idx = buffer.indexOf("\n\n")) !== -1) {
    const raw = buffer.slice(0, idx);
    buffer = buffer.slice(idx + 2);
    if (raw.startsWith("data: ")) {
      const data = raw.slice(6);
      if (data === "[DONE]") return;
      output.textContent += JSON.parse(data).text;   // paint each token
    }
  }
}

WebSockets: the two-way street

A WebSocket starts as a normal HTTP request with an Upgrade: websocket header. If the server agrees, the connection stops being HTTP and becomes a persistent pipe where either side can send frames at any moment. There is no request/response pairing.

sequenceDiagram participant C as Client participant S as Server C->>S: GET /chat Upgrade websocket S-->>C: 101 Switching Protocols Note over C,S: Persistent two-way pipe C->>S: user_message S-->>C: token Hel S-->>C: token lo C->>S: cancel S-->>C: done

SSE vs WebSocket: how to choose

• Direction. SSE is one-way (server to client). WebSocket is two-way. A chatbot that streams an answer needs mostly one-way, so SSE is enough.
• Simplicity. SSE is plain HTTP: works with standard proxies, load balancers, auth cookies, and auto-reconnects for free. WebSockets need special handling in infrastructure.
• Reconnection. SSE has built-in reconnect with Last-Event-ID. With WebSockets you build it yourself.
• Data types. SSE is text only. WebSocket can carry binary (audio, images).
• Scale and cost. Many long-lived connections cost memory on the server either way; SSE is generally lighter to operate.
• Choose WebSockets when the client must send frequent messages while receiving: live voice conversations, collaborative editing, multiplayer, interrupt/cancel controls, browser agents streaming screenshots.
• Choose SSE when the client sends one request and the server streams back a response: chat answers, progress updates, live logs.

A useful mental model: SSE is a radio broadcast you tune into. A WebSocket is a phone call.

Beyond text: what real-time AI apps also use

• WebRTC for low-latency audio and video (voice assistants that you can interrupt mid-sentence).
• Webhooks: the server calls your URL when a long job finishes (batch jobs, background agents).
• Job queues + polling for tasks that take minutes: return a job ID immediately, let the client poll for status, or push the result when done.

Production checklist for streaming

• Disable proxy buffering (for nginx: X-Accel-Buffering: no), or your tokens arrive in one lump.
• Send periodic heartbeats so idle connections are not closed.
• Handle client disconnects: stop generating, and stop paying for tokens nobody will read.
• Show a stop button and support cancellation.
• Time out and retry with backoff on 429 and 5xx.
• Log token counts and time-to-first-token per request.

# Write your solution here