Tokens, Context Windows and Cost

Why models read tokens not words, what a context window really is, and how to estimate cost and latency.

• Beginner
• 20 min read
• 3 objectives

Before this lessonLesson 1: What Is AI? What Is an LLM?

If you remember one technical detail about LLMs, make it this: models do not read words, they read tokens. Tokens decide how much you pay, how much fits in a prompt, why models sometimes misspell strange words, and why counting letters in "strawberry" is embarrassingly hard for them. Let us demystify them.

What is a token?

A token is a chunk of text, usually a common word, a piece of a word, or a punctuation mark. Before the model sees your text, a tokenizer chops it into tokens, and each token is replaced by a number (its ID). The model only ever sees the numbers.

Text:       "Unbelievably, ChatGPT is fun!"
Tokens:     ["Un", "believ", "ably", ",", " Chat", "G", "PT", " is", " fun", "!"]
Token IDs:  [ 3118, 30553, 2905,  11,  13149,  38,  2898,  382,  2823,   0 ]
            (illustrative, real IDs depend on the model)

Rules of thumb for English: 1 token is about 4 characters, or roughly three-quarters of a word. So 100 tokens is about 75 words, and a page of text is roughly 500–700 tokens. Code, other languages and unusual words split into more tokens, which makes them costlier.

How do tokenizers decide where to cut? A mini BPE

Most tokenizers use Byte Pair Encoding (BPE). Start with individual characters, then repeatedly merge the most frequent neighbouring pair into a new token. Frequent words end up as single tokens; rare words stay in pieces. Here is BPE in about fifteen lines, learning from a tiny corpus:

from collections import Counter

corpus = ["low", "lower", "lowest", "newer", "newest", "wider", "widest"] * 3

# start: every word is a tuple of characters
vocab = Counter(tuple(w) for w in corpus)

def most_frequent_pair(vocab):
    pairs = Counter()
    for word, freq in vocab.items():
        for a, b in zip(word, word[1:]):
            pairs[(a, b)] += freq
    return pairs.most_common(1)[0][0] if pairs else None

def merge(vocab, pair):
    new = Counter()
    for word, freq in vocab.items():
        out, i = [], 0
        while i < len(word):
            if i < len(word) - 1 and (word[i], word[i + 1]) == pair:
                out.append(word[i] + word[i + 1]); i += 2
            else:
                out.append(word[i]); i += 1
        new[tuple(out)] += freq
    return new

for step in range(1, 8):
    pair = most_frequent_pair(vocab)
    vocab = merge(vocab, pair)
    print(f"merge {step}: {pair[0]!r} + {pair[1]!r} -> {pair[0] + pair[1]!r}")

print()
for word in sorted(vocab):
    print(word)

merge 1: 'w' + 'e' -> 'we'
merge 2: 'l' + 'o' -> 'lo'
merge 3: 's' + 't' -> 'st'
merge 4: 'lo' + 'we' -> 'lowe'
merge 5: 'n' + 'e' -> 'ne'
merge 6: 'ne' + 'we' -> 'newe'
merge 7: 'w' + 'i' -> 'wi'

('lo', 'w')
('lowe', 'r')
('lowe', 'st')
('newe', 'r')
('newe', 'st')
('wi', 'd', 'e', 'r')
('wi', 'd', 'e', 'st')

Watch what happened: "low" and "est" and "er" became tokens because they appear in many words. A word the tokenizer never saw is built from smaller pieces, so it can represent anything, even typos and other languages, just less efficiently.

Why token weirdness explains model quirks

• Letter counting. "strawberry" might be tokens like ["str", "aw", "berry"]. The model never sees individual letters, so "how many r's?" requires reasoning about pieces it cannot see. (Give it a code tool and it is trivial.)
• Arithmetic. Numbers are chopped into tokens unevenly, so long multiplication is unreliable without a calculator tool.
• Non-English text costs more. Languages underrepresented in the tokenizer's training need more tokens per word, so the same sentence can cost 2–5× more.
• Leading spaces matter. " fun" and "fun" are different tokens.

The context window: the model's working memory

The context window is the maximum number of tokens the model can consider in one go: your instructions, the conversation so far, any documents you pasted, and the answer it is writing. Modern models range from 8,000 to over a million tokens.

Two ideas people mix up: the context window is not long-term memory. Nothing persists between requests. A chatbot "remembering" your name is your app re-sending the whole conversation every single time. When the conversation grows past the window, something must be dropped or summarised.

flowchart LR subgraph WIN["Context window - say 128k tokens"] direction LR A[System prompt] --- B[Conversation history] --- C[Retrieved documents] --- D[Your question] --- E[Answer so far] end

Everything here is re-read by the model on every request, and every token costs money.

What does it cost? Doing the math

APIs charge per token, with output tokens costing more than input tokens (often 3–5×), because generating is more work than reading. Prices are quoted per million tokens. Let us build a calculator (the prices are illustrative; always check the current price sheet).

def cost(input_tokens, output_tokens, in_price, out_price):
    """Prices are dollars per 1,000,000 tokens."""
    return input_tokens / 1e6 * in_price + output_tokens / 1e6 * out_price

# A support chatbot: 2,000 tokens in (docs + question), 300 tokens out
per_request = cost(2000, 300, in_price=3.00, out_price=15.00)
print(f"one request:      ${per_request:.5f}")
print(f"10,000 requests:  ${per_request * 10_000:,.2f}")
print(f"per month (1M):   ${per_request * 1_000_000:,.2f}")

# Same load on a small, cheap model
cheap = cost(2000, 300, in_price=0.15, out_price=0.60)
print(f"\nsmall model, 1M requests: ${cheap * 1_000_000:,.2f}")
print(f"savings: {1 - cheap / per_request:.0%}")

one request:      $0.01050
10,000 requests:  $105.00
per month (1M):   $10,500.00

small model, 1M requests: $480.00
savings: 95%

That last line is why model routing is a real engineering topic: send easy questions to a small model and only hard ones to the big one, and the bill can drop by an order of magnitude.

Latency: the other cost

• TTFT (time to first token): how long before anything appears. Depends on prompt length and server load. This is what makes a chatbot feel snappy or sluggish.
• Tokens per second: how fast the answer streams. Roughly 30–150 for typical hosted models.
• Total time ≈ TTFT + (output tokens ÷ tokens per second). A 500-token answer at 50 tokens/s takes about 10 seconds, which is exactly why streaming (lesson 5) matters: users start reading after 0.5 seconds instead of waiting 10.

Ways to cut cost and latency

• Trim the prompt: remove filler, retrieve fewer but better chunks.
• Cap the output with max_tokens and ask for concise answers.
• Use prompt caching: many providers charge much less for a long prefix that repeats (a big system prompt, a document you keep asking about).
• Route by difficulty (small model first, big model when needed).
• Cache whole answers for repeated questions.

# Write your solution here