LLMs and Transformers: How They Actually Work

Attention, tokenization, positional encoding, context windows, sampling, KV cache, mixture of experts and scaling laws, explained the way interviewers want to hear them.

• Intermediate
• 18 min read
• 11 questions

Before this lessonLesson 1: AI and Machine Learning Fundamentals

This is the lesson that separates people who have used an LLM API from people who understand what is happening inside. Expect these questions in almost every applied-AI, ML-engineer and research-engineer loop at large technology companies, and expect follow-ups such as "why the square root?" or "how much memory does that take?".

Keep two habits: draw the data flow on a whiteboard, and put rough numbers on anything you claim (parameters, memory, tokens per second). Numbers are what make an answer sound like experience.

The 11 questions in this lesson

12. What is a Large Language Model and how does it generate text?
13. Walk me through the transformer architecture and what happens in one forward pass of a decoder-only model.
14. Explain self-attention: what are Query, Key and Value, why do we divide by the square root of d_k, and what is causal masking?
15. What is multi-head attention, and why use several heads? What are multi-query and grouped-query attention?
16. What is tokenization? Explain BPE, WordPiece and SentencePiece, and why not just use words or characters?
17. Why do transformers need positional encoding? Compare sinusoidal, learned and rotary (RoPE) encodings.
18. What is the context window, why is it limited, and what is the 'lost in the middle' problem?
19. Explain temperature, top-k, top-p and greedy decoding. When would you use each?
20. Compare encoder-only, decoder-only and encoder-decoder transformers. Which is used for what?
21. What is the KV cache? Why is the first token slower than the rest, and how do you estimate its memory?
22. What is Mixture of Experts (MoE)? How does it differ from a dense model, and what are scaling laws?

12. What is a Large Language Model and how does it generate text?

Warm-up

An LLM is a neural network, almost always a transformer, with billions of parameters, trained on a very large text corpus to predict the next token given all the tokens before it. That single objective, learned at scale, produces grammar, facts, style and a surprising amount of reasoning.

Generation is a loop. The model reads the prompt, outputs a probability distribution over the whole vocabulary for the next token, one token is chosen (greedily or by sampling), it is appended to the sequence, and the loop repeats until a stop token or a length limit. This is called autoregressive decoding.

flowchart LR P[Prompt tokens] --> M[Transformer forward pass] M --> L[Logits over vocabulary] L --> S[Sample or pick next token] S --> A[Append token to sequence] A --> C{Stop token or limit?} C -- no --> M C -- yes --> O[Return text]

Training happens in stages: pre-training on raw text (next-token prediction), supervised fine-tuning on instruction-response pairs so it follows requests, then preference optimisation (RLHF or DPO) so answers are helpful and safe. The honest caveat to add: the model optimises for plausible text, not verified truth, which is the root cause of hallucination.

Follow-up: If the model only predicts the next token, why does it seem to plan and reason?

13. Walk me through the transformer architecture and what happens in one forward pass of a decoder-only model.

Core

A decoder-only transformer turns token ids into a next-token distribution in five steps: (1) look up an embedding vector per token; (2) inject position information (for example with RoPE); (3) pass the vectors through a stack of N identical blocks; (4) apply a final normalisation; (5) project each position with a linear layer to vocabulary-size logits, then softmax. Only the last position's distribution is used to pick the next token.

flowchart TB T[Token ids] --> E[Token embeddings] E --> B0 subgraph BLK["Transformer block x N"] B0[Input x] --> LN1[Layer norm] LN1 --> ATT["Masked multi-head self-attention"] ATT --> ADD1(("+ residual")) B0 --> ADD1 ADD1 --> LN2[Layer norm] LN2 --> FFN["Feed-forward network - expand, activation, project"] FFN --> ADD2(("+ residual")) ADD1 --> ADD2 end ADD2 --> FN[Final norm] FN --> LM[Linear to vocabulary logits] LM --> SM[Softmax - next-token probabilities]

Two sub-layers do different jobs. Attention mixes information across positions: each token gathers context from earlier tokens. The feed-forward network (FFN) processes each position independently and holds much of the model's stored knowledge; it usually contains about two thirds of a block's parameters. Residual connections let gradients flow through very deep stacks, and layer normalisation keeps activations stable. Modern models normalise before each sub-layer (pre-norm) because it trains more stably at depth, and often use RMSNorm and a gated activation such as SwiGLU.

Follow-up: Which sub-layer would you expect to dominate the parameter count, and which dominates compute at very long context?

14. Explain self-attention: what are Query, Key and Value, why do we divide by the square root of d_k, and what is causal masking?

Core

Every token is projected into three vectors. The Query says "what am I looking for?", the Key says "what do I contain?", and the Value says "what information do I hand over if selected?". Attention scores are dot products of a token's query with every key; a softmax turns scores into weights; the output is the weighted sum of values:

import math

def softmax(xs):
    m = max(xs)
    es = [math.exp(x - m) for x in xs]
    s = sum(es)
    return [e / s for e in es]

def attention(Q, K, V, causal=True):
    d = len(Q[0])
    out, weights = [], []
    for i, q in enumerate(Q):
        scores = [sum(a * b for a, b in zip(q, k)) / math.sqrt(d) for k in K]
        if causal:
            scores = [s if j <= i else float("-inf") for j, s in enumerate(scores)]
        w = softmax(scores)
        weights.append(w)
        out.append([sum(w[j] * V[j][c] for j in range(len(V))) for c in range(len(V[0]))])
    return out, weights

Q = K = V = [[1.0, 0.0], [0.5, 0.5], [0.0, 1.0]]      # 3 tokens, 2 dimensions
_, w = attention(Q, K, V)
for row in w:
    print([round(x, 3) for x in row])                   # upper triangle is exactly 0

Follow-up: What is the time and memory complexity of self-attention in sequence length, and what does FlashAttention change?

15. What is multi-head attention, and why use several heads? What are multi-query and grouped-query attention?

Core

Instead of one attention operation over the full embedding, the model splits it into h heads, each with its own smaller Q, K and V projections. Each head can specialise, for example one tracks syntax, another coreference, another position. Their outputs are concatenated and mixed by an output projection. Total compute is about the same as a single wide head, but the model can attend to different relationships at the same time.

The cost that matters in serving is the KV cache: with standard multi-head attention (MHA) every head stores its own keys and values for every past token. Multi-query attention (MQA) shares one K and V across all query heads, shrinking the cache by a factor of h at some quality cost. Grouped-query attention (GQA) is the compromise: query heads are split into groups, each group shares one K/V pair. It recovers most of MHA's quality while cutting cache size and memory bandwidth several-fold, which is why most recent open models (Llama 3 family and others) use it.

Follow-up: Why does reducing K/V heads speed up decoding even though the FLOPs barely change?

16. What is tokenization? Explain BPE, WordPiece and SentencePiece, and why not just use words or characters?

Core

A tokenizer converts text into integer ids the model can read. Words fail because vocabularies explode and any unseen word (a typo, a name, a new term) becomes unknown. Characters never fail but make sequences very long, which is expensive because attention cost grows with length. Sub-word tokenization is the compromise: common words stay whole, rare words split into reusable pieces.

• Byte Pair Encoding (BPE): start from characters (or bytes), repeatedly merge the most frequent adjacent pair into a new token until the vocabulary reaches a target size (typically 32k to 200k). Byte-level BPE (GPT family) can encode any string, so nothing is ever "unknown".
• WordPiece (BERT): similar, but chooses merges that maximise the likelihood of the training data rather than raw frequency, and marks continuation pieces with ##.
• SentencePiece: a toolkit, not a single algorithm. It treats text as a raw character stream (spaces become the symbol _), so it needs no language-specific pre-tokenisation and works well for languages without spaces. It can train BPE or a unigram language model.

Why it matters in practice: tokens set cost and context usage; they explain why models miscount letters ("strawberry") or struggle with arithmetic (numbers split unpredictably); and non-English text or code often costs more tokens per idea. If a domain uses special terms that shatter into meaningless fragments, options include adding tokens and continuing training, or fine-tuning with the existing vocabulary.

from collections import Counter

def bpe_merges(words, n_merges):
    # each word is a tuple of symbols, weighted by frequency
    vocab = Counter({tuple(w) + ("</w>",): c for w, c in words.items()})
    merges = []
    for _ in range(n_merges):
        pairs = Counter()
        for word, freq in vocab.items():
            for a, b in zip(word, word[1:]):
                pairs[(a, b)] += freq
        if not pairs:
            break
        best = max(pairs, key=pairs.get)
        merges.append(best)
        new_vocab = Counter()
        for word, freq in vocab.items():
            out, i = [], 0
            while i < len(word):
                if i < len(word) - 1 and (word[i], word[i + 1]) == best:
                    out.append(word[i] + word[i + 1]); i += 2
                else:
                    out.append(word[i]); i += 1
            new_vocab[tuple(out)] += freq
        vocab = new_vocab
    return merges, vocab

merges, vocab = bpe_merges({"low": 5, "lower": 2, "newest": 6, "widest": 3}, 8)
print(merges)
print(list(vocab))

Follow-up: A user reports the model cannot count the letter r in a word. Explain why, and how you would work around it.

17. Why do transformers need positional encoding? Compare sinusoidal, learned and rotary (RoPE) encodings.

Core

Self-attention treats its input as a set: shuffle the tokens and every token's output is unchanged apart from the shuffle. Yet "dog bites man" differs from "man bites dog". Positional information has to be injected.

• Sinusoidal (original Transformer): fixed sine/cosine waves of different frequencies added to the embeddings. No parameters, and in principle defined for any length.
• Learned absolute: a trainable vector per position (BERT, GPT-2). Simple, but cannot represent positions beyond the training length.
• Relative encodings / ALiBi: encode the distance between tokens, for example by adding a distance-based bias to the attention scores.
• Rotary Position Embedding (RoPE): rotates each query and key vector by an angle proportional to its position, so the dot product between two tokens depends only on their relative offset. It is the default in most modern open models and combines well with context-extension tricks such as position interpolation and NTK/YaRN scaling.

Why RoPE won: it captures relative position naturally, needs no extra parameters, does not add to the embedding (so it does not blur token identity), and it can be rescaled to stretch context length with a modest amount of extra training.

Follow-up: A model trained at 8k tokens is asked to read 32k. What breaks, and what are your options?

18. What is the context window, why is it limited, and what is the 'lost in the middle' problem?

Core

The context window is the maximum number of tokens a model can attend to in one request: system prompt, history, retrieved documents and the reply share it. It is not memory; the model is stateless, and 'remembering' means the application re-sends text.

Follow-up: With a 1M-token window available, when would you still use RAG?

19. Explain temperature, top-k, top-p and greedy decoding. When would you use each?

Core

The model outputs logits, one score per vocabulary token. A softmax converts them to probabilities and a decoding strategy picks the next token.

• Greedy: always take the highest-probability token. Deterministic and fast, but can be repetitive and dull, and one early mistake locks in.
• Temperature T: divide logits by T before softmax. T below 1 sharpens the distribution (more predictable), T above 1 flattens it (more diverse); T near 0 approaches greedy.
• Top-k: keep only the k most likely tokens, renormalise, sample. Fixed cutoff regardless of how confident the model is.
• Top-p (nucleus): keep the smallest set of tokens whose probabilities add up to p (for example 0.9). The set size adapts: tiny when the model is sure, larger when it is uncertain.
• Beam search: keep the best several partial sequences; used for translation and summarisation but rare in chat because it favours bland, repetitive text.

Rules of thumb: extraction, classification, code and factual Q&A use temperature 0 to 0.3; brainstorming and writing use 0.7 to 1.0 with top-p around 0.9. Change temperature or top-p, not both wildly. Even at temperature 0, hosted models are not guaranteed to be bit-for-bit identical across runs because of floating-point and batching effects.

import math, random

def sample(logits, temperature=1.0, top_k=None, top_p=None, rng=random):
    scaled = [x / max(temperature, 1e-6) for x in logits]
    m = max(scaled)
    probs = [math.exp(x - m) for x in scaled]
    total = sum(probs)
    probs = [p / total for p in probs]
    ranked = sorted(range(len(probs)), key=lambda i: -probs[i])
    if top_k:
        ranked = ranked[:top_k]
    if top_p:
        kept, acc = [], 0.0
        for i in ranked:
            kept.append(i); acc += probs[i]
            if acc >= top_p:
                break
        ranked = kept
    weights = [probs[i] for i in ranked]
    return rng.choices(ranked, weights=weights)[0]

random.seed(1)
logits = [4.0, 3.5, 1.0, 0.5, 0.1]
for T in (0.2, 1.0, 2.0):
    picks = [sample(logits, temperature=T) for _ in range(1000)]
    print("T=%.1f  share of token 0: %.2f" % (T, picks.count(0) / 1000))
print("top_p=0.8 only ever picks:", sorted(set(sample(logits, top_p=0.8) for _ in range(500))))

Follow-up: Why is 'temperature 0' still not perfectly deterministic on a hosted API?

20. Compare encoder-only, decoder-only and encoder-decoder transformers. Which is used for what?

Core

Today's general-purpose LLMs are almost all decoder-only: one simple objective scales well, the same model handles any task through prompting, and the KV cache makes generation efficient. Encoder-only models remain the workhorse for embeddings and rerankers because bidirectional context gives better representations of a whole sentence. Encoder-decoder models still shine when the input is fixed and the output is a transformation of it.

Also mention cross-attention: in encoder-decoder models the decoder's queries attend to the encoder's keys and values; multimodal models use a similar idea to let text tokens look at image features.

Follow-up: You need a model that scores how well a passage answers a query. Which architecture and why?

21. What is the KV cache? Why is the first token slower than the rest, and how do you estimate its memory?

Deep dive

During generation, attention for the new token needs the keys and values of all previous tokens. Recomputing them at every step would waste enormous work, so the KV cache stores each layer's K and V for every processed token and reuses them. Each decode step then only computes K/V for one new token and appends them.

That splits inference into two phases: prefill processes the entire prompt in parallel to fill the cache. It is compute-bound (big matrix multiplications) and determines time-to-first-token (TTFT). Decode then produces one token per step; each step must read all the model weights and the whole cache from GPU memory for very little arithmetic, so it is memory-bandwidth-bound and determines the per-token latency. That is why the first token feels slow and the rest stream quickly.

sequenceDiagram participant U as User participant G as GPU U->>G: Prompt of 3,000 tokens Note over G: PREFILL - all prompt tokens in parallel, fills KV cache (compute-bound) G-->>U: First token (TTFT) loop each new token Note over G: DECODE - read weights plus cache, emit one token (memory-bound) G-->>U: Next token end

def kv_cache_bytes(layers, kv_heads, head_dim, seq_len, batch=1, bytes_per=2):
    return 2 * layers * kv_heads * head_dim * bytes_per * seq_len * batch

per_token = kv_cache_bytes(80, 8, 128, 1)
print("per token: %.0f KB" % (per_token / 1024))
print("one 4k request: %.2f GB" % (kv_cache_bytes(80, 8, 128, 4096) / 1e9))
print("32 concurrent 4k requests: %.1f GB" % (kv_cache_bytes(80, 8, 128, 4096, 32) / 1e9))
print("same model with full MHA (64 KV heads): %.1f GB" % (kv_cache_bytes(80, 64, 128, 4096, 32) / 1e9))

Follow-up: How do PagedAttention, GQA and cache quantisation each reduce the pressure, and what does each cost?

22. What is Mixture of Experts (MoE)? How does it differ from a dense model, and what are scaling laws?

Deep dive

In a dense model every parameter is used for every token. In a Mixture of Experts model each feed-forward layer is replaced by several parallel "expert" FFNs plus a small router that picks the top few (often 1 to 2, sometimes more with shared experts) for each token. Only the chosen experts run, so total parameters can be very large while active parameters and compute per token stay small. Mixtral 8x7B, for instance, has about 47B total parameters but uses roughly 13B per token.

Follow-up: Your MoE model has good average quality but latency spikes. What would you check in the router and the serving layout?

Sources and further reading

• Vaswani et al., Attention Is All You Need (2017)
• Su et al., RoFormer: Rotary Position Embedding
• Ainslie et al., GQA: Grouped-Query Attention
• Hoffmann et al., Training Compute-Optimal LLMs (Chinchilla)
• Liu et al., Lost in the Middle
• Dao et al., FlashAttention
• AI Engineering interview questions (Outcome School, Apache-2.0)