Fine-Tuning, LoRA and Alignment (RLHF, DPO)

When to fine-tune, LoRA and QLoRA, GPU memory maths, RLHF vs DPO vs GRPO, data preparation, catastrophic forgetting and evaluation of a tuned model.

• Advanced
• 16 min read
• 11 questions

Before this lessonLesson 6: AI Agents, Tool Use and MCP

Fine-tuning questions test whether you know both the mechanics (what changes in the weights, how much memory it takes) and the judgement (most problems should not be solved by fine-tuning). Expect the classic "RAG or fine-tune?" plus at least one memory-arithmetic question.

Lead with restraint: the best answer usually starts with "first I would try a better prompt and retrieval, because they are cheaper to change" and then explains precisely when tuning earns its cost.

The 11 questions in this lesson

67. When should you fine-tune an LLM, and when should you use prompting or RAG instead?
68. Explain pre-training, supervised fine-tuning (SFT) and preference optimisation. How do they fit into one pipeline?
69. What is LoRA, how does it work, and what do rank and alpha control?
70. What is QLoRA and how does it let you fine-tune a large model on a single GPU?
71. Do the GPU memory maths: fine-tuning a 7B model with full fine-tuning, LoRA and QLoRA.
72. Explain RLHF: reward model, PPO and the KL penalty. What is reward hacking?
73. What is DPO, and why did it displace PPO-based RLHF at many labs? What are RLAIF, Constitutional AI, RLVR and GRPO?
74. How do you prepare a dataset for fine-tuning? How much data do you need and what goes wrong?
75. What is catastrophic forgetting and overfitting in fine-tuning, and how do you prevent them?
76. Which hyperparameters matter in fine-tuning, and how do you evaluate a fine-tuned model?
77. What is knowledge distillation, and how do you merge or manage multiple LoRA adapters?

67. When should you fine-tune an LLM, and when should you use prompting or RAG instead?

Core

Fine-tuning updates the model's weights on your examples. It is best at changing behaviour: a consistent tone or persona, a strict output format, a specialised skill (classification, extraction, a house coding style), following a domain's conventions, or making a smaller, cheaper, faster model good at one narrow task. It is poor at teaching a model many facts: they are stored unreliably, cannot be cited, cannot be access-controlled and go stale.

flowchart TB S[Quality not good enough] --> P{Clear, specific prompt with examples?} P -- no --> P2[Improve prompt, add few-shot and a schema] P -- yes --> K{Missing or changing knowledge?} K -- yes --> R[Add retrieval - RAG] K -- no --> B{Behaviour, format or style still wrong?} B -- yes --> F[Fine-tune - SFT, then preference tuning] B -- no --> M[Try a stronger model or better evaluation] P2 --> S R --> S

Also weigh the ongoing costs of tuning: building and cleaning data, training runs, evaluation, hosting a custom model, and redoing it when the base model changes. It is common to combine them: RAG for facts, light fine-tuning for behaviour.

Follow-up: A team wants to fine-tune on 10,000 internal documents so the model 'knows the company'. What do you tell them?

68. Explain pre-training, supervised fine-tuning (SFT) and preference optimisation. How do they fit into one pipeline?

Core

flowchart LR PT["Pre-training<br/>trillions of tokens<br/>next-token prediction"] --> SFT["Supervised fine-tuning<br/>instruction and response pairs"] --> PO["Preference optimisation<br/>RLHF, DPO, RLAIF"] --> RL["Optional: RL with verifiable rewards<br/>math, code, tools"] --> DEP[Deployed assistant]

• Pre-training builds language ability and world knowledge from raw text by predicting the next token. It is by far the most expensive stage and produces a base model that only continues text.
• SFT / instruction tuning trains on curated (prompt, ideal response) pairs so the model follows instructions and adopts a chat format. It teaches how to respond.
• Preference optimisation uses comparisons (response A is better than B) to push the model toward helpful, honest and harmless behaviour: RLHF with a reward model, or direct methods such as DPO.
• Reinforcement learning with verifiable rewards trains reasoning and agentic skills where correctness can be checked automatically (unit tests, exact math answers).

Interviewers like the point that SFT mostly unlocks and shapes capabilities already present from pre-training, and that low-quality SFT data can hurt more than a small amount of excellent data.

Follow-up: Why can't you just skip SFT and go straight from a base model to preference optimisation?

69. What is LoRA, how does it work, and what do rank and alpha control?

Core

Full fine-tuning updates every weight, which is memory-hungry and produces a full-size copy per task. LoRA (Low-Rank Adaptation) freezes the pre-trained weight matrix W (size d x k) and learns the update as the product of two small matrices: W' = W + (alpha / r) x B A, where A is r x k, B is d x r and the rank r is small (4 to 64). Only A and B are trained. B starts at zero so training begins exactly at the base model. The hypothesis, borne out in practice, is that fine-tuning updates have low intrinsic rank.

• Rank r: capacity of the update. Higher r can learn more but costs more parameters and can overfit; 8 to 32 covers most tasks.
• Alpha: a scaling factor (the update is multiplied by alpha / r). A common convention is alpha = r or 2r; it acts like a learning-rate multiplier for the adapter.
• Target modules: which layers get adapters. Attention projections (q, v, often all of q, k, v, o) are the classic choice; adding the MLP layers usually improves quality at higher cost.

Benefits: 100 to 1,000 times fewer trainable parameters, far less optimiser memory, small adapter files (megabytes) that can be swapped per customer or task on one shared base model, and the adapter can be merged into W for inference so there is no extra latency.

def lora_params(d, k, r):
    return r * (d + k)

d = k = 4096                       # one attention projection in a 7B-class model
full = d * k
for r in (4, 16, 64):
    p = lora_params(d, k, r)
    print("r=%-3d LoRA params %9d  = %.2f%% of the %d full matrix params" % (r, p, 100 * p / full, full))

layers, mats_per_layer = 32, 4     # adapt q, k, v, o in each of 32 layers
r = 16
total = layers * mats_per_layer * lora_params(d, k, r)
print("whole model, r=16 on 4 matrices/layer: %.1f M trainable parameters (vs about 7,000 M)" % (total / 1e6))

Follow-up: Why is B initialised to zero and A to random values, rather than both random?

70. What is QLoRA and how does it let you fine-tune a large model on a single GPU?

Deep dive

The result: a 65B model that needed hundreds of gigabytes to fine-tune fully fits on a single 48 GB GPU, and a 7B model can be tuned on a consumer card with 12 to 24 GB, with quality close to 16-bit LoRA in the original paper's experiments.

Follow-up: Which parts of a QLoRA run are in 4-bit, which in 16-bit, and which are trained?

71. Do the GPU memory maths: fine-tuning a 7B model with full fine-tuning, LoRA and QLoRA.

Deep dive

Memory during training has four parts: weights, gradients, optimiser states, activations. For full fine-tuning with mixed precision and Adam, the standard accounting is about 16 bytes per parameter: 2 (bf16 weights) + 2 (bf16 gradients) + 4 (fp32 master weights) + 8 (Adam's two fp32 moments). For 7 billion parameters that is roughly 112 GB, before activations, so full fine-tuning needs several 80 GB GPUs or sharding (FSDP or ZeRO).

• LoRA: frozen weights in bf16 = 14 GB; gradients and optimiser states exist only for the small adapters (tens of millions of parameters, well under 1 GB); plus activations. Roughly 16 to 24 GB depending on sequence length and batch size, so it fits on one 24 GB card with gradient checkpointing.
• QLoRA: frozen weights in 4-bit = about 3.5 GB (plus quantisation constants); adapters and their states small; plus activations. Roughly 6 to 12 GB.
• Activations grow with batch size x sequence length x layers; gradient checkpointing recomputes them to save memory at about 30 percent more compute; small micro-batches with gradient accumulation keep the effective batch size.

params = 7e9
GB = 1e9

full = params * (2 + 2 + 4 + 8) / GB                      # bf16 weights + bf16 grads + fp32 master + Adam m and v
print("full fine-tune (weights+grads+optimiser): %.0f GB, before activations" % full)

adapter_params = 40e6
lora = params * 2 / GB + adapter_params * (2 + 2 + 4 + 8) / GB
print("LoRA  : %.1f GB (frozen bf16 base + tiny trainable adapters), before activations" % lora)

qlora = params * 0.5 / GB * 1.1 + adapter_params * 16 / GB   # 4-bit base plus about 10% quantisation overhead
print("QLoRA : %.1f GB (4-bit base + adapters), before activations" % qlora)

Follow-up: Full fine-tuning does not fit on your 8 x 80 GB node with a 70B model. What are your options?

72. Explain RLHF: reward model, PPO and the KL penalty. What is reward hacking?

Deep dive

flowchart LR A["1. SFT model<br/>trained on demonstrations"] --> B["2. Collect comparisons<br/>humans rank pairs of responses"] B --> C["3. Train a reward model<br/>predicts which response is preferred"] C --> D["4. Optimise the policy with PPO<br/>maximise reward minus KL penalty"] A -. reference model .-> D

The reward model is trained on human comparisons to output a scalar score for any response. The policy (the LLM) is then optimised with a reinforcement-learning algorithm, classically PPO, to produce responses that score highly. A KL penalty against the original SFT model keeps the policy from drifting into gibberish that merely games the reward.

Follow-up: Your RLHF-tuned model got wordier and more sycophantic. Diagnose it.

73. What is DPO, and why did it displace PPO-based RLHF at many labs? What are RLAIF, Constitutional AI, RLVR and GRPO?

Deep dive

Where PPO-style online RL still wins: when you can generate fresh samples and score them (the policy explores beyond the fixed dataset), when the reward is verifiable, and for large improvements in reasoning. DPO is offline: it is limited to the quality and coverage of the preference dataset and can overfit or lower the likelihood of both responses.

Follow-up: You have 50,000 preference pairs and a limited budget. PPO, DPO or something else, and why?

74. How do you prepare a dataset for fine-tuning? How much data do you need and what goes wrong?

Core

1. Define the task and the target behaviour with an evaluation set before collecting data.
2. Collect real inputs from your product (with consent and privacy review) and write or curate the ideal outputs with domain experts; use a strong model to draft and humans to review, never blind synthetic data.
3. Format correctly: use the model's chat template and the exact structure used at inference (system prompt, roles, special tokens); mask the loss to train only on the assistant's response.
4. Clean: remove duplicates and near-duplicates, contradictory labels, PII, low-quality or unsafe responses, and anything that leaks evaluation data.
5. Balance and diversify: cover edge cases, refusals, different lengths and phrasings; do not let one category dominate.
6. Split into train, validation and a held-out test set, deduplicated across splits.

Common failure modes: inconsistent labelling between annotators (measure agreement), training on the model's own low-quality outputs, format mismatch between training and serving, and synthetic data that quietly copies a teacher model's mistakes or breaks its licence terms (check the terms of any model used to generate training data).

Follow-up: Your fine-tuned model repeats fragments of training examples verbatim. What went wrong in the data or training?

75. What is catastrophic forgetting and overfitting in fine-tuning, and how do you prevent them?

Core

• Use parameter-efficient methods (LoRA) which change little of the base model and can be turned off.
• Lower the learning rate (for full fine-tuning typically 1e-5 to 2e-5; LoRA tolerates about 1e-4 to 2e-4) and train for few epochs (1 to 3); stop when validation loss stops improving.
• Mix in general data: replay a portion of instruction or pre-training-style data alongside your domain data.
• Regularise: weight decay, dropout in adapters, early stopping, smaller rank.
• More diverse data rather than more epochs on the same examples.
• Evaluate on two sets: your task metric and a general-capability benchmark, so you notice regressions immediately.

Follow-up: After domain fine-tuning your model has forgotten how to answer in JSON. How do you recover?

76. Which hyperparameters matter in fine-tuning, and how do you evaluate a fine-tuned model?

Core

Follow-up: Validation loss keeps falling but human raters say the model got worse. What do you investigate?

77. What is knowledge distillation, and how do you merge or manage multiple LoRA adapters?

Deep dive

Follow-up: You serve 200 customers, each with a custom behaviour. Full fine-tunes or adapters? Justify with costs.

Sources and further reading

• Hu et al., LoRA: Low-Rank Adaptation of LLMs
• Dettmers et al., QLoRA
• Ouyang et al., Training language models to follow instructions with human feedback (InstructGPT)
• Rafailov et al., Direct Preference Optimization
• Bai et al., Constitutional AI
• Shao et al., DeepSeekMath (introduces GRPO)
• AI Engineering interview questions (Outcome School, Apache-2.0)