LLM Fine-tuning & Alignment Interview Topics: From LoRA/QLoRA to RLHF/DPO

Fine-tuning is the inevitable path for 'turning a general model into a business model,' and a topic algorithm-role interviews almost always ask. This article follows one real project chain: first decideshould I fine-tune or not(where the boundary with RAG and prompting lies) → how to prepare SFT instruction data and read the loss → why LoRA/QLoRA is the default and how it works → the alignment evolution from RLHF to DPO and each one's pitfalls → full-parameter vs PEFT training-engineering trade-offs, and finally answer skeletons for two fine-tuning system-design problems.

Compiled by a CAS PhD · senior algorithm engineer ｜ companion to the LLM Fine-tuning & Alignment course ｜ Updated Sep 2026

1. Should You Fine-tune? The Boundary Between Fine-tuning, RAG, and Prompting

The most common opening question: when should you fine-tune, and when are prompting / RAG enough?Must-answer

Three questions to decide: ① knowledge-shaped or capability-shaped—— if the model simply 'doesn't know' some fact / private material, feeding it via RAG is cheaper, updatable anytime, and carries no forgetting risk; only when the model's 'behavior / format / reasoning paradigm is wrong' do you need fine-tuning to change behavior. ② whether it must stably follow instruction templates—— for example, if you need it to always output in a fixed JSON schema or stick to a strict script, fine-tuning is more stable than prompting every time. ③ whether compute and maintenance costs are affordable—— fine-tuning needs data, training machines, evaluation regression, and maintaining multiple model versions. Nail the split 'knowledge rides on RAG, behavior on fine-tuning, lightweight tries on prompting' in your answer, and add 'stress-test prompting + RAG up to their limits first; only escalate to fine-tuning when you've confirmed it's a capability gap' — that's the differentiator.

Is the claim 'fine-tuning lets the model learn new knowledge' correct?Must-answer

Half right — and that's exactly the trap. Fine-tuning can indeed 'write' high-frequency facts into the weights, but at a high price: a piece of knowledge needs many repetitions to stick, it can pollute existing abilities (catastrophic forgetting), and new knowledge can't be updated quickly (changing one fact means retraining). So the production consensus isNew factual knowledge goes through RAG; fine-tuning mainly changes 'behavior / style / format / domain language'; to genuinely give the model a domain, turn the domain knowledge into RAG corpus and use fine-tuning to learn 'how to answer with that corpus' — rather than memorizing it directly.

2. SFT & Data: Quality, Format, and Reading the Loss

Which points of SFT (supervised fine-tuning) data preparation are most often overlooked?Must-answer

It's not 'more is always better.' Instead: ① quality > quantity: a few thousand high-quality aligned examples usually beat hundreds of thousands of scraped dirty examples — dirty data (off-topic or wrong answers) directly teaches the model bad behavior; ② diversity > repetition: flooding one template makes the model learn only one phrasing — cover input shapes, difficulty, and edge cases; ③ loss over answer tokens only: the instruction / user input is usually masked — don't let the model 'memorize the question'; ④ format is capability: if you need JSON out, examples must end in JSON — even the EOS habit gets learned. Calling out 'mask the prompt and take cross-entropy only over the answer' is a bonus detail.

Which signals do you watch during training? Does a falling loss mean it's good?Must-answer

The loss should fall, but don't watch only it: ① train loss down but validation loss not → overfitting; first check data duplication / learning rate too high / too many epochs; ② loss oscillating or stuck → a learning-rate, batch, or data-mix problem; check for dirty data first; ③ fine-tuning must guard against 'forgetting the base model's abilities'—— hold out a 'general-capability' eval set (general Q&A / reasoning / format) separate from the fine-tuning task set and run it every epoch, so domain ability goes up without general ability falling; ④ if business metrics can be measured directly, sample-evaluate during training (accuracy / pass rate) — when loss and business metrics disagree, trust business. In one sentence:the loss is a training signal; business evaluation is the acceptance bar。

3. Parameter-Efficient Fine-tuning: LoRA / QLoRA

Why does LoRA make it feasible to 'train large models'? What is the low-rank assumption?Must-answer

Full-parameter fine-tuning updates and stores the whole weight set (a 7B model means tens of GB of gradients / optimizer states). LoRA's core assumption:the fine-tuning weight update ΔW is low-rank, so no full-rank update is needed. ΔW is therefore decomposed into two small matrices A×B (A projects down, B projects up); during trainingfreeze the original weights and train only the two small matrices A/Bwith a rank r usually in the tens. Payoffs: trainable parameters drop from everything to ~0.1%–1%; multiple tasks each keep a small adapter that you stack or swap at inference instead of storing whole models; after training the adapter can be merged back into the weights with zero extra inference cost. Add the line 'why a low-rank update is enough — fine-tuning is a small directional correction relative to pretraining, so it needs no full-rank representation change.'

Beyond LoRA, what more does QLoRA save? What is NF4 for?Must-answer

QLoRA makes it possible tofine-tune a large model on a single GPU with three tricks: ① quantize thethe frozen base weightsweights to 4-bit (NF4: normal float, a 4-bit format bucketed along the normal distribution, which wastes less precision than uniform INT4 on real weight distributions), dropping base-model memory from tens of GB to a few GB; ② during training usedequantized computation—— weights are stored in 4-bit and dequantized to higher precision for the forward pass, while the LoRA adapter itself trains in BF16 to keep precision; ③ double quantization + paged optimizer: even the quantization constants are compressed, and optimizer states spill to CPU memory when needed — further savings. The takeaway: QLoRA balances 'low-precision base + high-precision adapter training' — a single 24GB GPU can fine-tune 7B/13B-class models; it's the default starting point for teams without a cluster.

LoRA's rank r — which layers it applies to and how to tune it?Advanced

A rank too small lacks expressiveness; too large slides back toward 'full-parameter memory / overfitting.' Practical default: start at r = 8/16 and raise it for complex tasks or large data; the adapter goes onthe attention layer's Q/K/V/O projectionsas the baseline; whether to also add MLP layers and how many should be decided by comparing on task evaluation, not by gut. Differentiator: explaining 'r scales with task complexity / data size, r is set with an eval set, and post-merge adapter error is negligible' beats reciting defaults.

4. Alignment: From RLHF to DPO

Why RLHF after SFT? What do its four steps do?Must-answer

SFT only 'learns how humans answer' — it doesn't directly optimize 'which answer is better,' and it has only seen labeled examples, never how open-ended generations get judged. RLHF models 'good' explicitly: ① first train areward modelreward model — learn a scorer from humans ranking several answers to the same prompt; ② use PPO to have the policy model generate answers and update against the reward model's score — but directly maximizing reward invites gaming, so you introduce areference model + KL penaltyreference model + KL penalty: the policy can't drift too far from the SFT / reference model, stopping it from emitting gibberish or sycophancy for a high score; ③ iteratively collect new preferences and retrain the reward model. Interview point: RLHF is essentially 'turning human preference into an optimizable reward signal' — PPO is just the vehicle; the KL constraint is the key to preventing reward hacking.

Why does DPO need 'no reward model and no PPO'? What's the cost?Must-answer

The key DPO derivation: in RLHF's 'KL-constrained reward-maximization' objective, the optimal policy and the reward function have anclosed-form solution— so the reward function can be eliminated and rewritten directly as 'a classification loss on preference data': make chosen responses' log-prob higher than rejected ones, with the gap bounded by the reference model. Result: no reward model to train, no RL sampling / PPO stability tuning — train directly on a preference dataset, far simpler engineering. Costs: ① no explicit reward signal, so complex feedback 'beyond ranking' is hard to handle; ② more sensitive to preference-data quality; ③ still prone to 'over-optimization' — if the chosen response itself is flawed, it gets amplified. Interview takeaway: DPO is the 'closed-form equivalent of RLHF,' not a completely different thing — articulating that relationship is the biggest plus.

What does reward hacking mean? How do you prevent it?Advanced

The reward model is an approximation, and the policy will find 'high-score but actually bad' answers — classically: the longer and more sycophantic, keyword-stuffed filler the model thinks 'humans like to hear,' while real quality flatlines or drops. Four-layer defense: ① KL constraintkeep it from drifting too far from the reference; ② make the reward model good and broadly covered — don't just pat yourself on the back within the training distribution; ③ throughout training keep watchingreal business / sampled human evaluationto see 'reward score went up — did the business metric follow?' rather than trusting the RM score alone; ④ cap over-optimization and roll back to an intermediate checkpoint when needed. Voluntarily saying 'the reward signal is an approximation; you must guard against gaming during training and regress against real acceptance' is what someone with alignment experience sounds like.

5. Training Engineering: Full-Parameter vs PEFT, Memory, ZeRO

How do you budget memory / cost for full-parameter fine-tuning vs PEFT? When is full-parameter worth it?Must-answer

A 7B in full-parameter BF16 training has its memory dominated by weights + gradients + optimizer states (Adam keeps 2 states per parameter), plus activations — usually ~70–140GB+; a single 24GB GPU can't hold it at all → you must go distributed withZeRO / tensor parallelism / pipeline parallelismsomething like ZeRO / tensor parallel / pipeline parallel; LoRA freezes the base and trains only the adapter, so optimizer states cover ~0.1–1% of parameters and single-GPU or small setups suffice. Full-parameter is worth it when: data is large, the task sits far from the base distribution, you need maximum expressiveness, and you have a cluster plus tuning budget; otherwise start with PEFT. Differentiator: clearly explaining ' training statestraining states (gradients + optimizer) — not the weights — are the memory hog, and freezing parameters removes those states' beats reciting ZeRO terminology.

What is the idea behind ZeRO? How is it different from 'model parallelism'?Advanced

ZeRO is another way to attack 'doesn't fit in memory': in data parallelism every GPU stores the full parameters + optimizer states, which is hugely redundant. ZeRO takes that whole setshard states per cardand shards it across cards — each GPU stores one shard (Stage 1 shards the optimizer, Stage 2 adds gradients, Stage 3 shards parameters too, all-gather when used) — memory drops linearly with card count while compute barely changes. The difference from model parallelism: model parallelism 'slices each layer's weights so different GPUs compute different shards' (intra-layer tensor split), whereas ZeRO is 'every GPU holds all layers but doesn't duplicate states — communication is traded for memory.' Saying 'ZeRO decouples storage from compute and trades communication for memory — it's not carving the model into pieces each computing its own' shows you've actually used it.

6. Fine-tuning System-Design Problems (hands-on)

'Fine-tune and ship a vertical customer-service model' — from which layers do you answer?Bonus

Follow 'data → training → evaluation → feedback': ① decide: first decide whether it's 'missing knowledge' (domain FAQ goes through RAG) or 'wrong behavior' (script / refusals / format) — fine-tune only the latter; don't pour the whole library into the weights; ② data: clean historical support conversations into (instruction, expected answer) pairs — quality control + dedup + diversity sampling, desensitize sensitive / private info, and have humans rewrite poorly-labeled ones first; ③ train: start with LoRA/QLoRA, mask the prompt, run few epochs at a small learning rate, and hold out a general-capability eval set against forgetting; ④ evaluate: business metrics (resolution rate / hand-off rate / sampled satisfaction) + domain-benchmark accuracy + refusal rate, paired with LLM-as-judge rough screening + sampled human review; ⑤ feed back: periodically harvest online bad cases back into the training set, closing the loop 'discover → add data → retrain → ship.' Takeaway: shipping a fine-tuned model = data ops + an evaluation loop — training itself is just one step in the middle.

'The model confidently makes things up (hallucination) / fails to refuse when it should — how do you suppress that with alignment?'Bonus

First separate two kinds of illness: ① factual hallucination— at its core it's often 'making things up even when unsure'; the first tool is RAG / grounding so it 'looks things up when it can and says it doesn't know when it can't,' then add SFT/DPO to teach 'express uncertainty / refuse when confidence is low'; ② failing to refuse / answering out of bounds— that's a boundary behavior; directly build paired data of 'violating / out-of-scope request → graceful refusal' and use DPO to train 'refusing is the better choice.' Don't expect one alignment pass to fix everything: watch 'compliant-refusal rate vs false-refusal rate on normal questions' with an eval set — over-alignment turns into refusing everything, so balance it. Giving the division 'facts via external tools, behavior via alignment, false refusals via evaluation tuning' is what shipped experience sounds like.

FAQ