A self-check map for anyone preparing for LLM / AI algorithm-role interviews: from Transformer internals to LLM pretraining, large-model fine-tuning (SFT / RLHF/DPO alignment), RAG, Agent, and large-model inference optimization (KV-Cache & quantization deployment). Every section opens the matching systematic course and online tools.
Why Self-Attention instead of RNN/LSTM?
Three reasons: ① Long-range dependency — an RNN passes hidden state step by step, so gradients vanish over distance; Self-Attention models every position against every position directly, with an O(1) hop path. ② Parallelism — attention has no sequential dependency, so the whole sequence trains in parallel; an RNN can only proceed step by step. ③ Expressiveness — attention weights are decided dynamically by content relevance (content-based), while the RNN's position prior is fixed.
Why does Multi-Head Attention use multiple heads?
A single head averages the attention distribution, so it finds it hard to attend to several kinds of dependency (syntactic relations, coreference, semantics) at once. Multiple heads project into different subspaces, and each head learns one relational pattern; the outputs are concatenated and linearly transformed, broadening the model's ability to attend to information at different positions from different subspaces.
Why add positional encoding? What makes RoPE better than absolute positional encoding?
Attention itself is orderless (a set operation), so position has to be injected. Absolute positional encoding adds a position vector into the embedding; RoPE encodes relative position into the inner product via rotation matrices — attention scores then depend only on relative displacement, which generalizes better to longer contexts. That's why RoPE is now the mainstream choice in LLaMA, Qwen, and friends.
Why train with teacher forcing / cross-entropy instead of directly optimizing the final answer?
Autoregression decomposes sequence modeling into per-token maximum likelihood (teacher forcing: at each step, predict the next token from the true preceding context), which parallelizes fully and gives a stable objective; directly optimizing the whole output discriminatively offers no per-token supervision signal. The downside is a train/inference mismatch (exposure bias), usually mitigated with curriculum learning and decoding-time techniques.
How do SFT, pretraining, and instruction fine-tuning differ?
Pretraining learns the general language distribution (massive unlabeled data, next-token). SFT learns 'answer as instructed' behavior from high-quality instruction–response pairs. Alignment (RLHF/DPO) further matches human preferences — telling apart what the model 'can say' from what it 'should say.'
Why do LoRA / QLoRA save GPU memory?
Freeze the original weights W and learn only a low-rank update ΔW=BA (B ∈ R^{d×r}, A ∈ R^{r×k}, r ≪ d), which slashes the memory and the trainable parameter count that go through forward/backward passes. QLoRA further quantizes the base weights to 4-bit (NF4) with double quantization, saving even more — fine-tuning a 7B model fits on a single consumer GPU.
Walk through the rough RLHF flow (reward model + PPO). Why is DPO now used more often?
RLHF: ① train an RM — score several responses to the same prompt to learn human preference; ② optimize the policy with PPO using the RM as the reward, plus a KL penalty to keep it from drifting off the SFT model. RM labeling is hard and PPO training is unstable, which is why DPO appeared — it writes preference directly as an implicit reward of the policy, needing only a single classification-style training pass on preference data. Stable and cheap.
Why is RAG needed? Which scenarios fit it?
LLM knowledge has a cutoff, can hallucinate, and cannot cover private / vertical knowledge. RAG feeds external knowledge to the model through 'retrieve → assemble → generate,' suiting scenarios where knowledge changes fast, sources must be cited, and factual accuracy matters (customer service, Q&A, internal documents). Cases needing low latency or strong reasoning, or with no external material, may not be worth it.
When RAG performs poorly, which links are usually to blame?
Mostly the retrieval side: poor chunking, embeddings that don't match the domain, Top-K pulling irrelevant passages, missing reranking; and on the generation side: an over-long context diluting attention, or the model being misled by irrelevant passages. In engineering, first run offline evaluation (recall / hit rate / answer accuracy) to decide whether it's a recall problem or a generation problem, then fix accordingly.
What is the core capability of an Agent? How is it different from simply calling an LLM?
Agent = LLM + tool calling + planning + memory + an execution loop: the model decides 'which tool to call next / with what arguments,' then decides further from the tool's output — so it can finish multi-step real tasks. A plain chat only does 'text in → text out' and cannot operate external systems.
What are the common Agent paradigms?
ReAct (reason–act–observe interleaving), Function Calling / Tool Calling (structured tool interfaces), LangGraph-style graph / state-machine orchestration, and multi-agent collaboration (planner–executor–reviewer). Interviews often ask about: error recovery in multi-step tasks, tool-argument hallucination, and guardrails for runaway Agent loops.
What is KV-Cache? How much memory and compute does it save respectively?
Autoregressive decoding needs, at each step, only the new token's Q to attend against historical K/V. Caching the already-computed K/V and reusing it avoids recomputing the whole history at every step — a large compute saving. The cost is memory that grows with context (roughly 2 × layers × head dim × bytes-per-param per token), which is why KV-management optimizations like PagedAttention / vLLM exist.
How do the bottlenecks of the Prefill and Decode phases differ?
Prefill is compute-bound (many matrix multiplies run in parallel); Decode is memory-bound (one token per step, bandwidth-limited). So the levers differ: prefill improves with operator / tensor parallelism; decode relies on KV-Cache, batching (continuous batching), quantization to cut bandwidth, and speculative decoding to reduce serial steps.
What problem does each of FlashAttention / GPTQ / AWQ solve?
FlashAttention: IO-aware tiling avoids writing the full matrix back to HBM, saving memory and accelerating attention. GPTQ / AWQ: both quantize weights to 4-bit to cut memory and bandwidth; GPTQ compensates error layer-by-layer with second-order information, while AWQ protects important channels by activation sensitivity — and inference needs no dequantization back to high precision.
They cover the shared topics for LLM algorithm roles / application engineers / inference-optimization roles. Agent/RAG-application oriented candidates should focus on Sections 3–4; training & alignment, Section 2; deployment, Section 5. If your target is an algorithm role, work through the Full-Stack Program systematically by direction.
The checklist lets you self-check 'which topics you missed,' but interviews want 'depth in your answers + a project you can explain clearly.' The course pairs 200+ real problems with detailed explanations, hand-derivation drills (Attention math, complexity), and full mock interviews — turning the checklist into performance in the room.
Follow 「兔老板工作室」 on Xiaohongshu and DM 「资料」 to receive a free PDF of curated big-tech interview questions — free forever, no tricks, no phone number required.
Two 30-minute in-depth resume sessions + one 60-minute full mock interview; written feedback is delivered within 24 hours after the interview (per-question scoring + reusable talking points). Schedule over WeChat — see 'Resume + mock interview 1-on-1.'
The topics are region-agnostic. Two paths by goal: returning to China for domestic big-tech → plan backward from the autumn / spring hiring timeline, finish the modules, and practice Chinese technical explanations; staying local (North America / Singapore / Japan, etc.) → the underlying LLM / Agent / RAG knowledge is shared, so additionally prep your project experience to tell fluently in English (or the local language). Before the crunch you can book a 1v1 mock interview, scheduled remotely over WeChat.
📚 Free long-form series:LLM algorithm-role high-frequency checklist · 10 RAG Interview Questions · Agent system design thinking · LLM fine-tuning & alignment points · LLM inference optimization points · Big-tech talent programs compared · Résumé & project pitfall guide · AI Jobs for Chinese Students in the US · AI Infra free practice question bank