LLM Inference-Optimization Interview Questions: From KV Cache to vLLM and Quantized Deployment

Getting an LLM to run and getting it to hold up in production are two different things — and LLM inference-optimization interview questions test the latter. Not whether you can recite concepts, but whether you can articulate trade-offs. This piece follows a real deployment path: why generation is slow (autoregression and the Prefill/Decode split) → KV Cache and long context → PagedAttention and continuous batching → FlashAttention and quantization → vLLM/TensorRT-LLM deployment and speculative decoding — each question with the answer skeleton and engineering trade-offs an interviewer wants to hear.

Compiled by a CAS PhD · senior algorithm engineer ｜ companion to the Transformer & Inference-Optimization course ｜ Updated Sep 2026

1. Why Generation Is Slow: Autoregression and the Two Stages

Why is LLM inference more complex than a single 'one forward pass' in training?Must-answer

LLM generation isautoregressive: to generate each token you feed 'the full prefix + the new token' through the model again. So 'generating 100 tokens' isn't one forward pass but 100 serial ones — latency is inherently serial, which is the root reason it's far pricier than a classifier. Say this clearly in the interview: total latency ≈ the sum over all steps; being 'fast at math' isn't enough — each step must be as short as possible and you must reduce what gets recomputed.

What are the bottlenecks of the Prefill and Decode stages, respectively?Must-answer

The same request is split into two phases:Prefillhandles the prompt, computing attention for all positions in parallel at once —compute-bound / compute-heavy(compute-bound) — the bottleneck is GPU compute, surfacing as TTFT;Decodegenerates token by token — each step adds just one token but must attend its Query against all historical Keys/Values —memory-bound / memory-heavy(memory-bound) — the bottleneck is memory bandwidth, surfacing as TPOT. In one inference step, training does 'huge parallel matrix work' while decode does 'read a lot of cache to compute very little' — which is why techniques like quantization and KV Cache compression mostly benefit decode.

2. KV Cache and Long Context

What is the KV Cache? Why is it called inference's 'necessity' and also its 'liability'?Must-answer

In attention, each historical position's K and V are determined by the full prefix up to that position; at each decode step you really only need the new token's Q to dot against all historical K/V. The KV Cache stores already-computed K/V for reuse, so you never recompute all history each step. It drops decode from 'O(seq²) recompute' to 'one incremental attention per step.' The cost: the KV Cache grows memory linearly withsequence length × concurrent requestsand with long contexts plus large batches, more than half of memory can be KV Cache — that's why it becomes a 'liability.' Add the line 'full recompute without KV Cache is the naive baseline; nobody ships that.'

Exactly how much memory does the KV Cache take? You should be able to derive the formula.Must-answer

Each token and each layer stores two copies, K and V; each copy = number of heads × dimension per head (altogether the hidden dim). So
KV Cache size ≈ 2 × batch × layers × hidden_dim × seq_len × bytes-per-param。
Example: a 7B-class model at ~32 layers, hidden 4096, FP16 (2 bytes), one request with a 4K context ≈ 32×4096×4096×2 ≈ on the order of 1GB+; concurrency or longer contexts make memory balloon fast. Deriving the formula is far more differentiating than reciting the conclusion.

What do MQA / GQA optimize?Must-answer

MHA (multi-head attention) gives each head its own K/V, so the KV Cache grows with 'heads × dim.' MQA shares one K/V set across all Q heads; GQA is the middle ground — group Q heads and share K/V per group (e.g. 8 groups). The cost is slightly less expressiveness; the payoff is that KV Cache and decode memory traffic shrink in proportion to the grouping. Almost all mainstream open models use GQA as a structural 'make KV small at training time' lever — a model-side built-in optimization, complementary to deployment-side KV Cache quantization.

3. Memory Management & Batching: PagedAttention and Continuous Batching

What problem does PagedAttention solve?Must-answer

A naive implementation pre-allocates contiguous memory 'enough for the maximum length' to every request, so an under-used KV Cache leaves fragmentation and occupies memory other requests can't use. PagedAttention borrows an OS idea —paging / virtual memory: slice the KV Cache into fixed-size blocks (pages) and allocate on demand via a page table, physically non-contiguous. Requests of wildly different lengths no longer block each other, memory utilization jumps → the same GPU fits a bigger batch → throughput steps up. This is vLLM's core selling point.

What's the difference between static and continuous batching? Why is the former wasteful?Must-answer

Static batching must wait for a whole batch to fill before computing, so early requests idle until 'departure'; mid-run, new requests can't join and finished ones still occupy slots until the batch ends. Continuous batching (a.k.a. iteration-level scheduling) checks every step: a request that emitted an EOS leaves immediately and the freed memory instantly admits a new request — the batch is dynamic each step. Payoff: lower queuing latency, higher throughput; it's the standard scheduler in frameworks like vLLM/TGI.

If you want both low latency and high throughput server-side, why is it always a trade-off?Advanced

A bigger batch uses the GPU more fully and raises throughput, but each request waits for others in the batch to advance together, so per-request TPOT grows; a smaller batch is closer to exclusive use — low latency but wasted throughput. Production typically runs 'as big a batch as the latency budget allows + dynamic scheduling,' pluspriority / preemption: pool 'interactive online requests' and 'offline batch jobs' separately, so one long offline job can't tank the online P99. Explaining 'the trade-off + pooling' clearly beats reciting parameters.

4. Kernels & Quantization: FlashAttention, GPTQ/AWQ

Why is FlashAttention fast? Which layer does it change?Must-answer

It doesn't change the algorithm's result — it changes thethe memory-access strategy: a normal implementation first computes the whole n×n attention-score matrix S and writes it back to HBM, then reads it again after softmax to weight — materializing S once already costs O(n²) memory traffic. FlashAttention doesIO-aware tiling: Q/K/V are tiled into small blocks; partial softmax is computed on-chip in SRAM and stitched together with online normalization (running max/sum), never materializing the full S. v1 proved tiling works; v2 went further, dropping S materialization and putting parallelism on the sequence dimension; v3 targets H100 with finer-grained parallelism and pipelining. The point to land:it saves HBM round-trips, not FLOPs— memory-bound scenarios like decode benefit most.

How do you choose quantization — what's the idea behind GPTQ / AWQ / GGUF?Must-answer

All are PTQ (post-training quantization) that squeeze weights to 4/8-bit. GPTQ compensates errors layer by layer using 'second-order Hessian information,' spreading quantization noise across the remaining weights; AWQ takes an activation-aware route — it stats which channels matter more to activations, protects them by importance first (no quantization or higher precision), then uses scaling to bring error down; GGUF emphasizes 'cross-device portability + sharded storage,' letting models run on CPU / low-memory / consumer GPUs (the llama.cpp ecosystem). Selection guidance: tight memory / local runs → GGUF or AWQ 4-bit; throughput-first servers commonly use W8A8 or INT4 + KV Cache quantization;quantization trades for memory and bandwidth at the cost of accuracy and a little compute overheadand before shipping you must run accuracy regression on your task set.

How do you choose among vLLM / TGI / TensorRT-LLM?Advanced

vLLM: throughput-oriented — PagedAttention + continuous batching, great ecosystem, quick to adopt, today's default first choice; TGI: from HuggingFace, integrates smoothly with the HF ecosystem and model hub, with message-queue and inference-microservice capabilities; TensorRT-LLM: deeply optimized by NVIDIA, pushing graph-compile-time optimization + kernel fusion to the extreme — suited to production where 'the model and shapes are stable and you want to squeeze every drop from one GPU,' at the cost of heavy compile and engineering overhead. How to answer: describe the requirements first (throughput / latency / many models / how often you swap models), then land on 'frameworks are essentially doing scheduling, memory, and kernel-layer optimization; choosing is a trade-off, not picking the best.'

5. Decoding Speedup: Speculative Decoding

How does Speculative Decoding 'speed up without losing quality'?Bonus

A small/fast model (the draft model) first guesses the next K tokens in one shot, then the large modelin one forward pass, verify in parallelthose K in a single forward pass; verified tokens are accepted as-is, and on the first mismatch you roll back from there. Because decode is memory-bound, verifying a few extra tokens has low marginal forward cost — the more you guess right (acceptance rate), the more you win. The key point:the result distribution matches the original autoregression(rejection sampling guarantees unbiasedness), so it isn't approximate speedup. It works when the draft and the large model 'think alike'; code/math tasks typically have lower acceptance than casual chat, so measure the gain on real traffic before adopting it.

6. Inference System-Design Problems (hands-on)

'A production LLM service has high latency / flat throughput — which layers do you inspect?'Bonus

Follow 'metrics → locate → treat → verify': ① quantify first: TTFT / TPOT / ITL, QPS, memory usage and fragmentation, queue length — tell apart whether prefill is heavy, decode is slow, or it's queuing; ② locate: long requests with slow first token → prefill or long context; high concurrency with growing TPOT → KV Cache pressure or a batching-policy issue; memory OOM/fragmentation → bring in PagedAttention / quantization / lower concurrency; ③ treat: free levers first (continuous batching + batch tuning + KV Cache quantization + driver/kernel upgrades), then paid ones (bigger-memory GPUs, multi-GPU sharding, Speculative Decoding), and only last switch the model; ④ verify: run the same task set before/after for latency distribution and throughput, guarding against 'faster but worse.' Saying 'quantify layer by layer before acting, never guess' is the core differentiator.

'Optimize inference for a read-a-very-long-document-then-Q&A product' — how would you design it?Bonus

Decompose first: this scenario is 'one deep read + many short Q&As' — don't stuff the whole book into prefill every time. Four layers: ① content side: chunk the document and retrieve (RAG), only bringing relevant passages into context, turning 'long prefill' into 'short prefill'; ② when long context is truly needed, use chunked/async prefill or long-context parallelism to compress TTFT; ③ across follow-ups, reuse the computed KV Cache (don't re-prefill the same history) and do KV Cache summarization/eviction when needed; ④ service side: quantize and pool by document length so one 100K-token request can't block all short ones. Takeaway:inference optimization should find solutions in front-end content design too — the answer isn't just tinkering inside the GPU.

FAQ