Retrieval-Augmented Generation (RAG) is a must-know area for LLM-application and algorithm roles in 2026. This article follows one real build pipeline: why RAG → chunking → embeddings → retrieval → reranking → multi-hop → hallucination & evaluation. For each question you get “the answer skeleton interviewers want to hear + engineering pitfalls.”
What problem does RAG solve? How does it relate to long context and fine-tuning?Must-answer
RAG supplies generation with “currently trustworthy knowledge” from external retrieval, solving three problems: knowledge freshness, private knowledge, and hallucination. Versus the other two paths: long context stuffs documents into the window — intuitive, but cost grows linearly with tokens and models get “lost in the middle” of long text; fine-tuning writes knowledge into weights — expensive, slow to update, hard to trace. In practice they complement each other: RAG owns freshness and provenance, long context is for “one deep read,” and fine-tuning is for style, format, and domain tone.
When is RAG actually the wrong approach?Advanced
① Strong-reasoning problems (math, multi-step logic) — retrieval can’t fetch a “reasoning process,” so focus on reasoning models / CoT instead; ② Low latency with high QPS — one retrieval pass plus long context raises latency noticeably; ③ The answer simply isn’t in the external store (purely internalized knowledge) — retrieval only adds noise. Ask “where does the answer live” before deciding to retrieve, rather than slapping RAG on everything.
How should you chunk? Fixed-length, recursive, or semantic?Must-answer
There’s no universal answer — chunk so that “one chunk can independently answer a class of questions.” For heavily structured documents (Markdown headings/tables/sections), cut along the structure; for ordinary prose use recursive character splitting with an overlap of 10–20% to preserve context boundaries; for code, split by function/class. Small chunks retrieve precisely but can lack context; large chunks carry full context but add noise and hit model-window and embedding-length limits. Tune chunking together with the downstream question types — don’t guess.
Why is hybrid search (vectors + BM25 full-text) so common?Advanced
Dense vectors excel at semantic similarity but go wrong on proper nouns / IDs / exact terms (contract numbers, model numbers, names); BM25 excels at exact-term matching but doesn’t understand semantics. They complement each other — common practice is RRF (Reciprocal Rank Fusion) or weighted score fusion, then a reranker does fine-grained ordering. In production a typical flow is “vector recall Top200 + BM25 Top200 → fuse → Rerank Top20.”
Why split retrieval into two stages (recall + rerank)?Must-answer
The recall stage must be broad (don’t miss candidates across a huge corpus), so it uses efficient vector/inverted indexes to take Top-K; the rerank stage must be precise (computing fine-grained relevance over tens of candidates), so it can use a stronger Cross-Encoder — which fully interactively encodes the query with each doc. It’s accurate but expensive per call, so it only runs over Top-K. Broad first, then precise — balancing quality and cost.
How do you retrieve for multi-hop / complex questions?Advanced
Simple RAG does one retrieval and one generation. For complex questions you either decompose into sub-questions (query rewriting / routing to sub-retrievals), iterate retrieval (supplement based on what’s been generated), or go GraphRAG-style — build an entity-relation graph first, then pull the relevant subgraph. The interview key is explaining “why a single retrieval isn’t enough” and the rationale for your decomposition strategy (question type, knowledge granularity).
If relevant content is retrieved, why do models still answer wrong or hallucinate?Advanced
Three root causes: ① The context never reached the model — recall missed the key passage (a recall problem); ② It was provided but the model ignored it — the context was too long and diluted by irrelevant passages, or the prompt didn’t demand “answer only from the material”; ③ The material itself is self-contradictory. Fix it methodically: first check offline whether “the answer is in the Top-K passages” to attribute blame to retrieval or generation, then improve recall, or add grounding/citation constraints and post-retrieval denoising.
How do you make answers traceable and verifiable?Advanced
Force citations in generation: in the prompt require the model to cite “chunk ids” and answer only from the given material; surface citation anchors in the frontend. During evaluation, measure whether the citations actually support the answer (attribution / groundedness), and treat “sentences with no source” as a hallucination signal.
How do you evaluate a RAG system in production?Bonus
Evaluate in layers: the retrieval layer uses recall, hit rate, MRR/NDCG (is the answer in Top-K); the generation layer uses answer correctness (including LLM-as-Judge scoring and human sampling). In engineering, maintain at least one evaluation set of query → expected document → expected answer, and run an offline regression every time you change chunking / embeddings / reranking. Don’t rely on “it looks better by eye.”
How do you monitor degradation in production?Bonus
Instrument three things: retrieval hit rate (which document a user opened / follow-up asked), the share of answers without citations, and bad-case feedback loops (negative signals). After a knowledge-base update, rerun recall regression on the new corpus so a batch of new chunks doesn’t silently degrade quality.
It’s layered from “must-answer” to “advanced” to “bonus”: foundations, chunking & hybrid search, two-stage recall + rerank, multi-hop retrieval, hallucination & grounding, and the differentiator — evaluation and monitoring. It largely covers applied / AI-engineer roles; for senior roles, try to pair each question with a real production story of your own.
Audit yourself against this article first, then pick one or two areas (hybrid retrieval or the eval system, say) and actually build a small system. In the interview, talk through the pitfalls you hit and the numbers you compared — far stronger than reciting definitions. 1-on-1 coaching includes end-to-end project walkthroughs and 200+ real questions explained in depth.
Add me on WeChat (ID: pyao2015) and send 「资料」 for a free PDF of selected big-tech interview questions — free forever, no sign-up. The LLM evaluation platform and the Agent evaluation learning material are part of the Songshu AI Project Lab membership; add WeChat pyao2015 to activate and walk the evaluation pipeline.
The content is location-independent — use this article as a layered self-check first. If you are targeting Chinese employers: drill 「what RAG systems have you built and what broke」 into a pitch you can deliver in both Chinese and English, and time it to the domestic hiring windows. If you are targeting local roles: being able to talk chunking, reranking and evaluation at engineering granularity counts for more. If you want project walkthroughs and full mock interviews, add me on WeChat (pyao2015) and we can run 1-on-1 remotely.
📚 Free long-form series:LLM algorithm-role high-frequency checklist · Agent system design thinking · LLM fine-tuning & alignment points · LLM inference optimization points · Big-tech talent programs compared · Résumé & project pitfall guide · AI Jobs for Chinese Students in the US · AI Infra practice question bank (members)