RAG or Fine-Tuning: Choose by the Failure
You're building an assistant over company data. When would you use RAG, when would you fine-tune, and when would you combine them?
what they're testing: The interviewer is testing whether you can diagnose a knowledge problem versus a behavior problem and choose an architecture from measurable trade-offs.
Use RAG when answers depend on proprietary, extensive, or frequently changing information. At request time, retrieval selects relevant passages and adds them to the model's context. You can refresh the corpus or index without retraining the model, and the application can expose the retrieved passages as evidence. That does not guarantee that an answer is supported by them.
Use fine-tuning when the main failure is stable behavior, such as formatting, tone, classification, or instruction following, after prompting and examples fall short. Supervised fine-tuning trains on input-output examples and updates model weights. It requires curated training data and evaluation. It is a poor default for changing facts because incorporating new facts requires new training and still does not provide request-time evidence.
The approaches can work together. A support assistant can retrieve the latest policy while a fine-tuned model applies a consistent response format. Start with representative evals and a prompt-only baseline, then test the smallest change that addresses the measured failure. For RAG, measure retrieval quality as well as end-to-end grounding, latency, and cost. For fine-tuning, measure task quality against the baseline and include training, maintenance, and serving costs. Neither approach eliminates hallucinations.
Where people slip
the tempting wrong answer, and what's actually true
Fine-tuning is the best way to load company documents into the model.
Fine-tuning is a poor default for loading documents. RAG can retrieve proprietary or changing documents at request time, and the corpus can be refreshed without retraining the model.
RAG eliminates hallucinations because every answer is grounded in retrieved text.
RAG can improve grounding, but retrieval can return irrelevant context and the model can still generate unsupported claims.
RAG and fine-tuning are competing approaches, so a system should choose one.
They can be combined, with RAG supplying current facts and fine-tuning shaping stable task behavior.
If they push further
How would you prove that RAG is the right choice?
Build an eval set from real queries and measure retrieval relevance, answer grounding, latency, and cost against a prompt-only baseline.
When does fine-tuning become worth its operational cost?
Use it when repeated eval failures point to a stable behavior or narrow task, and a high-quality example set produces enough accuracy, latency, or cost improvement to justify training and maintenance.
What would you monitor in a hybrid system?
Separate retrieval failures from generation failures, then track end-to-end quality, source grounding, latency, cost, and drift after document or model updates.
Sources
- Microsoft Learn: Augment large language models with RAG or fine-tuning ↗learn.microsoft.com
- AWS: Architecting a Successful Generative AI Proof of Concept ↗docs.aws.amazon.com
- OpenAI API: Model optimization ↗developers.openai.com
Now answer it yourself.
Reading a strong answer is easy. Producing one under pressure is the skill the interview tests. Gapmap grades your answer against the same bar an interviewer would.
beta
The interviewer part is in the works.
The diagnostic, personal maps, and AI mock interviews are being finished right now. The notes stay free either way. Leave an email and you'll get the first-cohort invite, plus a month of Pro when it opens.