RAG vs. Fine-tuning vs. Prompt Engineering: Interview Trade-offs

Hello! In our last session, we designed a complex multi-agent system and briefly touched upon the trade-off between using RAG for knowledge retrieval and tools for taking action. That discussion was a perfect entry point for today's topic, which lies at the very heart of modern AI system design.

The learning outcome for this lesson is to evaluate and articulate the trade-offs between RAG, fine-tuning, and prompt engineering in interview scenarios. This isn't just a theoretical exercise; "When would you use RAG versus fine-tuning?" is one of the most frequently asked questions in AI engineer interviews. Your ability to provide a nuanced, well-reasoned answer, backed by an understanding of business and engineering constraints, is a key differentiator that signals senior-level thinking.

By the end of this 60-minute lesson, you will have a clear framework for making these critical design decisions and the vocabulary to justify them confidently under interview pressure.

The Three Pillars of LLM Customization

Before we can compare these techniques, let's establish a clear, shared understanding of what they are and why they're necessary. Why can't we simply use a powerful base model like GPT-4 or Claude 3 Opus for every task?

The Stanford CS230 lecture on Agents provides an excellent overview of the inherent limitations of vanilla pre-trained LLMs.

As the video highlights, base models are powerful but general-purpose. They often lack the specific, up-to-date, or proprietary knowledge a business needs. They can also struggle with consistency, style, and providing verifiable sources for their answers. To solve these problems, we have three main strategies, or "pillars" of customization:

1. Prompt Engineering: Changing the model's behavior by crafting better instructions within the input prompt. This is the fastest, cheapest, and most iterative approach. It includes techniques like zero-shot, few-shot, and Chain-of-Thought (CoT) prompting, which you're already familiar with.
2. Retrieval-Augmented Generation (RAG): Giving the model access to external knowledge at inference time. This is about changing what the model knows by retrieving relevant information from a database and adding it to the prompt as context.
3. Fine-Tuning: Updating the model's internal parameters (weights) on a curated dataset before inference. This is about changing how the model behaves by teaching it a new skill, style, or domain-specific language.

This infographic provides a concise summary of the pros and cons of each strategy.

A Framework for Decision-Making

Now for the crucial part: how do you choose? In an interview, a vague "it depends" is a weak answer. A strong answer presents a clear decision framework based on specific criteria.

The following article from the CallSphere Blog provides an excellent, battle-tested framework that is directly applicable to interview scenarios.

From that reading, we can extract a set of key decision axes:

• Primary Goal: Are you trying to teach the model a new skill/behavior or give it new knowledge?

  • Behavior/Skill/Style: Fine-tuning (e.g., teaching it to always respond in iambic pentameter or generate SQL queries).
  • Knowledge: RAG (e.g., answering questions about your company's latest internal documents).

• Data Freshness (The "Knowledge Velocity" Test): How often does the underlying information change?

  • High Velocity (daily, weekly): RAG is the clear winner. Updating a vector database is fast and cheap.
  • Low Velocity (static, monthly, quarterly): Fine-tuning becomes a viable option, as the cost and time of retraining are less frequent.

This next infographic visualizes this "Knowledge Velocity Test" beautifully.

• Traceability & Hallucination Risk: Does the answer need to be verifiable?

  • High Need for Cites/Low Hallucination Tolerance: RAG is superior because it grounds the answer in retrieved documents, which can be cited as sources.
  • Lower Need for Cites: Fine-tuning can be used, but you accept a higher risk of ungrounded, "creative" responses.

• Cost, Latency & Effort: What are your operational constraints? This is where you demonstrate senior-level thinking by considering the total cost of ownership, not just API fees.

The article "Essential AI Toolkit for PMs" provides fantastic real-world examples of how these costs play out. While written for Product Managers, its focus on business and engineering trade-offs is exactly what's expected from a top-tier AI Engineer.

The Pragmatic Escalation Path

Combining these frameworks, a pragmatic, defensible strategy emerges—one you should articulate in an interview. You don't jump to the most complex solution; you escalate based on need.

1. Start with Prompt Engineering: Always begin here. Use few-shot examples, Chain-of-Thought, and clear instructions. This is your cheapest, fastest lever. You might find it's all you need.
2. Layer on RAG for Knowledge: If prompting alone is insufficient because the model lacks specific, changing, or proprietary knowledge, introduce RAG. This addresses the "knowledge gap" without the high cost of training.
3. Use Fine-Tuning as a Last Resort: Only consider fine-tuning when both sophisticated prompting and RAG fail to meet your requirements for a specialized behavior, style, or task. Fine-tuning is justified when the business cost of failure is high and you've exhausted simpler methods. The author of the PM article puts it perfectly: fine-tuning is "capitulation to complexity."

The "Nuance That Gets You Hired" section in the CallSphere blog post also introduces an important hybrid approach: RAG with a fine-tuned embedding model. This gives you the best of both worlds—domain-specific retrieval with a general-purpose generator—at a fraction of the cost of fine-tuning a large language model.

Articulating Your Choice: The Interview Scenario

Knowing the framework is half the battle. The other half is articulating it.

Let's use the "Mistakes to Avoid" section from the Johnny Mai article as a guide. It perfectly contrasts weak, generic justifications with strong, evidence-based ones.

Now, let's put it all together.

Interviewer: "We want to build an AI assistant for our junior software engineers. It should answer questions about our internal codebase, which is very large and updated hundreds of time a day. Would you use prompt engineering, RAG, or fine-tuning?"

Your Model Answer:

"That's an excellent and highly relevant problem. My approach would be to start with the simplest, most effective solution and escalate only as necessary, based on performance and constraints.

Phase 1: RAG as the Foundation.
Given the two primary requirements—a knowledge base that is internal (proprietary codebase) and extremely high-velocity (hundreds of updates per day)—RAG is the immediate and most logical starting point. Fine-tuning is a non-starter here because it would be impossible to retrain the model frequently enough to keep it current.

So, I would design a RAG pipeline. The ingestion process would listen to repository updates, automatically chunking new code, generating embeddings, and indexing them into a vector database with metadata like file path, author, and commit date.

Phase 2: Sophisticated Prompt Engineering.
On top of the RAG pipeline, I'd use advanced prompt engineering. The prompt template would instruct the LLM to act as a senior staff engineer, to use the retrieved code snippets to formulate its answer, and to cite the source files for every claim it makes. I would also use few-shot examples in the prompt to guide the model on the desired tone and format for code explanations.

Phase 3: Fine-Tuning as a Targeted Optimization.
I would only consider fine-tuning after measuring the performance of the RAG system and identifying a specific, persistent failure mode. For example, if we found that the base model consistently struggles to understand our company's specific Go coding patterns or frequently misunderstands our internal library APIs, even with RAG context, then I would evaluate fine-tuning.

However, instead of fine-tuning the entire large generator model, I would first investigate fine-tuning the embedding model. By creating a small dataset of (code-related question, relevant code chunk) pairs, we could fine-tune the retriever to become an expert at finding the right context. This is often far more cost-effective and can provide a significant performance lift. Full fine-tuning of the generator model would be the last resort, justified only if the business impact of the remaining errors outweighs the significant costs of data collection and training."

This answer demonstrates that you understand the trade-offs, think in terms of a practical, phased approach, and are conscious of cost and operational complexity—all hallmarks of a senior AI engineer.

Conclusion

Today, we've dissected one of the most fundamental decisions in applied AI engineering. You now have a robust mental model for evaluating and choosing between prompt engineering, RAG, and fine-tuning.

Key Takeaways:

• No "Best" Tool: The optimal choice is always driven by specific constraints: the goal (knowledge vs. behavior), data freshness, traceability requirements, and total cost of ownership (including latency and maintenance).
• The Pragmatic Path: Always start with Prompt Engineering, escalate to RAG for external knowledge needs, and treat Fine-Tuning as a powerful but expensive last resort for specialized tasks or styles.
• Articulate Your Reasoning: In an interview, don't just state your choice. Justify it by explaining the trade-offs you're making and outlining the phased approach you would take to validate your decision. Show that you think like an engineer who builds sustainable, cost-effective systems.

In our next lesson, we will shift our focus from technical design to personal presentation. You will learn how to structure answers to behavioral questions by strategically leveraging the systems we've designed and the concepts we've discussed to build a compelling narrative about your experience as an AI Engineer.