Fine-Tuning vs Prompting vs RAG: A Practical Decision Guide for ML Engineers

Every few weeks, someone on a team asks the same question in a different costume: "Should we just fine-tune it?" Usually the real problem is a handful of bad prompts or a missing retrieval layer, and fine-tuning is the most expensive way to find that out.

This decision matters more than most teams admit. Pick prompting when you needed RAG, and you'll spend months patching hallucinations with longer and longer system prompts. Pick fine-tuning when you needed RAG, and you'll burn GPU hours retraining a model every time your knowledge base changes. Pick RAG when a fine-tuned model would have done the job for a fraction of the latency, and you've added a retrieval system, a vector database, and an entire new failure surface to a problem that didn't need one.

The three approaches solve genuinely different problems:

• Prompting steers a model's existing behavior at inference time, using nothing but the input you send it.
• RAG gives a model access to information it doesn't have, by retrieving relevant context and injecting it before generation.
• Fine-tuning changes the model itself, adjusting its weights so a behavior, style, or task becomes baked in rather than instructed.

None of them is "better." Each one optimizes for a different constraint: cost, latency, accuracy, or how often your data changes. The rest of this guide walks through how each one actually works under the hood, where each one breaks, and gives you a framework you can use the next time someone asks "should we just fine-tune it?"

1. Prompting (In-Context Learning)

How it actually works

Prompting works because large language models are trained on enough data that a huge range of tasks already live somewhere inside their weights. You're not teaching the model anything new. You're activating a capability that already exists by giving it instructions, examples, and context inside the input window, all at inference time, with zero changes to the underlying weights.

This is why it's called in-context learning. The model conditions its output on everything in the context window: your system prompt, few-shot examples, the user's query, and any intermediate reasoning you've asked for. Nothing persists once the call ends. The next request starts from a blank slate unless you resend that context yourself.

When it works well

• You're prototyping and need to validate an idea before investing in infrastructure.
• The task is general enough that the base model has likely seen similar patterns during pretraining (summarization, classification, extraction, rewriting).
• You can fully describe the task and edge cases in a reasonable number of tokens.
• Requirements change often, and retraining a model every time would be unworkable.
• You need to ship today, not after a data collection and training cycle.

Limitations

• Context window ceiling. Even with models offering large context windows, stuffing in your entire knowledge base isn't free. Larger prompts mean higher latency and higher per-call cost, and accuracy tends to degrade the more irrelevant content you cram in.
• Brittleness. Small rewordings of the prompt can shift outputs in ways that are hard to predict or test for systematically.
• No persistence. The model doesn't "remember" anything between calls. Every piece of context the model needs has to be resent every single time.
• No real knowledge injection. If the information genuinely isn't in the model's training data or your prompt, no amount of clever phrasing will produce it reliably. You'll get a fluent, confident, wrong answer instead.

Example: a prompting call

import anthropic

client = anthropic.Anthropic()

system_prompt = """You are a support ticket classifier for a SaaS company.
Classify each ticket into exactly one category: Billing, Bug, Feature Request, or Account Access.
Respond with only the category name, nothing else."""

response = client.messages.create(
model="claude-sonnet-4-6",
max_tokens=20,
system=system_prompt,
messages=[
{
"role": "user",
"content": "I was charged twice for my subscription this month and need a refund."
}
]
)

print(response.content[0].text) # Billing

Pros and cons

Pros

• No training cost or infrastructure
• Fastest path from idea to working prototype
• Trivial to iterate, version, and A/B test
• Works well for general-purpose tasks

Cons

• Doesn't scale well to large or proprietary knowledge
• Output quality is sensitive to phrasing
• Recurring per-call cost grows with prompt length
• No persistent improvement across sessions

Quick takeaway: prompting answers "can I steer the model with instructions?" It does not answer "does the model actually know this?" That second question is what RAG and fine-tuning exist for.

2. Retrieval-Augmented Generation (RAG)

Architecture overview

RAG solves the knowledge problem, not the behavior problem. Instead of hoping the model already knows something, you retrieve the relevant information from an external source and hand it to the model right before generation. The model never has to "remember" your documents. It just has to read them well, every time.

A typical RAG pipeline looks like this:

User Query
|
v
[ Embedding Model ] --> converts query into a vector
|
v
[ Vector Database Search ] --> finds nearest neighbor chunks
|
v
[ Top-K Retrieved Chunks ]
|
v


[ Reranker (optional) ] --> reorders chunks by true relevance
|
v
[ Context + Original Query ]
|
v
[ LLM ]
|
v
Response

Core components

Chunking strategy. How you split documents directly determines retrieval quality. Common approaches:

• Fixed-size chunking: split every N tokens. Simple, fast, but frequently cuts sentences or tables in half.
• Recursive/semantic chunking: split along natural boundaries (headings, paragraphs, sections) first, then fall back to size limits. Preserves meaning better, costs more compute upfront.
• Sliding window with overlap: chunks overlap by a fixed percentage so context isn't lost at the boundary, at the cost of redundant storage.

Embeddings. Each chunk is converted into a dense vector that captures semantic meaning. The choice of embedding model matters more than most teams expect: general-purpose embeddings struggle with domain-specific jargon (legal, medical, internal product terminology), and a mismatch here silently degrades every downstream retrieval.

Vector databases. Stores embeddings and performs similarity search at scale. Options like Pinecone, Weaviate, Qdrant, and pgvector each make different tradeoffs around managed infrastructure, metadata filtering, and cost at scale.

Retrieval and reranking. Pure vector similarity search isn't always enough. Hybrid search (combining dense vector search with keyword-based search like BM25) catches cases where exact terms matter. A reranker, typically a cross-encoder model, then reorders the top candidates by actual relevance to the query, since vector similarity and "relevance to this specific question" aren't always the same thing.

A simplified RAG pipeline

def rag_query(user_query: str) -> str:
# 1. Embed the query
query_vector = embedding_model.encode(user_query)

# 2. Retrieve top-k similar chunks
candidates = vector_db.search(query_vector, top_k=20)

# 3. Rerank for true relevance
reranked = reranker.rank(user_query, candidates)
top_chunks = reranked[:5]

# 4. Build context and call the LLM
context = "\n\n".join(chunk.text for chunk in top_chunks)
prompt = f"""Answer the question using only the context below.
If the answer isn't in the context, say you don't know.

Context:
{context}

Question: {user_query}"""

return llm.generate(prompt)

Failure cases

• Chunking destroys context. Splitting a table or a multi-step procedure mid-way produces chunks that look relevant but are missing the information that actually answers the question.
• Confident hallucination on bad retrieval. If the wrong chunks get retrieved, the model often still produces a fluent, confident answer using whatever context it was given. The failure looks like a hallucination, but the root cause is upstream in retrieval.
• Stale indexes. If your vector database isn't kept in sync with the source of truth, you get answers that are correct for documents that no longer exist.
• Context dilution. Retrieving too many chunks, or chunks that are only tangentially related, buries the genuinely relevant information and hurts accuracy rather than helping it.

Tradeoffs

RAG adds real infrastructure: an embedding pipeline, a vector store, and an extra network hop on every single query, which adds latency that prompting alone doesn't have. In exchange, you get a system whose knowledge can be updated by editing documents, not retraining a model, and that scales to far more information than any context window could hold.

Comparison Table

3. Fine-Tuning

What actually changes

Fine-tuning updates the model's weights based on a curated dataset, so a behavior becomes part of the model rather than something you have to instruct every time. This is the right tool when the problem is how the model behaves, not what it knows. Fine-tuning a model to know your company's return policy is the wrong move; that policy will change next quarter and the model won't. Fine-tuning a model to always respond in your brand's tone, or to reliably output a specific structured format, is exactly what fine-tuning is good at.

Full fine-tuning updates every parameter in the model. It gives maximum flexibility but requires enormous GPU memory and compute, and carries real risk of catastrophic forgetting, where the model loses general capabilities while overfitting to the new task.

LoRA (Low-Rank Adaptation) freezes the base model and trains small, low-rank adapter matrices injected into specific layers. This cuts trainable parameters by orders of magnitude, which means dramatically lower memory and compute requirements, while still capturing most of the benefit for narrow tasks.

QLoRA takes this further by quantizing the frozen base model to 4-bit precision before training the LoRA adapters on top of it. This is what makes fine-tuning large models feasible on a single high-memory GPU instead of a multi-GPU cluster.

When it's necessary versus overkill

Necessary when:

• You need consistent style, tone, or output format across thousands of calls, and prompting alone can't hold that consistency reliably.
• The task requires domain-specific reasoning patterns that general pretraining doesn't cover well (specialized code styles, niche technical domains, structured extraction with strict schemas).
• Latency is critical and you can't afford the extra round trip that retrieval adds.
• You have enough high-quality labeled data to actually move the needle (generally a few hundred to several thousand well-curated examples, depending on task complexity).

Overkill when:

• The actual problem is a knowledge gap, not a behavior gap. Use RAG instead.
• A handful of well-written few-shot examples in the prompt would solve it just as well.
• Your underlying data or requirements change frequently. Every change means another training run.
• You don't have an evaluation pipeline in place yet. Fine-tuning without rigorous evals is how teams ship regressions without noticing.

A high-level training pipeline

from peft import LoraConfig, get_peft_model
from transformers import AutoModelForCausalLM, AutoTokenizer, TrainingArguments, Trainer

base_model = AutoModelForCausalLM.from_pretrained("base-model-id", load_in_4bit=True)
tokenizer = AutoTokenizer.from_pretrained("base-model-id")

lora_config = LoraConfig(
r=16,
lora_alpha=32,
target_modules=["q_proj", "v_proj"],
lora_dropout=0.05,
task_type="CAUSAL_LM"
)

model = get_peft_model(base_model, lora_config)

training_args = TrainingArguments(
output_dir="./fine-tuned-adapter",
per_device_train_batch_size=4,
num_train_epochs=3,
learning_rate=2e-4,
logging_steps=10,
save_strategy="epoch"
)

trainer = Trainer(
model=model,
args=training_args,
train_dataset=train_dataset,
eval_dataset=eval_dataset
)

trainer.train()

Infrastructure and cost considerations

The training run itself is often the smallest line item. The real cost lives in curating a clean, representative dataset, building an evaluation harness that catches regressions before deployment, and the iteration cycle when the first fine-tune doesn't behave the way you expected. Budget for multiple training runs, not one.

Risks

• Catastrophic forgetting, where the model becomes very good at the new task but noticeably worse at everything else, especially with small or narrow training sets.
• Overfitting, where the model memorizes training examples instead of generalizing the underlying pattern.
• Stale knowledge, since anything the model "learns" during fine-tuning is frozen at training time, exactly like its original pretraining.
• MLOps overhead, since you now need versioning, rollback plans, and continuous evaluation for a model artifact, not just a prompt string in a config file.

4. Real-World Engineering Considerations

Latency. Prompting alone is the fastest path at small scale, since it's a single inference call. RAG adds a retrieval round trip, typically tens to a few hundred milliseconds depending on your vector database and reranking setup, on top of generation time. Fine-tuning removes that overhead entirely once deployed, but the lead time to get there (data prep, training, evaluation) is measured in days or weeks, not milliseconds.

Cost. Prompting and RAG both carry recurring per-call token costs, and RAG's costs are usually higher per call since retrieved context adds tokens to every request. Fine-tuning flips this: a large upfront training cost that gets amortized across every future inference call, which can make it cheaper than RAG at high volume for narrow, stable tasks.

Scaling. Prompting scales as far as your API rate limits allow. RAG's scaling bottleneck shifts to your vector database and retrieval infrastructure as your document collection grows. Fine-tuning scales cleanly at inference time, but every behavior or knowledge update means another training cycle.

Observability. None of these approaches are "ship and forget." You need a golden evaluation set for prompting changes, retrieval precision and recall metrics for RAG, and regression suites for every fine-tuned checkpoint. Without this, you're shipping changes based on vibes, and vibes don't catch the edge case that breaks in production three weeks later.

Real failure case patterns worth knowing:

• A support bot without RAG confidently quoting outdated pricing because that information lived only in its pretraining data, not in any document it could check.
• A model fine-tuned on a few hundred narrow examples losing general reasoning ability on anything slightly outside that distribution.
• A RAG system that looked correct in testing but silently degraded in production because chunking split a critical table mid-row, and nobody had a retrieval-quality metric to catch it.

5. Decision Framework

Use these as practical rules, not absolutes:

• If you need the model to know information that changes often → use RAG, not fine-tuning.
• If you need consistent tone, style, or output format at scale → fine-tune; prompting alone tends to drift under load.
• If you're prototyping or validating an idea → start with prompting. Add complexity only when prompting demonstrably fails.
• If latency is critical and the task is narrow and stable → fine-tune to remove the retrieval round trip.
• If your data changes weekly or monthly → use RAG; retraining that often is not sustainable.
• If you have limited labeled data → prompting or RAG, not fine-tuning. Fine-tuning on too little data tends to overfit.
• If hallucination on private or proprietary documents is your main problem → RAG, since the root cause is missing knowledge, not bad behavior.
• If you need both grounding in current data and tightly controlled behavior → combine RAG and fine-tuning. Fine-tune the model to use retrieved context well and follow your output format, and let RAG handle the knowledge.

Decision flowchart

Need fresh or private knowledge?
|
-------------+-------------
| |
YES NO
| |
RAG Need consistent behavior/style at scale?
|
-------------+-------------
| |
YES NO
| |
Fine-Tuning Prompting

Common Mistakes

• Misusing RAG as a search engine substitute. RAG is for grounding generation in retrieved facts, not for replacing a proper search product. If users need to browse and filter documents, that's a search UI problem, not an LLM problem.
• Overusing fine-tuning to teach facts. Fine-tuning is expensive and slow to update. Using it to inject knowledge that should live in a retrievable document store means you're paying training costs to solve a problem RAG solves more cheaply and keeps current.
• Poor chunking with no validation. Splitting documents on a fixed token count without checking what actually ends up inside each chunk is one of the most common, most invisible causes of bad RAG performance.
• Shipping without an evaluation pipeline. Whichever method you choose, if you can't measure whether a change made things better or worse, you're not engineering, you're guessing with extra steps.

Future Trends

• Agent-based systems that combine tool use, retrieval, and narrow fine-tuned sub-models for specific steps, rather than relying on one monolithic approach for the entire pipeline.
• Hybrid architectures where a fine-tuned retriever or reranker improves RAG quality, and a fine-tuned generator handles output formatting and tone on top of retrieved context.
• Smaller, specialized models distilled from larger ones for narrow, high-volume tasks, cutting inference cost dramatically once a task is well understood enough to no longer need a general-purpose model's full capability.

Closing Thoughts

None of these three methods is a default. Each one trades cost, latency, accuracy, and maintenance burden differently, and the right call depends on what's actually broken: a knowledge gap, a behavior gap, or just an underdeveloped prompt. Get specific about which one you're solving before you reach for infrastructure or a training run.

If you're building or scaling production AI systems and want a second pair of eyes on the architecture, you can check out my work and projects here: anber.me