Start With Clear Instructions
Before any of the three techniques below, write down plainly what you want: the task, the output format and the edge cases. A capable model with clear instructions gets further than most teams expect, and it gives you a baseline to measure everything else against. The rest of this guide is about what to add when that isn't enough.
Sooner or later someone on the team says "we should fine-tune it". Usually the model is getting your categories, your terminology or your house style slightly wrong, and training it on your own data sounds like the thorough fix.
Sometimes it is. More often the gap closes with clearer instructions, a few good examples, or retrieving the right context at run time, all of which you can change in minutes instead of retraining for days. This guide compares the three on cost, time to ship and the kind of gap each one actually closes, and is specific about when fine-tuning earns its place: today that is mostly cost and consistency on narrow, high-volume tasks.
- Your team jumped straight to fine-tuning and spent weeks preparing training data before testing if few-shot would have worked
- Your RAG pipeline retrieves documents, but the model still gives generic answers
- You're not sure if your problem needs knowledge (RAG) or behavior change (fine-tuning)
- Stakeholders keep asking "can't we just fine-tune it?" without understanding the trade-offs
Three Levers, Three Different Problems
The three approaches solve different problems. Understanding which problem you have is the key to choosing correctly:
flowchart TD
Q["What does the model\nneed to change?"] --> K{"Need external\nknowledge?"}
K -->|"Yes"| RAG["RAG\n(Retrieval-Augmented\nGeneration)"]
K -->|"No"| B{"Need behavior\nchange?"}
B -->|"Style/format"| FS["Few-Shot\nPrompting"]
B -->|"Deep skill"| FT{"Training data\navailable?"}
FT -->|"Hundreds of examples"| FINE["Fine-Tuning"]
FT -->|"Only a few"| FS2["Few-Shot +\nSystem Prompt"]
style Q fill:#4f46e5,color:#fff,stroke:#4338ca
style RAG fill:#059669,color:#fff,stroke:#047857
style FS fill:#d97706,color:#fff,stroke:#b45309
style FINE fill:#dc2626,color:#fff,stroke:#b91c1c
style FS2 fill:#d97706,color:#fff,stroke:#b45309
Few-Shot Prompting
Include examples of desired input/output pairs directly in the prompt. Zero infrastructure required. Best for: style adaptation, format enforcement, and tasks where 3-5 examples are enough to demonstrate the pattern. Limitation: consumes context window tokens and can't encode deep domain knowledge.
RAG (Retrieval-Augmented Generation)
Retrieve relevant documents at runtime and inject them into the prompt context. Best for: questions that require specific, up-to-date, or proprietary knowledge. The model's behavior doesn't change — it just gets more context. Limitation: quality depends entirely on retrieval quality. If you retrieve the wrong documents, you get wrong answers.
Fine-Tuning
Train the model on your labeled examples to change its default behavior. Best for: consistent style/tone, domain-specific reasoning patterns, and tasks where the model needs to "think" differently, not just "know" more. Limitation: requires hundreds of quality training examples, ongoing maintenance, and careful evaluation to avoid regressions.
Trying Each One on the Same Task
Step 1: When the model misses your house style, add examples
Examples show the model your categories and your phrasing in a way instructions can't. They take minutes to add, and there's nothing to host.
var messages = new List<ChatMessage>
{
new SystemChatMessage("""
You are a code reviewer for our .NET team.
Follow our review style: direct, specific, actionable.
Reference our coding standards when relevant.
"""),
// Few-shot examples
new UserChatMessage("Review: if (x == null) return;"),
new AssistantChatMessage("""
Use pattern matching: `if (x is null) return;`
Reason: Consistent with our standard CS-0042.
Pattern matching can't be overridden, making it more reliable.
"""),
new UserChatMessage("Review: catch (Exception ex) { Log(ex); throw ex; }"),
new AssistantChatMessage("""
Use `throw;` instead of `throw ex;` — preserves the stack trace.
Wrap in a more specific catch if possible (CS-0015).
"""),
// Actual review request
new UserChatMessage($"Review: {codeToReview}")
};
Check it: Score the prompt on a fixed set of cases, with the examples and without them. Keep both numbers. They're the baseline that retrieval and fine-tuning have to beat.
Step 2: When the model lacks facts, retrieve them
Examples change how the model answers. They don't give it your documents. If the gap is knowledge, fetch the relevant text at run time and put it in the prompt.
public class CodeReviewWithContext
{
private readonly ISearchClient _searchClient;
private readonly ChatClient _chatClient;
public async Task<string> ReviewAsync(string code)
{
// Retrieve relevant coding standards
var standards = await _searchClient.SearchAsync<CodingStandard>(
code,
new SearchOptions { Top = 5, Select = { "id", "title", "rule", "example" } });
var context = string.Join("\n\n", standards.Value
.GetResults()
.Select(s => $"[{s.Document.Id}] {s.Document.Title}: {s.Document.Rule}"));
var messages = new List<ChatMessage>
{
new SystemChatMessage($"""
You are a code reviewer. Reference these coding standards:
{context}
Cite standard IDs when making suggestions.
"""),
new UserChatMessage($"Review this code:\n```csharp\n{code}\n```")
};
var response = await _chatClient.CompleteChatAsync(messages);
return response.Value.Content[0].Text;
}
}
Check it: For five wrong answers, read what was retrieved. If the right document wasn't there, the fault is in retrieval, and neither prompt work nor training will fix it.
Step 3: When both have stopped improving it, fine-tune for behaviour
Fine-tuning changes how the model responds by default. It's the slowest of the three to change, so it comes after you've measured where the other two stop.
// training-data.jsonl — Fine-tuning format
{"messages": [
{"role": "system", "content": "You are a .NET code reviewer..."},
{"role": "user", "content": "Review: var items = list.Where(x => x.Active).ToList(); foreach (var item in items) {...}"},
{"role": "assistant", "content": "Remove unnecessary `.ToList()`. LINQ's `Where` returns `IEnumerable` — iterating directly avoids allocating a new list. Standard: CS-0067 (Avoid premature materialization)."}
]}
{"messages": [
{"role": "system", "content": "You are a .NET code reviewer..."},
{"role": "user", "content": "Review: public async void HandleEvent(Event e) {...}"},
{"role": "assistant", "content": "Change `async void` to `async Task`. Callers can't catch exceptions from async void methods or await them. Only use async void for event handlers in UI frameworks. Standard: CS-0089 (Async method signatures)."}
]}
Check it: Evaluate the tuned model on the same cases as before, and on a set of general tasks too. It should beat your baseline on the first without getting worse on the second.
Decision Matrix: Quick Reference
| Criterion | Few-Shot | RAG | Fine-Tuning |
|---|---|---|---|
| Setup time | Minutes | Days | Weeks |
| Training data needed | 3-10 examples | Document corpus | Hundreds of labeled examples |
| Updates knowledge | ❌ | ✅ Real-time | ❌ Requires retraining |
| Changes behavior | ⚠️ Limited | ❌ | ✅ Deep |
| Per-query cost | Higher (long prompts) | Medium (retrieval + LLM) | Lower (shorter prompts) |
Where You Can Still Fine-Tune (Late 2026)
The decision framework above has not changed, but the options behind the fine-tuning branch have narrowed, and that should influence how early you reach for it.
- Hosted fine-tuning is shrinking. OpenAI is winding down its fine-tuning platform, and its list of fine-tunable bases doesn't include the GPT-6 models. Existing fine-tuned models stay available for inference only until their base model is retired.
- Azure still offers it, on older bases. The fine-tunable models there are the GPT-4.1 family, GPT-4o, GPT-4o mini and o4-mini, each with published retirement dates for training and for deployment. A model you fine-tune today has a known end date, so check the lifecycle page before you invest in training data.
- Open-weight models are the durable route. LoRA or QLoRA on Llama, Mistral, Qwen or Phi gives you a model that nobody can retire. You take on serving and evaluation in exchange.
The economics moved too. With economy-tier models at around $0.10 per million input tokens, million-token context windows, and cached prompt prefixes billed at a small fraction of the normal rate, a long few-shot prompt plus retrieval covers many cases that used to justify fine-tuning on cost alone. Fine-tune when you need behaviour that prompting cannot reach, a small model that must run on your own hardware, or latency that a long prompt cannot deliver. Cost per token is now rarely the reason.
1. Fine-tuning for knowledge
Fine-tuning doesn't reliably teach factual knowledge. It teaches behavior patterns. If the model needs to know your API schema or product docs, use RAG. Fine-tuning teaches it how to respond, not what to know.
2. Skipping the few-shot baseline
Always benchmark few-shot before building RAG or fine-tuning pipelines. In many cases, a handful of well-chosen examples in the prompt get you most of the quality for a fraction of the effort. It's your null hypothesis.
3. RAG with bad retrieval
RAG quality is bounded by retrieval quality. If you retrieve irrelevant documents, the model generates answers grounded in wrong context — which is worse than no context at all. Invest in chunking strategy, embedding quality, and retrieval evaluation before scaling.
4. Not evaluating regression after fine-tuning
Fine-tuning on domain data can degrade general capabilities. Always evaluate your fine-tuned model on both domain tasks AND general tasks. Track the trade-off explicitly.
Try It on Your Own System This Week
Three things to do before anyone prepares training data, about an hour in total:
- Say whether the gap is knowledge or behaviour. If the model doesn't know something, that's retrieval. If it knows and still answers in the wrong form or tone, that's examples first and fine-tuning later.
- Collect fifty labelled cases and set them aside. Never put them in a prompt or a training file. Every option gets scored on these.
- Look up the retirement date of the base model. A hosted fine-tune ends when its base model does. Find that date before you estimate what the training is worth.
Check the Exit Before You Start
Before you train anything, write down how you would leave. Hosted fine-tuned models end when their base model retires, so you are signing up to redo the work on a known date. Few-shot examples and retrieval move to a new model with a config change. That difference alone should push you towards them first.
- Start with few-shot. Always. It takes minutes, costs nothing to set up, and often gets you surprisingly far. It's your mandatory baseline.
- Use RAG when the model needs knowledge it doesn't have. Company docs, recent data, domain-specific facts — these are retrieval problems, not training problems.
- Use fine-tuning when the model needs to behave differently. Consistent tone, domain-specific reasoning, format adherence across all prompts — these are behavior problems.
- Combine them. The best production systems use fine-tuning for behavior + RAG for knowledge + few-shot for edge cases that fall through. They're complementary, not competing.
- Evaluate at each stage. Measure few-shot quality, then RAG improvement over few-shot, then fine-tuning improvement over RAG. Only add complexity when it adds measured quality.