In Short
RAG controls what the model sees; fine-tuning controls how the model behaves. Long-context is the third option most teams forget: when the task fits in one document or a small corpus, you can often just load it into the prompt and skip the debate entirely. If retrieval itself is new to you, start with our plain-English walkthrough of what RAG actually does.
The quick version of every decision in this guide:
- Facts that change -> RAG. Prices, policies, product docs, and news should be retrieved at query time, not baked into weights.
- Fixed behavior, format, or tone -> fine-tune. It is for stable domains, private vocabulary, and output shape.
- Single document or small corpus -> long-context. Load it and skip the infrastructure.
- Citations and governance -> RAG. Provenance is native, and access control can be enforced at the retrieval layer.
- No labeled data -> RAG. It needs a corpus, not hundreds of labeled examples.
- High volume and latency-sensitive -> fine-tune. One call, no retrieval hop, smaller prompts.
- OpenAI self-serve fine-tuning is winding down, but open-weight fine-tuning routes remain fully alive.
Knowledge vs Behavior: The Distinction That Decides Everything
Vendors rarely agree on anything, but on this question they do. Databricks frames it as injecting knowledge at inference time versus baking domain expertise into model weights; Red Hat guidance says the same thing in different words: RAG supplements the model from external sources without modifying it, while fine-tuning adjusts weights and parameters to communicate intent. Strip the branding away and the whole decision reduces to one question - is your problem knowledge, or is it behavior?
Fine-tuning is a poor way to add facts. Training does not insert a neat row into a database inside the model; it shifts token probabilities across billions of parameters. A fine-tuned model that knows your return policy has no pointer to where that policy lives. It cannot cite the source, cannot prove the claim, and cannot be audited against it. When the policy changes, the fix is not an edit but another training run on refreshed data, plus re-evaluation - and every fact you bake in subtly competes with everything else the model has learned.
RAG inverts all of that. The document is the source of truth; the model reads it at query time, quotes from it, and can point back to it. Freshness becomes an indexing problem instead of a training problem. That is why practitioners converge on the same default: knowledge problems get retrieval, behavior problems get fine-tuning.
RAG vs Fine-Tuning vs Long-Context at a Glance
The table below compresses the decision. One 2026 wrinkle to flag first: repeated API price cuts keep eroding the long-context cost penalty, which weakens one of the traditional RAG edges for small corpora. Conceptually, though, the trade-offs have not moved.
| Dimension | RAG | Fine-tuning | Long-context |
|---|---|---|---|
| What it changes | What the model sees at query time | How the model behaves by default | What the model sees, one call at a time |
| Data needed | A corpus, no labels required | Roughly 500 to 5,000 labeled examples | The document itself |
| Freshness | Re-index; no retraining | Retrain on new data | Reload the document |
| Cost shape | Low upfront; per-call cost in longer prompts plus retrieval | Upfront training and labeling; cheaper per call at volume | Per-call token cost; 2026 price cuts weaken the RAG edge for small corpora |
| Citations | Native | None | Manual |
| Pick when | Large corpus, citations needed, corpus too big for the window | Stable domain, fixed format or tone, private vocabulary, latency or prompt compression | Single-document tasks |
Two notes on reading it honestly, because the table can look cleaner than reality. Every row hides operational work: someone owns chunking quality, embedding refreshes, and retrieval evaluation on the RAG side, and someone owns dataset curation, evaluation runs, and drift monitoring on the fine-tuning side. The 500 to 5,000 range is a working guideline, not a law: quality beats quantity, and fine-tuning on too little data tends to produce unreliable gains. And the RAG column quietly assumes you have indexing infrastructure - chunking, embeddings, somewhere to store vectors. If you are building that layer, our vector database explainer is the companion piece.
The 5-Question Decision Framework
Work through these in order; these are the questions every team hits, and most teams can stop at question two.
- Does your data change? If product docs, prices, policies, or news move on any cadence -> RAG. Baked-in facts start aging the moment training finishes, and refreshing them means a new training run plus re-evaluation, not an index update.
- Facts, or behavior? If the model already has access to the right facts and still answers with the wrong structure, tone, or format -> fine-tuning. If it simply lacks the facts -> RAG or long-context. This is the question that resolves most real decisions, so answer it before the others.
- Do you have labeled data? Fine-tuning wants roughly 500 to 5,000 high-quality input-output examples that actually demonstrate the behavior you want, reviewed by someone who knows the domain. If you do not have them and cannot create them soon -> RAG, which needs only the raw corpus.
- Are you latency-sensitive at high volume? Retrieval adds a hop: embed, search, rank, then generate, so the response arrives later than a direct call. A fine-tuned model answers in one call and can shrink prompts dramatically, which also cuts token spend. If per-call economics dominate and the domain is stable -> fine-tuning.
- Do you need citations or governance? Regulated, audited, or citation-bound products -> RAG, where provenance and access control are native. A fine-tuned model can never show its sources, because the sources dissolved into the weights during training.
One honest caveat before the flowchart: check whether you need fine-tuning at all. If the goal is just reliable machine-readable output, structured outputs may solve it, and fine-tuning is overkill for format compliance. Similarly, if the whole task lives in one document, long-context is not a compromise - it is the correct answer.
The rule of thumb: default to RAG, or to long-context for single-document tasks. Fine-tune only for behavior problems, and only when the underlying data is stable.
2026 Platform Reality: The OpenAI Fine-Tuning Wind-Down
Now the part that changed recently. OpenAI is winding down its self-serve fine-tuning platform. OpenAI's pricing page says it is no longer accessible to new users, and fine-tuned models stay available for inference until their base models are deprecated. OpenAI's deprecations page adds the dates: new organizations were cut off on May 7, 2026, and on January 6, 2027, active customers can no longer create new fine-tuning jobs.
The old per-model training prices (gpt-4.1, gpt-4.1-mini and others) no longer appear on that page, so check OpenAI's pricing page before you budget a training run.
Two more pricing realities shape the decision. OpenAI's pricing page lists GPT-6 Astra at $10 per 1M input tokens for short-context calls and $20 for long-context calls. And API prices keep falling: gpt-5.6-luna lists at $0.20 input and $1.20 output per 1M tokens on the standard tier, and its successor gpt-6-luna lists at $0.10 and $0.50. Cheap long-context input makes load-the-whole-document cheaper every quarter; for budget modeling of a full agent stack, see our breakdown of what an AI agent actually costs.
Keep the wind-down in perspective: this is one vendor closing one hosted self-serve option, not the technique dying. Fine-tuning open-weight models - self-hosted or through other providers - remains fully available. What it does change is the default: if your plan was to fine-tune on OpenAI eventually, plan around RAG or long-context now.
When the Answer Is Both: The RAFT Hybrid
There is a named technique for teams who genuinely need both: RAFT, Retrieval-Augmented Fine-Tuning, described in the RAFT: Adapting Language Model to Domain Specific RAG paper from 2024. The idea is to fine-tune the model to use retrieved documents well - training it on question-plus-document sets that deliberately include distractor documents, so it learns to ignore retrieval noise and answer from the passages that matter.
Honest positioning: RAFT is an advanced move, not a starting point. It assumes you already have a working RAG baseline, a labeled dataset of question-plus-retrieval examples, and a specific, measured gap in how the model handles retrieved context - the model fumbles multi-document questions, leans on the wrong passage, or ignores the context you paid to retrieve. It also inherits both maintenance burdens: the retrieval pipeline and the training pipeline, each with its own evaluation loop.
And note the dependency order. If your retrieval pipeline is immature, fine-tuning on top of it mostly industrializes your retrieval bugs; the model learns to work around bad passages instead of the passages getting better. For most applications the sequence is: RAG first, long-context for single-document work, fine-tuning only for durable behavior gaps - and RAFT last, if ever.
Bottom Line
RAG controls what the model sees; fine-tuning controls how it behaves; long-context just loads the document. Start there and most decisions make themselves: changing facts, citations, or missing labeled data point to RAG; a single document points to long-context; stable behavior, fixed format, or latency at volume point to fine-tuning. The OpenAI self-serve wind-down removes one convenient route but changes none of the logic - open-weight fine-tuning remains, and 2026 price cuts make retrieval and context-loading cheaper, not less relevant. Prices and dates are as published by OpenAI in October 2026 and can change - confirm on the official site. And if you are evaluating tools for whichever path you choose, our tool directory is a fast way to compare them.
