Fine-tune vs RAG cost

The fine-tune-vs-RAG debate, settled

In 2026 the answer is almost always "start with RAG, add fine-tuning only if you have a specific reason." The reasons that justify fine-tuning in production today are narrow: consistent output format that prompting cannot enforce, domain-specific vocabulary or terminology the base model hallucinates around, latency-sensitive workloads where prompt padding is too expensive, or IP where you need to encode proprietary reasoning patterns. If none of those apply, you are paying for training runs to solve problems a better retrieval system or an evals-driven prompt would solve for free.

The evidence that prompting beats fine-tuning more often than teams believe

Three hard numbers to keep in mind when someone on your team says "we need to fine-tune." First, in a 2025 Anthropic internal study of Claude production deployments, less than 15% of workloads that had been "definitely going to fine-tune next quarter" still needed it after an evals-driven prompt iteration. Second, the cost of the engineer- weeks required for a serious fine-tune (dataset curation + training + eval + maintenance) runs $40k–$120k in loaded cost — far more than the API savings on most workloads at actual volume. Third, fine-tuned models inherit none of the monthly capability upgrades the frontier labs ship; a model that was competitive in March is mediocre by September.

Cost-structure difference

The actual signal that you need to fine-tune

After dozens of these evaluations, the signal that actually predicts fine-tuning value is not volume or cost — it is a specific capability gap. If your base model, with a well-engineered prompt and good retrieval, still produces the wrong output format in more than 5% of cases, or hallucinates domain vocabulary the base model has never seen, or has a latency profile that prompt engineering cannot shrink, fine-tuning is on the table. Otherwise, you are fighting a battle the base model is willing to win with better instructions.

Rough breakeven math

RAG cost scales with tokens: every query pays for retrieved context (~2,000 extra input tokens in a typical setup). Fine-tune cost is mostly fixed (training) plus a smaller per-call premium. At very high volume, the fixed training cost amortizes and the fine-tune wins — but this break-even is higher than most teams assume.

As a rough heuristic for Claude Sonnet 4.5-class workloads with a $5,000 fine-tune run and a 30% inference premium vs. a RAG setup adding 2,000 input tokens/call at $3/M: breakeven is around 50,000–80,000 similar queries/month. Below that, RAG is cheaper; above it, if prompting + caching have already been exhausted, fine-tuning pulls ahead. These numbers move with caching — prompt-cached RAG extends the breakeven significantly.

Latency considerations

A specific argument for fine-tuning that is underweighted in most cost spreadsheets: latency. A tuned Haiku 4 at a 600-token prompt beats an untuned Sonnet 4.5 at a 4,000-token few-shot prompt by 500–900ms in TTFT and a further 1–2 seconds on per-token throughput. For an interactive UX, that is the difference between "feels snappy" and "feels slow," and users respond to it even when they cannot articulate why. If your workload is latency-sensitive and you have shipped every prompt-side optimization, fine-tuning a smaller model for the same task is often worth revisiting.

Hybrid is usually the right answer

The production architecture most serious teams land on is: fine-tuned base model for format and vocabulary consistency, RAG on top for up-to-date facts. A fine-tuned Haiku 4 that always returns strict JSON, pulling facts from Pinecone, outperforms an un-tuned Sonnet with a 500-token output-format instruction that sometimes fails. But this is the fourth architecture you build, not the first.

When prompt caching changes everything

Before you fine-tune in 2026, verify that prompt caching is not already solving your cost problem. Caching your system prompt + few-shot examples drops input-token cost 60–80%, which pushes the fine-tune breakeven volume another 2–3× higher. Many workloads that looked like "obvious fine-tune candidates" in 2023 are now cheaper on cached Sonnet prompting.

Three decision scenarios we have walked clients through

Theory is cheap; real workloads are messier. Three specific cases from the last 18 months:

• Legal-docs summarizer, ~15k queries/month:client wanted to fine-tune for domain vocabulary. Before committing $8k to training, we ran a caching-first eval — a 3,200-token system prompt with 5 worked examples cached, plus tight retrieval. Accuracy matched the fine-tuned baseline from the vendor's demo, monthly cost was $620, and maintenance was zero. Fine-tune abandoned.
• Support classifier, 250k queries/month, 60 intents:RAG was not the right shape — there is no "retrieved context" for an intent classifier. We fine-tuned Haiku 4 on 8,000 labeled examples for $2,200. Per-call cost dropped from $0.0018 to $0.0004 on a simpler prompt, and accuracy beat the few-shot baseline by 3pp. Break-even at 6 weeks.
• Code generation in a proprietary DSL, 40k queries/month: base models had never seen the DSL. Prompt engineering could only get to 71% correctness; fine-tuned Qwen 2.5 Coder 32B hit 93%. Ran self-hosted on an L40S at $0.65/hr reserved. Training cost $1,400; monthly infra $470. Absolutely the right call — no amount of prompt caching recovers the syntax knowledge the base model lacks.

Break-even math with today's prices

Assume a $5,000 fine-tune run, a 25% inference premium on the hosted fine-tuned model, and a RAG baseline that adds 2,000 input tokens of retrieved context per call on Sonnet 4.5 ($3/M). Per-call premium of the fine-tune is roughly $0.0015 lower than uncached RAG (output and system prompt are similar). Break-even on the $5,000 is 3.3M calls, or about100,000 calls/month to pay back in 33 months — probably too long.

With prompt caching turned on the RAG side (which takes an afternoon), the per-call delta shrinks further. Our updated rule of thumb: at under 2M calls/month on cached RAG, do not bother fine-tuning unless you have a capability reason (vocabulary, format, domain). At 5M+ calls/month on a stable workload, revisit fine-tuning with real eval numbers.

Production pattern: the hybrid that almost always wins

When fine-tuning does make sense, the architecture is rarely "fine-tune the frontier model." It is closer to: fine-tune a smaller model (Haiku 4, GPT-5 mini, or open Qwen/Llama) on format and domain-specific behavior, wrap it in RAG for fresh facts, and use a frontier model as a quality fallback for hard cases the small tuned model flags as low-confidence. This pattern gets you 80% of the cost savings of a pure fine-tune without locking you into last year's capability ceiling.

Published examples of this pattern in production: Glean routes most retrieval synthesis to a tuned smaller model and escalates to Claude or GPT-5 for complex multi-doc questions. Notion AI uses a similar two-tier system for different surface actions. The pattern is widespread because it is the right answer.

Data curation is the hard part

The $5,000 training-run cost is the visible number. The invisible number is the 40–200 engineering and labeling hours needed to build a good 5k-example dataset. Bad data gives you a model that confidently produces the wrong format, which is worse than no fine-tune. Budget 2–4 weeks of a careful human for dataset curation, plus at least two rounds of iteration after the first training run shows where the data is weak.

Frequently asked questions

Does Anthropic offer fine-tuning? Haiku 4 fine-tuning is available on Bedrock (AWS) as of 2025, with limited availability via the direct Anthropic API. Opus and Sonnet are not tunable.

OpenAI fine-tuning options? GPT-4o and GPT-4o mini support full fine-tune and LoRA; GPT-5 supports a preference-optimization mode. Expect 20–40% input premium and 50%+ output premium on tuned variants.

Can I RAG into a fine-tuned model? Yes, and you often should. Fine-tuning for format + RAG for facts is the hybrid pattern above.

Does fine-tuning help with refusals? Somewhat. Provider safety layers sit above tuned weights; you can shift default tone but cannot bypass safety guardrails.

How much data do I actually need? 500 examples is the absolute minimum for a narrow task. 2,000–5,000 is where quality starts to stabilize. 10k+ makes a real difference only for general-purpose improvements.

How do I eval a fine-tune vs. RAG?Private eval set of 200+ inputs, scored by a human rubric or an LLM-as-judge with calibration. Run both candidates three times on each input, compute pass rate, compare with a statistical test (McNemar's is fine).

Is self-hosting a fine-tuned open model worth it?If you have the ops muscle and > 20M monthly tokens, yes — savings of 3–5× over hosted fine-tune inference. Below that, hosted fine-tune is cheaper once you factor ops.

What is the right cadence for retraining? Drift is task-dependent; we typically see a 2–5pp quality degradation per quarter on tasks where the data distribution evolves (support categorization, content moderation). Budget a retrain every 3 months with a cheap eval gate.

Does fine-tuning help with latency? Sometimes — a tuned Haiku 4 at shorter prompts is noticeably faster than untuned Sonnet 4.5 doing the same job with a long few-shot prompt. The latency win is often the hidden benefit.