Build vs buy: RAG vs fine-tuning

# Build vs buy: RAG vs fine-tuningfine-tuningFine-tuning adapts a pre-trained model to a specific task or domain by continuing training on a smaller, targeted dataset, improving accuracy and style for that use case.Voir la définition complète →

In 2023, a mid-sized European insurer stood up a GenAI "center of excellence" and committed €4M to fine-tuningfine-tuningFine-tuning adapts a pre-trained model to a specific task or domain by continuing training on a smaller, targeted dataset, improving accuracy and style for that use case.Voir la définition complète → an open-weight model on its claims history. Eighteen months later, the fine-tuned model was quietly deprecated. The problem wasn't the model, it was that their claims policies changed quarterly, and every change required a retraining cycle they couldn't afford. A three-week RAG pilot, built by two engineers, ended up carrying the production load. The €4M bought them an expensive lesson: they had solved a knowledge-freshness problem with a behavior-shaping tool.

This is the most consequential and most misunderstood build decision you will make in enterprise GenAI. The vendor ecosystem is financially incentivized to sell you the heaviest option. Your job is to match the mechanism to the actual problem, and to know when the answer is "none of the above, just write a better prompt."

The three mechanisms, and what each actually changes

Strip away the marketing and there are three levers, arranged by cost and complexity.

PromptingPromptingPrompt engineering is the practice of designing and refining text inputs to guide large language models toward accurate, relevant, and reliable outputs.Voir la définition complète → changes what you *ask* the model, in the moment. You supply instructions, examples, and constraints in the context windowcontext windowThe context window is the maximum amount of text (measured in tokens) a language model can process at once, including both the input prompt and the generated output.Voir la définition complète →. Nothing about the model changes. This is your default, and it is more powerful than most executives assume, modern frontier models with a well-engineered system prompt and a few in-context examples ("few-shot") solve a surprising share of enterprise use cases outright.

Retrieval-Augmented Generation (RAG) changes what the model *knows* at inference time. You retrieve relevant documents from your own corpus, inject them into the prompt, and the model reasons over them. The model's weights are untouched; you're feeding it fresh, proprietary context on demand.

Fine-tuningFine-tuningFine-tuning adapts a pre-trained model to a specific task or domain by continuing training on a smaller, targeted dataset, improving accuracy and style for that use case.Voir la définition complète → changes how the model *behaves*, its default tone, format, structure, and task-specific instincts. You continue training the base model on curated examples so those patterns become baked into the weights.

The single most useful framing to carry into any architecture review is this:

> RAG gives the model knowledge. Fine-tuning gives the model skills. Prompting directs whatever knowledge and skills it already has.

The insurer's error is now obvious in this language. Their problem was *knowledge that changes*, a RAG problem. They reached for fine-tuningfine-tuningFine-tuning adapts a pre-trained model to a specific task or domain by continuing training on a smaller, targeted dataset, improving accuracy and style for that use case.Voir la définition complète →, which is a *skills and behavior* tool, and inherited a retraining treadmill for information that should have lived in a queryable index.

The decision test

When a use case lands on your desk, run it through three questions in order:

1. Can a strong prompt do it? If a well-crafted system prompt plus 3-5 examples gets you to acceptable quality, stop. You are done. Ship it.

2. Does the task require proprietary or fresh knowledge the model doesn't have? If yes, that's RAG. Think: "answer from our current policy documents," "cite our contracts," "reference this quarter's product catalog."

3. Does the task require a consistent behavior, format, or specialized skill that prompting can't reliably enforce? If yes, that's fine-tuningfine-tuningFine-tuning adapts a pre-trained model to a specific task or domain by continuing training on a smaller, targeted dataset, improving accuracy and style for that use case.Voir la définition complète →. Think: "always output valid JSON in our schemaschemaA schema is the formal blueprint that defines how data is structured, named, typed, and related within a database, file, or message.Voir la définition complète →," "adopt our brand's specific writing register," "classify support tickets into our 40 internal categories with high consistency."

Most enterprise GenAI value in 2024-2025 sits at levels 1 and 2. Fine-tuningFine-tuningFine-tuning adapts a pre-trained model to a specific task or domain by continuing training on a smaller, targeted dataset, improving accuracy and style for that use case.Voir la définition complète → is real and valuable, but it is the *last* tool you reachreachThe number of unique people exposed to your message in a given period. Unlike impressions, reach counts each person once, no matter how often they see it.Voir la définition complète → for, not the first, and crucially, RAG and fine-tuning are not mutually exclusive. The most sophisticated production systems fine-tune for behavior *and* use RAG for knowledge.

RAG: the default for enterprise knowledge

RAG has become the workhorse for one reason: enterprises are drowning in proprietary text, contracts, policies, tickets, wikis, research, that no base model has ever seen and that changes constantly. RAG lets you expose that corpus to the model without retraining, without moving data into someone else's weights, and with the ability to update in near-real-time.

But "we'll just do RAG" is where most programs quietly fail, because the hard part is not the LLMLLMA Large Language Model is an AI system trained on vast text data to predict and generate language, enabling tasks like writing, summarizing, and answering questions.Voir la définition complète →, it's the retrieval. A CDO should interrogate a RAG proposal on three dimensions:

Chunking and retrieval quality. How is the corpus split, indexed, and searched? Naive fixed-size chunking destroys context; a paragraph cut in half retrieves garbage. This is a data engineering problem you already understand, it's ETLETLETL (Extract, Transform, Load) is a data integration process that pulls data from sources, reshapes it into a consistent format, and writes it into a target system.Voir la définition complète → for unstructured text. The quality of your answers is capped by the quality of your retrieval, full stop.

Grounding and citation. Can the system show *which source* produced each claim? For a regulated enterprise, an ungrounded answer is a liability, not a feature. Insist that retrieval returns source references the model must cite.

Freshness and access control. Who is allowed to see which documents? RAG must respect your existing entitlements, a retrieval system that surfaces HR salary data to a junior analyst because "it was in the index" is a governance incident waiting to happen. Your access controls must live at the retrieval layer, not be bolted on after.

A useful mental model for the retrieval pipelinepipelineAll active sales opportunities across the stages of the sales process, together with their combined potential value and probability of closing.Voir la définition complète →:

query → embed → vector search (filtered by user's access rights)
      → rerank top-k → assemble context + citations
      → LLM generates grounded answer

The unglamorous truth: 80% of RAG engineering effort is in the retrieval and data pipeline, not the model. This is good news for a CDO, it's largely a discipline your organization already has. The teams who win at RAG are the ones who treated it as a search-and-data-quality problem, not an AI problem.

When RAG is over-engineering

If your knowledge base is small, stable, and fits comfortably in a modern model's context windowcontext windowThe context window is the maximum amount of text (measured in tokens) a language model can process at once, including both the input prompt and the generated output.Voir la définition complète →, you may not need a retrieval system at all, you can simply include the relevant documents in the prompt. Standing up a vector databasevector databaseA vector database stores data as high-dimensional numeric vectors (embeddings) and retrieves items by similarity rather than exact matches, powering semantic search and AI applications.Voir la définition complète →, an embeddingembeddingAn embedding is a numerical vector that represents data (text, images, or items) in a way that captures meaning, so similar items sit close together in space.Voir la définition complète → pipelinepipelineAll active sales opportunities across the stages of the sales process, together with their combined potential value and probability of closing.Voir la définition complète →, and a reranker to serve a 30-page employee handbook that changes twice a year is infrastructure theater. Match the machinery to the scale.

Fine-tuningFine-tuningFine-tuning adapts a pre-trained model to a specific task or domain by continuing training on a smaller, targeted dataset, improving accuracy and style for that use case.Voir la définition complète →: powerful, narrow, and frequently misused

Fine-tuningFine-tuningFine-tuning adapts a pre-trained model to a specific task or domain by continuing training on a smaller, targeted dataset, improving accuracy and style for that use case.Voir la définition complète → earns its place in exactly three situations, and you should be skeptical of any proposal that doesn't fall cleanly into one:

1. Behavior and format you can't reliably prompt. When you need the model to *always* produce output in a strict structure, adopt a consistent voice, or follow a domain convention that no amount of promptingpromptingPrompt engineering is the practice of designing and refining text inputs to guide large language models toward accurate, relevant, and reliable outputs.Voir la définition complète → enforces at scale. Fine-tuningFine-tuningFine-tuning adapts a pre-trained model to a specific task or domain by continuing training on a smaller, targeted dataset, improving accuracy and style for that use case.Voir la définition complète → bakes the pattern in.

2. Latency and cost at high volume. A fine-tuned smaller model can match a large model's quality on a *narrow* task while being dramatically cheaper and faster per call. At millions of calls a day, this arithmetic dominates. You're trading upfront training cost and rigidity for per-inference efficiency.

3. A specialized skill or classification the base model does poorly. Highly domain-specific tasks, medical coding, legal clause classification, niche language, where the base model's general training simply doesn't cut it.

What fine-tuningfine-tuningFine-tuning adapts a pre-trained model to a specific task or domain by continuing training on a smaller, targeted dataset, improving accuracy and style for that use case.Voir la définition complète → is *not* for: teaching the model facts. This is the industry's most expensive misconception. Fine-tuningFine-tuningFine-tuning adapts a pre-trained model to a specific task or domain by continuing training on a smaller, targeted dataset, improving accuracy and style for that use case.Voir la définition complète → is a poor and unreliable way to inject knowledge, the model may memorize some of it, hallucinate around the edges, and you'll need to retrain every time a fact changes. That's a RAG job. Every time.

The parameter-efficient reality

Modern fine-tuningfine-tuningFine-tuning adapts a pre-trained model to a specific task or domain by continuing training on a smaller, targeted dataset, improving accuracy and style for that use case.Voir la définition complète → rarely means retraining the whole model. Techniques like LoRA (Low-Rank Adaptation) train a small set of adapter weights on top of a frozen base model, cutting cost by orders of magnitude and letting you maintain multiple task-specific adapters over one base. When a vendor or team proposes "fine-tuningfine-tuningFine-tuning adapts a pre-trained model to a specific task or domain by continuing training on a smaller, targeted dataset, improving accuracy and style for that use case.Voir la définition complète →," your first question is: *full fine-tune or parameter-efficient?* If they're proposing a full fine-tune of a large model for a narrow task, that's usually a red flag for over-engineering.

The economics that actually matter to a CDO:

| Dimension | PromptingPromptingPrompt engineering is the practice of designing and refining text inputs to guide large language models toward accurate, relevant, and reliable outputs.Voir la définition complète → | RAG | Fine-TuningFine-TuningFine-tuning adapts a pre-trained model to a specific task or domain by continuing training on a smaller, targeted dataset, improving accuracy and style for that use case.Voir la définition complète → |

|---|---|---|---|

| Upfront cost | Near zero | Moderate (pipelinepipelineAll active sales opportunities across the stages of the sales process, together with their combined potential value and probability of closing.Voir la définition complète →) | High (data + training) |

| Handles fresh knowledge | Limited | Excellent | Poor |

| Enforces behavior/format | Limited | Limited | Excellent |

| Update speed | Instant | Near-real-time | Retraining cycle |

| Auditability | High | High (citations) | Low (opaque weights) |

Notice the auditability row. In a regulated environment, RAG's ability to cite sources is not a technical nicety, it's often what makes the deployment defensible to your risk and compliance functions. A fine-tuned model's reasoning is baked into weights you cannot inspect.

Making the call on monday morning

Here is how this translates into governance you can actually run.

Establish a "prompt-first" mandate. Require every GenAI use case to demonstrate that promptingpromptingPrompt engineering is the practice of designing and refining text inputs to guide large language models toward accurate, relevant, and reliable outputs.Voir la définition complète → alone is insufficient *before* approving RAG budget, and that RAG is insufficient before approving fine-tuningfine-tuningFine-tuning adapts a pre-trained model to a specific task or domain by continuing training on a smaller, targeted dataset, improving accuracy and style for that use case.Voir la définition complète → spend. This single policy will save you the insurer's €4M mistake. Make teams earn their way up the complexity ladder.

Separate the knowledge problem from the behavior problem, explicitly, on paper. For every use case, force the team to write one sentence: "This needs the model to *know* X and *behave* like Y." The moment you decompose it this way, the architecture chooses itself. Knowledge → RAG. Behavior → fine-tuningfine-tuningFine-tuning adapts a pre-trained model to a specific task or domain by continuing training on a smaller, targeted dataset, improving accuracy and style for that use case.Voir la définition complète →. Both → both.

Treat RAG as a data product, not an AI project. It should sit under the same governance, lineage, and access-control regime as your other data products. The retrieval index inherits the entitlements of the underlying sources, you already know how to do this. The failure mode is teams building shadow indexes that bypass your governance entirely.

Budget fine-tuning as an optimization, not a foundation. The correct sequence for most high-value products is: prove value with promptingpromptingPrompt engineering is the practice of designing and refining text inputs to guide large language models toward accurate, relevant, and reliable outputs.Voir la définition complète → + RAG, measure cost and latency in production, *then* fine-tune a smaller model to drive down per-call cost once volume justifies it. Fine-tuningFine-tuningFine-tuning adapts a pre-trained model to a specific task or domain by continuing training on a smaller, targeted dataset, improving accuracy and style for that use case.Voir la définition complète → as a first move is speculation; fine-tuningfine-tuningFine-tuning adapts a pre-trained model to a specific task or domain by continuing training on a smaller, targeted dataset, improving accuracy and style for that use case.Voir la définition complète → as an optimization of a proven workload is sound engineering.

Watch the total cost of ownership, not the demo. Fine-tuningFine-tuningFine-tuning adapts a pre-trained model to a specific task or domain by continuing training on a smaller, targeted dataset, improving accuracy and style for that use case.Voir la définition complète →'s cost isn't the training run, it's the retraining treadmill, the MLOpsMLOpsMachine Learning Operations: combining ML and DevOps practices to industrialise, deploy, monitor, and retrain models reliably in production.Voir la définition complète → to version and evaluate adapters, and the organizational muscle to maintain evaluation datasets. RAG's cost isn't the vector databasevector databaseA vector database stores data as high-dimensional numeric vectors (embeddings) and retrieves items by similarity rather than exact matches, powering semantic search and AI applications.Voir la définition complète →, it's the ongoing data pipelinedata pipelineETL (Extract, Transform, Load) is a data integration process that pulls data from sources, reshapes it into a consistent format, and writes it into a target system.Voir la définition complète → hygiene. Ask vendors and internal teams to model the *18-month* cost, including maintenance, not the pilot.

The strategic point: the market will keep pushing you toward the most complex, most expensive mechanism because that's where margins live. Your edge as a CDO is the discipline to ask "what does this use case actually require?" and to route it to the cheapest mechanism that meets the bar. Most of the time, that's a good prompt over solid retrieval, and knowing that is worth more than any single model.

Key Takeaways

1. Route every use case through the ladder, prompt, then RAG, then fine-tune, and require teams to earn each rung. A "prompt-first" mandate is the single highest-ROIROIReturn on Investment: the ratio of net profit to the cost of an investment. A 300% ROI means each dollar invested returns $3.Voir la définition complète → governance policy you can install.

2. Never solve a knowledge problem with fine-tuning. Fresh, proprietary, or changing information belongs in a RAG retrieval layer. Fine-tuningFine-tuningFine-tuning adapts a pre-trained model to a specific task or domain by continuing training on a smaller, targeted dataset, improving accuracy and style for that use case.Voir la définition complète → is for behavior, format, and specialized skills, not facts.

3. Treat RAG as a data product under your existing governance. 80% of the effort is retrieval and pipelinepipelineAll active sales opportunities across the stages of the sales process, together with their combined potential value and probability of closing.Voir la définition complète → quality; access controls must live at the retrieval layer, and every answer should cite its source for auditability.

4. Budget fine-tuning as a cost/latency optimization of a proven workload, and demand parameter-efficient methods (LoRA) by default, not a speculative first move or a full retrain of a large model for a narrow task.

5. Evaluate on 18-month total cost of ownership, not the demo. The retraining treadmill and pipelinepipelineAll active sales opportunities across the stages of the sales process, together with their combined potential value and probability of closing.Voir la définition complète → hygiene are where budgets die, model the maintenance, not just the build.