Choosing the right approach: prompt, RAG, fine-tune, or agent

A startup spent four months and $80,000 fine-tuningfine-tuningFine-tuning adapts a pre-trained model to a specific task or domain by continuing training on a smaller, targeted dataset, improving accuracy and style for that use case.View full definition → a model to answer customer questions about their product. Then a competitor shipped the same feature in a weekend by pasting their help docs into a prompt. The competitor won. Picking the wrong approach is the most expensive mistake in an AI project, and it is almost always avoidable.

There are four common ways to build with large language models (LLMs, the AI behind ChatGPT, Claude, and Gemini). Each solves a different problem. Most people reach for the complicated one when the simple one would have worked.

Let's fix that.

The four approaches in plain language

Prompting. You write good instructions and send them to the model. No setup, no code required.

RAG (Retrieval-Augmented Generation). You give the model your own documents to read before it answers. "Retrieval" means it looks things up; "augmented" means it uses what it found to write a better reply.

Fine-tuning. You train the model on many examples so it absorbs a style, format, or specialized behavior permanently.

Agents. You let the model take actions: search the web, send an email, query a database, run other tools, in a loop, until a task is done.

Here is the order you should consider them: prompt first, then RAG, then agent, and fine-tune almost last. The simplest approach that works is the right one.

A decision tree you can actually use

Anchor everything in one real scenario: you run a mid-size law firm and want AI to help draft and answer questions about contracts.

Walk it down the tree.

Question 1: Can a smart, well-written prompt do it?

Try this first. Always.

Modern models (GPT-5, Claude 4.5, Gemini 2.5) are extremely capable out of the box. If your task relies on general knowledge or reasoning, a prompt may be all you need.

For the law firm: "Explain the difference between an indemnity and a warranty clause in plain English" needs zero setup. Just ask.

from openai import OpenAI
client = OpenAI()

response = client.responses.create(
    model="gpt-5",
    input="Explain the difference between an indemnity "
          "and a warranty clause in a commercial contract, "
          "in plain English, for a non-lawyer."
)
print(response.output_text)

If a prompt answers your question reliably, stop here. You are done. Do not build anything else.

For a deeper guide to writing prompts that work, the Anthropic prompt engineering docs are free and excellent.

Question 2: Does it need YOUR specific information?

The model does not know your contracts, your clients, or your internal policies. If the answer lives in your documents, you need RAG.

For the law firm: "What's the termination notice period in the Henderson account contract?" The model has never seen that contract. So you give it to them.

How RAGRAGA method that lets an AI model answer using your own documents, retrieving relevant passages before generating a response instead of relying only on training data.View full definition → works in three steps:

1. Break your documents into chunks and store them in a searchable database (a "vector databasevector databaseA vector database stores data as high-dimensional numeric vectors (embeddings) and retrieves items by similarity rather than exact matches, powering semantic search and AI applications.View full definition →" that finds text by meaning, not just keywords).
2. When a question comes in, retrieve the most relevant chunks.
3. Paste those chunks into the prompt so the model answers from your real data.

A minimal version is just smart pasting:

relevant_clause = search_contracts("Henderson termination notice")

prompt = f"""Using ONLY the contract text below, answer the question.
If the answer isn't present, say so.

CONTRACT TEXT:
{relevant_clause}

QUESTION: What is the termination notice period?"""

response = client.responses.create(model="gpt-5", input=prompt)
print(response.output_text)

RAG is the workhorse of business AI in 2026. Customer support, internal knowledge bases, document Q&A: most of it is RAG. It keeps answers grounded in your real, current data and reduces made-up answers (called "hallucinations").

Shortcut for small document sets: today's models have huge context windows (Gemini 2.5 handles over a million tokenstokensA token is the basic unit of text that language models process, often a word fragment, whole word, or punctuation mark rather than a single character.View full definition →, roughly a few thousand pages). If you only have a handful of documents, you can skip the vector database and paste everything in directly. Build the complex RAG pipelinepipelineAll active sales opportunities across the stages of the sales process, together with their combined potential value and probability of closing.View full definition → only when your document pile is too big to fit.

Question 3: Does it need to ACT, not just answer?

If the task involves doing things in the real world (multiple steps, using tools, making decisions along the way), you need an agent.

For the law firm: "Review this new contract, flag any clause that conflicts with our standard terms, check whether the counterparty has past disputes in our system, and draft a summary email to the partner." That is not one answer. It is a sequence of actions with tool use.

An agent loops: think, act, observe the result, think again, until the job is finered.

response = client.responses.create(
    model="gpt-5",
    input="Review the attached contract. Flag clauses that "
          "conflict with our standard terms, look up the "
          "counterparty in our disputes database, and draft "
          "a summary email to the partner.",
    tools=[
        {"type": "file_search"},      # read the contract
        {"type": "function", "name": "query_disputes_db"},
        {"type": "function", "name": "draft_email"}
    ]
)

Agents are powerful and the hot topic of 2026, but they are harder to control and can fail in surprising ways. Give an agent the smallest set of tools it needs, and keep a human approving anything risky (like sending that email). Start agents narrow.

Question 4: Do you need a consistent STYLE or FORMAT the model just won't follow?

This is where fine-tuning comes in, and it is last for a reason.

Fine-tuning means showing the model hundreds or thousands of examples until a behavior becomes second nature. It is the right tool when:

• You need a very specific tone or format every single time (your firm's exact contract-drafting style).
• You have many high-quality examples to train on.
• PromptingPromptingPrompt engineering is the practice of designing and refining text inputs to guide large language models toward accurate, relevant, and reliable outputs.View full definition → and RAG have genuinely fallen short, not just on your first try.

For the law firm: if you have 2,000 past contracts written in your firm's distinct voice and want every draft to match it precisely, fine-tuning can bake that style in.

The critical distinction people get wrong: fine-tuning teaches *style and behavior*, not *facts*. It is bad at knowledge. If you fine-tune to add information about the Henderson contract, the model will confidently invent details. For facts, use RAG. For style, consider fine-tuning.

# Fine-tuning starts with example pairs, not live questions
{"messages": [
    {"role": "user", "content": "Draft a confidentiality clause."},
    {"role": "assistant", "content": "<your firm's exact house style here>"}
]}
# You provide hundreds of these, then train a custom model.

Fine-tuning costs the most in time, data, and money. By 2026, fewer projects need it than people assume, because base models and RAG keep getting better. ReachReachThe number of unique people exposed to your message in a given period. Unlike impressions, reach counts each person once, no matter how often they see it.View full definition → for it only when the simpler tools have failed.

Putting it together: they combine

These are not rivals. The best systems stack them.

Our law firm's final tool might be: an agent that uses RAG to read contracts, runs on a fine-tuned model for house style, all driven by carefully written prompts.

But they did not build that on day one. They started with a prompt. Then added RAG when it needed their documents. Then an agent when it needed to act. They only considered fine-tuning after everything else was solid.

That sequence is the whole point. Each step adds cost and complexity, so you earn your way up the ladder.

A quick reference

• Prompt: general knowledge or reasoning. Zero setup. *"Explain this clause."*
• RAG: answers from your own documents. *"What does our Henderson contract say?"*
• Agent: multi-step tasks and tool use. *"Review, check the database, draft the email."*
• Fine-tune: consistent style or format from many examples. *"Always write in our house voice."*

Key Takeaways

1. Always start with a prompt. It is free, instant, and surprisingly capable. Most ideas should be tested here before you build anything.
2. Use RAG for facts, fine-tuning for style. Mixing these up is the most common and most expensive error. Your documents go in RAG; your tone goes in fine-tuning.
3. Earn your way up the ladder. Prompt, then RAG, then agent, and fine-tune almost last. Each step adds cost, so only climb when the simpler one provably fails.
4. For small document sets, skip RAG entirely. Today's large context windows let you paste a few documents straight into the prompt. Build the pipeline only when the pile gets too big.
5. Combine approaches in real systems, but add them one at a time. Validate each layer before stacking the next.