Agentic RAG: retrieval inside the agent loop

# Agentic RAG: retrieval inside the agent loop

Ask a chatbot "Can a contractor expense international flights?" and plain retrieval grabs the three chunks of text that look most similar to your question, stuffs them into the prompt, and answers. If the real rule lives in a "Contractor Travel" section that never used the word "expense," you get a confident, wrong answer.

Agentic RAG fixes this by letting the AI act like a researcher instead of a vending machine. It searches, reads what came back, notices the answer is thin, and searches again with better terms. Same 500-page policy library. Very different result.

First, the vocabulary

RAG stands for Retrieval-Augmented Generation. In plain terms: before the AI answers, it looks things up in your documents and uses what it finds. This keeps answers grounded in your actual content instead of the model's memory.

Retrieval here means searching a vector databasevector databaseA vector database stores data as high-dimensional numeric vectors (embeddings) and retrieves items by similarity rather than exact matches, powering semantic search and AI applications.View full definition →: a store where your documents are chunked into passages and indexed by meaning, so a search for "reimbursement" can also surface a passage about "getting paid back for costs."

One-shot RAG does exactly one search, then answers. Agentic RAG puts retrieval inside a loop, so the AI can search multiple times and decide when it has enough.

One-shot RAG: fast, cheap, sometimes wrong

Here is the classic flow for the contractor question:

1. Convert the question into a search.

2. Retrieve the top 3 to 5 matching passages.

3. Feed them to the model.

4. Model answers.

This is great when the answer sits in one obvious place. "What is our PTO accrual rate?" almost always lives in one paragraph. One search finds it. Done. One model call, low cost, fast.

The problem is questions that span sections. Our contractor question actually needs three facts:

• Are contractors eligible for travel reimbursement at all?
• What are the international travel rules?
• Is there a pre-approval requirement?

Those might live in three different chapters. A single search optimized for "contractor expense international flights" may return two relevant passages and one about domestic mileage. The model answers from incomplete evidence and sounds sure of itself.

Agentic RAG: the AI keeps digging

Now give the model a tool it can call: a search_policies function. A tool is just an action the model is allowed to trigger, like running a search and getting results back. Crucially, the model chooses *when* and *what* to search, and it can do this several times.

Here is the same contractor question, agentic style:

1. Search: "contractor travel reimbursement eligibility." Finds: contractors are eligible if travel is pre-approved.

2. Read and notice a gap: eligibility is confirmed, but nothing yet on international rules.

3. Search again: "international flights travel policy limits." Finds: international travel over $2,000 needs VPVPA clear statement of the benefits your product delivers, the problems it solves and why customers should choose you over alternatives.View full definition → sign-off.

4. Search again: "pre-approval process contractor." Finds the approval form and timeline.

5. Answer: "Yes, with two conditions..." and cites all three passages.

The AI decided it needed three lookups. Nobody hard-coded that. It read the intermediate results and judged them insufficient. That judgment is the whole point.

What the loop looks like

The pattern is provider-neutral. Every major agent SDK (OpenAI Agents SDK, the Claude Agent SDK, Google's ADK) implements the same shape: give the model a tool, let it call the tool, feed results back, repeat until it stops. See the provider deep-dive blocks for exact syntax. The core idea:

tools = [{
    "name": "search_policies",
    "description": "Search the company policy library. Returns top passages.",
    "parameters": {"query": "string"}
}]

messages = [{"role": "user", "content": "Can a contractor expense international flights?"}]

for step in range(5):  # cap the loop so it can't run forever
    response = model.generate(messages, tools=tools)

    if response.tool_call:
        results = search_policies(response.tool_call.query)   # your vector DB
        messages.append(response)
        messages.append({"role": "tool", "content": results})
        # loop again: the model reads results and decides its next move
    else:
        print(response.text)   # model chose to answer, so we're done
        break

Read the loop plainly: the model either asks to search or gives a final answer. If it searches, we run the search, hand back the results, and go around again. The range(5) is a safety cap so a confused agent cannot search forever.

If you want a solid, vendor-neutral primer on RAG concepts before layering the agent loop on top, the Hugging Face RAG documentation is free and clear.

When the extra loop is worth it

The loop is not free. Each search plus each model call costs time and money. Three round-trips might turn a 2-second, one-cent answer into an 8-second, four-cent answer. Multiply that across thousands of daily questions and it matters.

Use this rough guide.

ReachReachThe number of unique people exposed to your message in a given period. Unlike impressions, reach counts each person once, no matter how often they see it.View full definition → for agentic RAG when:

• The answer spans multiple sections. Policy questions, legal questions, and "compare A to B" questions usually do.
• Wording rarely matches. Users say "expense flights," the document says "reimbursable air travel." Iteration lets the agent try new phrasings.
• Being wrong is expensive. Compliance, medical, financial, or contract questions justify the extra care.
• You need citations. An agent that gathers several passages can point to each one.

Stick with one-shot RAG when:

• The answer lives in one place. Definitions, single-fact lookups, FAQ-style questions.
• Volume is huge and stakes are low. A support bot answering "what are your hours?" a million times should not loop.
• Latency is critical. A live phone assistant cannot pause for four searches.

A practical middle path: run one-shot by default, and only trigger the loop when the first result set looks weak. Many teams score the retrieved passages and, if the top match is below a confidence threshold, hand control to the agent to dig deeper. Cheap most of the time, thorough when it counts.

Making the loop reliable

An agent that can search freely can also spiral: search ten times, chase tangents, or loop on the same query. Three guardrails keep it useful.

1. Cap the iterations

Set a hard limit (the range(5) above). If the agent has not answered after five searches, force it to answer with what it has and flag low confidence. Better a bounded "here is what I found" than an infinite loop.

2. Tell it when to stop

The system prompt should make stopping explicit. Something like:

> "Search the policy library as many times as needed. After each result, decide: do I have enough to answer completely and cite sources? If yes, answer. If no, search with different terms. Do not exceed 5 searches. If still unsure, say what is missing."

Without this, some models search once and stop; others search endlessly. The instruction shapes the behavior.

3. Vary the query, do not repeat it

A common failure: the agent runs the same search twice and gets the same weak results. Good promptingpromptingPrompt engineering is the practice of designing and refining text inputs to guide large language models toward accurate, relevant, and reliable outputs.View full definition → nudges it to *rephrase*: try synonyms, narrow to a section, or break a compound question into parts. "Contractor international flight approval" becomes three targeted searches, not the same one three times.

A concrete before-and-after

Question: "Can a contractor expense international flights?"

One-shot RAG answer:

> "Contractors may be reimbursed for pre-approved travel." (Missed the international sign-off rule entirely. Technically true, practically incomplete.)

Agentic RAG answer:

> "Yes, with two conditions. Contractors are eligible for reimbursement on pre-approved travel (Travel Policy, sec. 3.1). International flights over $2,000 require VPVPA clear statement of the benefits your product delivers, the problems it solves and why customers should choose you over alternatives.View full definition → approval before booking (sec. 4.2). Submit the pre-approval form at least 10 business days out (sec. 4.5)."

The second answer took three searches and cost a few cents more. For a compliance-sensitive question, that is an easy trade. For "what time does the office open," it would be waste.

Key Takeaways

• Agentic RAG = retrieval in a loop. The model searches, reads, and searches again until it has enough, instead of doing one lookup and hoping.
• Match the method to the question. One-shot for single-fact, high-volume, low-stakes queries. Agentic for multi-section, high-stakes, or citation-heavy questions.
• Always cap the loop. A hard limit (for example, 5 searches) plus a clear "when to stop" instruction prevents runaway cost and infinite loops.
• Prompt it to rephrase, not repeat. Tell the agent to vary search terms so each iteration adds new evidence.
• Consider a hybrid. Default to one-shot, and escalate to the agent loop only when the first results look weak. This keeps costs low while protecting quality where it matters.