# Agentic RAG: Retrieval Inside the Agent Loop
Ask a chatbot "Can a contractor expense international flights?" and plain retrieval grabs the three chunks of text that look most similar to your question, stuffs them into the prompt, and answers. If the real rule lives in a "Contractor Travel" section that never used the word "expense," you get a confident, wrong answer.
Agentic RAG fixes this by letting the AI act like a researcher instead of a vending machine. It searches, reads what came back, notices the answer is thin, and searches again with better terms. Same 500-page policy library. Very different result.
RAG stands for Retrieval-Augmented Generation. In plain terms: before the AI answers, it looks things up in your documents and uses what it finds. This keeps answers grounded in your actual content instead of the model's memory.
Retrieval here means searching a vector databasevector databaseA vector database stores data as high-dimensional numeric vectors (embeddings) and retrieves items by similarity rather than exact matches, powering semantic search and AI applications.View full definition →: a store where your documents are chunked into passages and indexed by meaning, so a search for "reimbursement" can also surface a passage about "getting paid back for costs."
One-shot RAG does exactly one search, then answers. Agentic RAG puts retrieval inside a loop, so the AI can search multiple times and decide when it has enough.
Here is the classic flow for the contractor question:
1. Convert the question into a search.
2. Retrieve the top 3 to 5 matching passages.
3. Feed them to the model.
4. Model answers.
This is great when the answer sits in one obvious place. "What is our PTO accrual rate?" almost always lives in one paragraph. One search finds it. Done. One model call, low cost, fast.
The problem is questions that span sections. Our contractor question actually needs three facts:
Those might live in three different chapters. A single search optimized for "contractor expense international flights" may return two relevant passages and one about domestic mileage. The model answers from incomplete evidence and sounds sure of itself.
Now give the model a tool it can call: a search_policies function. A tool is just an action the model is allowed to trigger, like running a search and getting results back. Crucially, the model chooses *when* and *what* to search, and it can do this several times.
Here is the same contractor question, agentic style:
1. Search: "contractor travel reimbursement eligibility." Finds: contractors are eligible if travel is pre-approved.
2. Read and notice a gap: eligibility is confirmed, but nothing yet on international rules.
3. Search again: "international flights travel policy limits." Finds: international travel over $2,000 needs VPVPA clear statement of the benefits your product delivers, the problems it solves and why customers should choose you over alternatives.View full definition → sign-off.
4. Search again: "pre-approval process contractor." Finds the approval form and timeline.
5. Answer: "Yes, with two conditions..." and cites all three passages.
The AI decided it needed three lookups. Nobody hard-coded that. It read the intermediate results and judged them insufficient. That judgment is the whole point.
The pattern is provider-neutral. Every major agent SDK (OpenAI Agents SDK, the Claude Agent SDK, Google's ADK) implements the same shape: give the model a tool, let it call the tool, feed results back, repeat until it stops. See the provider deep-dive blocks for exact syntax. The core idea:
tools = [{
"name": "search_policies",
"description": "Search the company policy library. Returns top passages.",
"parameters": {"query": "string"}
}]
messages = [{"role": "user", "content": "Can a contractor expense international flights?"}]
for step in range(5): # cap the loop so it can't run forever
response = model.generate(messages, tools=tools)
if response.tool_call:
results = search_policies(response.tool_call.query) # your vector DB
messages.append(response)
messages.append({"role": "tool", "content": results})
# loop again: the model reads results and decides its next move
else:
print(response.text) # model chose to answer, so we're done
breakRead the loop plainly: the model either asks to search or gives a final answer. If it searches, we run the search, hand back the results, and go around again. The range(5) is a safety cap so a confused agent cannot search forever.
If you want a solid, vendor-neutral primer on RAG concepts before layering the agent loop on top, the Hugging Face RAG documentation is free and clear.
The loop is not free. Each search plus each model call costs time and money. Three round-trips might turn a 2-second, one-cent answer into an 8-second, four-cent answer. Multiply that across thousands of daily questions and it matters.
Use this rough guide.
A practical middle path: run one-shot by default, and only trigger the loop when the first result set looks weak. Many teams score the retrieved passages and, if the top match is below a confidence threshold, hand control to the agent to dig deeper. Cheap most of the time, thorough when it counts.
Knowledge check
1. What is the key distinction between one-shot RAG and agentic RAG?
2. Why can plain one-shot retrieval give a confident but wrong answer to the contractor flights question?
3. For which type of question is one-shot RAG typically the best fit?
4. Select ALL correct answers. What advantages does agentic RAG offer over one-shot RAG?
Select all the correct answers.
5. Select ALL correct answers. Which statements about RAG and vector databases are accurate?
Select all the correct answers.
An agent that can search freely can also spiral: search ten times, chase tangents, or loop on the same query. Three guardrails keep it useful.
Set a hard limit (the range(5) above). If the agent has not answered after five searches, force it to answer with what it has and flag low confidence. Better a bounded "here is what I found" than an infinite loop.
The system prompt should make stopping explicit. Something like:
> "Search the policy library as many times as needed. After each result, decide: do I have enough to answer completely and cite sources? If yes, answer. If no, search with different terms. Do not exceed 5 searches. If still unsure, say what is missing."
Without this, some models search once and stop; others search endlessly. The instruction shapes the behavior.
A common failure: the agent runs the same search twice and gets the same weak results. Good promptingpromptingPrompt engineering is the practice of designing and refining text inputs to guide large language models toward accurate, relevant, and reliable outputs.View full definition → nudges it to *rephrase*: try synonyms, narrow to a section, or break a compound question into parts. "Contractor international flight approval" becomes three targeted searches, not the same one three times.
Question: "Can a contractor expense international flights?"
One-shot RAG answer:
> "Contractors may be reimbursed for pre-approved travel." (Missed the international sign-off rule entirely. Technically true, practically incomplete.)
Agentic RAG answer:
> "Yes, with two conditions. Contractors are eligible for reimbursement on pre-approved travel (Travel Policy, sec. 3.1). International flights over $2,000 require VPVPA clear statement of the benefits your product delivers, the problems it solves and why customers should choose you over alternatives.View full definition → approval before booking (sec. 4.2). Submit the pre-approval form at least 10 business days out (sec. 4.5)."
The second answer took three searches and cost a few cents more. For a compliance-sensitive question, that is an easy trade. For "what time does the office open," it would be waste.