Agentic RAG: Let the Agent Search

utility we construct is a RAG app.

The recipe is easy: chunk, embed, retrieve, then reply.

It seems to be clear on paper. However when you apply it to actual circumstances, issues get messy in a short time: similarity search finds comparable wordings however not essentially helpful chunks, the suitable proof by no means exhibits up within the retrieved context because it ranks too low, or necessary context could also be cut up throughout chunk boundaries.

With inadequate context, the LLM has little room to recuperate.

So, how about we make retrieval iterative? What if the mannequin can search, learn, resolve whether or not it has sufficient proof, and search once more when wanted? In all probability we don’t even want the vector embeddings within the first place.

That’s the premise of agentic RAG.

On this submit, we’ll construct a mini agentic RAG workflow with the OpenAI Brokers SDK. We’ll look at how the agent iteratively searches, reads, and grounds its reply.

On the finish, we’ll take a step again and briefly focus on the concerns for constructing a sensible agentic RAG answer.

1. Case Research: Answering a Coverage Query with Agentic RAG

For our case examine, we’ll construct a coverage RAG agent over an organization coverage doc assortment.

1.1 Curating The Doc Assortment

Right here, I created six artificial firm coverage docs. They’re all markdown recordsdata. Each has a title, an efficient date, a brief abstract, and the coverage textual content.

To be sensible, these docs cowl 6 frequent firm coverage areas:

1. approval_matrix.md, containing approval ranges for frequent enterprise journey choices, efficient on July 1, 2025.
2. conference_guidelines.md, containing guidelines for attending exterior occasions, efficient on Could 15, 2025.
3. faq.md, containing casual solutions to frequent journey questions, efficient on September 1, 2025.
4. policy_updates_2026.md, containing updates to lodging, convention journey, and approval timing for 2026, efficient on January 1, 2026.
5. remote_work_policy.md, containing guidelines for distant work, efficient on February 1, 2026.
6. travel_policy.md, containing normal journey reserving guidelines for flights, lodging, meals, and transportation, efficient on March 1, 2025.

We made it intentional that the reply to a coverage query could not reside in a single doc. This enables us to see the specified agentic habits.

You will discover the total artificial paperwork and the agentic RAG implementation pocket book right here.

1.2 Defining The Agent

Subsequent, we configure the agent. For that, we use the OpenAI Brokers SDK.

At a excessive degree, the agent is simply this:

# pip set up openai-agents
from brokers import Agent

agent = Agent(
    identify="Coverage analysis assistant",
    directions=INSTRUCTIONS,
    mannequin="gpt-5.4",
    instruments=[list_docs, search_docs, read_doc],
)

Two elements we have to undergo: the agent instruction, and the instruments it has entry to.

First, the instruction. That is the place we outline the specified search habits:

# Be aware: This instruction is iterated with AI
INSTRUCTIONS = """
[Role]
You're a cautious inner coverage analysis assistant.

[Research behavior]
Reply worker coverage questions utilizing the doc instruments.
Discover sufficient related proof to help the reply.
Maintain conclusions grounded within the coverage paperwork.

[Expected output]
Give a direct reply first.
Then briefly clarify the proof.
Cite the doc filenames used for every necessary declare.
""".strip()

For this case examine, we require that the agent can solely contact the docs through three pre-defined instruments:

The primary one is a instrument that provides the agent a fast overview of what paperwork exist:

@function_tool
def list_docs() -> checklist[dict]:
    """Listing accessible coverage paperwork with out returning their physique textual content."""
    return [
        {
            "doc_name": doc["doc_name"],
            "title": doc["title"],
            "efficient": doc["effective"],
            "abstract": doc["summary"],
        }
        for doc in docs.values()
    ]

The second instrument is a keyword-search instrument. We maintain it easy right here: every doc is cut up into paragraph chunks, and every question is matched in opposition to these chunks by token overlap:

@function_tool
def search_docs(question: str) -> checklist[dict]:
    """Search coverage paperwork and return the highest three quick snippets."""
    query_tokens = tokenize(question)
    scored = []

    for chunk in chunks:
        rating = len(query_tokens & chunk["tokens"])
        if rating:
            scored.append((rating, chunk))

    scored.type(key=lambda merchandise: merchandise[0], reverse=True)

    outcomes = []
    for rating, chunk in scored[:3]:
        snippet = chunk["text"].substitute("n", " ")
        if len(snippet) > 420:
            snippet = snippet[:417].rstrip() + "..."
        outcomes.append({
            "doc_name": chunk["doc_name"],
            "title": chunk["title"],
            "part": chunk["section"],
            "snippet": snippet,
            "rating": spherical(rating, 2),
        })

    return outcomes

The final instrument is what permits the agent to open one doc by filename:

@function_tool
def read_doc(doc_name: str) -> str:
    """Learn one coverage doc by filename."""
    if doc_name not in docs:
        legitimate = ", ".be part of(sorted(docs))
        return f"Unknown doc: {doc_name}. Legitimate paperwork: {legitimate}"

    return docs[doc_name]["text"]

That’s the total RAG agent.

1.3 Operating One Coverage Query

Now we check the agent with one concrete query:

We run the agent with:

from brokers import Runner

end result = await Runner.run(agent, PROMPT, max_turns=12)

The agent produced the suitable reply: sure, the worker can guide the official convention lodge if there’s a sensible enterprise cause. It acquired that info from conference_guidelines.md.

For the approval half, the agent first recognized that approval is required because the lodge is above the conventional cap. Then it gave the corresponding approval circumstances. The agent used travel_policy.md, approval_matrix.md, and policy_updates_2026.md to help its reply, which is strictly what we might anticipate.

The extra fascinating half is the hint, from which we are able to find out how the agent thinks. We will present the hint within the following method:

for merchandise in end result.new_items:
    print(sort(merchandise).__name__, merchandise)

end result.new_items accommodates the intermediate instrument calls and gear outputs produced by the agent. In my run, I can see that the agent first referred to as search_docs() with key phrases like convention lodge, lodge cap, approval, and Berlin. Then, it referred to as list_docs() to examine the accessible coverage paperwork. After that, it opened the related recordsdata with read_doc(). Solely then did it produce the ultimate reply.

That is precisely the agentic loop we wished to see.

3. What to Determine Earlier than Constructing Agentic RAG

The case examine we simply went by way of solely scratched the floor. To actually construct a sensible agentic RAG answer, primarily based on my expertise, I counsel you reply the next 5 questions:

Q1: How a lot freedom ought to the agent have?

One frequent possibility is strictly what we’ve carried out within the earlier case examine: we uncovered a few fastidiously curated instruments, and the agent is simply allowed to make use of these instruments to do the investigation. That is simple by way of controlling, testing, and auditing.

However we are able to additionally give the agent broader entry, equivalent to shell and file system. This fashion, the agent can instantly run scripts to go looking and examine recordsdata, and perhaps even do additional information processing to generate helpful artifacts, all by itself.

This sample may be rather more highly effective, however it additionally will increase danger and makes habits tougher to foretell.

So for many RAG functions, I’d begin with curated instruments first, and solely add shell/file-system entry when the duty complexity justifies it.

Q2: Ought to the agent search uncooked textual content solely?

Most RAG initiatives would possibly begin with plain textual content like PDFs, wiki pages, manuals, and so on. That’s wonderful.

However in apply, we are able to typically make retrieval simpler by deriving a information layer on prime of the uncooked texts.

These derived information artifacts may be doc metadata, summaries, cross-document hyperlinks, or we are able to go additional and implement a correct information graph.

These derived information artifacts assist the agent navigate the corpus, whereas the uncooked texts stay because the supply of fact.

Q3: Will we nonetheless want embeddings?

Agentic RAG doesn’t essentially imply embeddings are gone.

Vector embeddings are nonetheless an environment friendly approach to discover semantically related texts, and it typically outperforms a pure key phrase search technique.

In agentic RAG, what modified basically is that the retrieval turns into an “motion” the agent can take. Beneath this framing, “motion” can nonetheless be powered by an embedding-based retriever, a keyword-based one, or perhaps a hybrid one.

So embeddings can nonetheless be helpful. They’re only one doable approach to energy the agent’s search instrument.

This fall: Ought to one agent deal with the whole lot?

The best agentic RAG setup is only one agent that does the search, learn, and reply.

However as the duty will get extra advanced, you would possibly need to cut up the work amongst a number of brokers. Extra concretely, you would possibly have to undertake a multi-agent technique.

You possibly can cut up the work by position. For instance, the planner-retriever-writer cut up, the place the planner decides what proof is required, the retriever collects it, and the author produces the ultimate reply by utilizing the collected proof.

It’s also possible to cut up by supply sort, the place every agent is supplied with personalized instruments and focuses on one particular sort of supply.

Simply take into accout: A multi-agent setup provides coordination complexity, and there’s no assure that it’s going to carry out higher than a single-agent setup. Empirical testing is essential.

Q5: Ought to we all the time use agentic RAG?

Perhaps not all the time.

Simply because agentic RAG turns into a stylish matter doesn’t essentially imply it’s best to all the time default to it.

Agentic RAG offers extra flexibility, however that comes with prices. That price is just not solely about latency or token price, but in addition much less predictable agent habits.

All the time begin easy, then add agentic loops when the query really wants iterative retrieval.