# Agentic Retrieval

Letting the model decide what to retrieve, evaluate whether it sufficed, and search again — retrieval as a loop rather than a single step.

**Rests on:** Multi-Document Retrieval, Reranking

**Leads to:** Analyst Agents

*LLMs and Generative AI · Retrieval-Augmented Generation*


## Why learn this

This unlocks the ability to build systems that can answer complex questions no single search can handle. You will be able to handle questions that require multiple pieces of evidence in sequence, recover from a bad first guess, and keep costs under control. Consider the difference between two assistants: one that gives up after one search because it didn't find an exact match, versus one that keeps digging, checking different angles until it finds the answer. This skill directly enables you to build AI Agents that can navigate large amounts of information, retrieve documents from many sources simultaneously, and reduce the chance of making things up by sticking to the facts.


## The intuition

Imagine you are a detective. A witness gives you a vague clue: 'The suspect was seen near a restaurant that's famous for its biryani.' You start your search with that clue. You find a list of biryani restaurants, but you need more: which ones are near the crime scene? You check the map, narrow it down. Now you have a shortlist, but you need the suspect's name. You interview staff at each restaurant. 'Did you see anyone suspicious?' One says, 'Yes, a man in a red jacket.' Now you search for 'man in red jacket' in the records. You find a match. This is not a single search; it is a chain of searches, each step guided by the result of the last. Agentic retrieval is exactly this: instead of one query-and-done, the model decides what to search for, looks at the results, decides if it has enough to answer, and if not, searches again with a refined query. It is a loop of 'search, evaluate, refine' until it is confident.


## An analogy

Think of a chef preparing a dish from a complex recipe. The recipe says 'add spices to taste.' The chef does not just add a fixed amount; they add a little, taste it, evaluate if it is too bland, too salty, or just right. If it is not right, they adjust the spice, taste again, and repeat until it meets their standard. Agentic retrieval is the same. The 'query' is the initial spice. The 'taste' is the model evaluating whether the retrieved documents contain the answer. The 'adjustment' is the next refined query. This loop is natural for a chef, but for a computer program, it is a novel idea because traditional programming is a straight line: input -> process -> output. No loop back. The analogy breaks down when you consider that the chef is a single, integrated intelligence; in AI, the 'chef' is a language model that can both generate queries and evaluate results, but it has a limited 'taste'—it can only see a certain number of documents at once, and it might be wrong about what it is tasting. So unlike a chef, the model needs a strict limit on how many times it can taste, or it will keep cooking forever, burning the budget.


## Definition

Agentic retrieval is a strategy where a language model iteratively generates search queries, evaluates the retrieved results against its need for information, and decides to either generate the final answer or perform another search, repeating until it satisfies a stopping condition.


## How it connects

This concept builds directly on Retrieval-Augmented Generation (RAG). In a standard RAG pipeline, you have a single query: you embed it, search your vector database, get a few chunks, and generate. A vector database is a specialized storage system for searching by meaning rather than just keywords, and chunks are specific pieces of text extracted from documents. Agentic retrieval is a loop around that pipeline. It also connects to your notes on Retrieval Quality and Context Precision and Recall because the model needs a way to evaluate if the retrieved chunks are relevant (precision) and if it has all the needed chunks (recall). It relates to Multi-Document Retrieval because in each iteration, the model might assemble evidence from multiple documents, and it relates to Chunking because if your chunks are too small or too large, the model will struggle to find the answer and will need more iterations to piece things together.


## The mathematics

The mathematics here is more about logic and counting than heavy algebra, but it rests on a foundation of simple arithmetic. The branch is 'arithmetic' and 'basic probability'—specifically, we will use arithmetic to track how many searches we do and to compute a score that tells us whether we have enough information. Let's build that scoring idea. Imagine you are grading a test. For each question, you give 1 point if the answer is correct, and 0 points if it is wrong. If you have 10 questions, you add up the points. If you get 7 points, you have a score of 7/10 = 0.7, which is a percentage. That's a basic 'score'—a number between 0 and 1 that represents success. In agentic retrieval, the model needs a similar score to evaluate each retrieved document: 'Is this document likely to help me answer the question?' A simple score could be a measure of how well the query terms appear in the document (like a keyword match count). But that's not the only thing. The model also needs a 'confidence score' for the entire loop: 'Based on the documents I have, am I confident enough to answer?' This confidence is also a number between 0 and 1, computed from a simple average of the individual document scores. The loop's stopping condition is a rule: 'Stop if confidence is above 0.8, or if I have searched 5 times.' That rule is a piece of logic that uses the arithmetic scores. So the mathematical content is: (1) compute a score for each retrieval round, (2) average those scores into a confidence, (3) compare the confidence against a threshold using a simple greater-than comparison. All of this is arithmetic and comparison, which you already use in daily life.

Let C be the confidence score, R the number of retrieval rounds so far, and s_i the relevance score of the i-th retrieved document in the current round. The confidence is the average score: C = (s_1 + s_2 + ... + s_k) / k, where k is the number of retrieved documents considered. The loop continues while C < T (where T is the threshold, e.g., 0.7) and R < M (where M is the maximum number of rounds, e.g., 5).


## A worked example

```
A user asks: 'What is the capital of the country where the Nile River starts, and what is its population?' We have a system that can generate a query, retrieve a small set of documents, and score each document for relevance from 0 (not relevant) to 1 (highly relevant). We will compute a confidence score for each round as the average of the top-3 document scores. The stopping rule: generate the answer if the confidence is ≥ 0.7, or after 3 rounds.

Step 1: Round 1: Query = 'capital of country where Nile starts'   # The model starts with the first part of the question.
Step 2: Retrieved document relevance scores: [0.8, 0.6, 0.2]   # The scores from the retrieval system.
Step 3: Sum = 0.8 + 0.6 + 0.2 = 1.6   # Add the scores.
Step 4: Confidence = 1.6 / 3 = 0.533   # Average the scores.
Step 5: Is 0.533 >= 0.7? No. Round 2.   # The model is not confident enough, so it searches again.
Step 6: Round 2: Query = 'Nile River source country name'   # The model refines the query to find the country.
Step 7: Retrieved document scores: [0.9, 0.7, 0.5]   # Better scores.
Step 8: Sum = 0.9 + 0.7 + 0.5 = 2.1; Confidence = 2.1 / 3 = 0.7   # Average is now 0.7.
Step 9: Is 0.7 >= 0.7? Yes. Stop loop.   # The confidence threshold is met. The model will generate the answer using this retrieved context.

Result: The loop stops after 2 rounds because the confidence score reaches the 0.7 threshold. The model will now use the documents from the second round (which likely include the answer: the country is Rwanda, etc.) to generate the final answer.
```


## From scratch, in Python

```python
# Agentic Retrieval from scratch
# Imitates a simple language model's decisions without using AI libraries.

# ----------
# Python building blocks
# ----------

# A list is an ordered collection that can hold multiple values.
# Example: fruits = ['apple', 'banana']
# A dictionary stores key-value pairs, looked up by name.
# Example: person = {'name': 'Anjali', 'age': 30}
# A for loop repeats actions for each item in a list.
# A function is a reusable block of code, defined with 'def'.

# ----------
# Step 1: Define a simple corpus (our 'knowledge base')
# ----------

corpus = {
    "doc_1": "The Nile River starts in Rwanda.",
    "doc_2": "The capital of Rwanda is Kigali.",
    "doc_3": "Kigali's population is about 1.2 million.",
    "doc_4": "The Nile is the longest river in the world.",
    "doc_5": "The Amazon river is in South America."
}

# ----------
# Step 2: A very simple search function (keyword matching)
# ----------

def search(query, corpus):
    """Return a list of documents that contain any word from the query."""
    words = query.lower().split()  # turns query into a list of words
    results = []
    for doc_id, doc in corpus.items():
        doc_lower = doc.lower()
        # Check if any word from the query appears in the document
        if any(word in doc_lower for word in words):
            results.append(doc)
    return results

# ----------
# Step 3: A simple relevance scorer (count of shared words)
# ----------

def score_relevance(query, doc_text):
    """Return a score between 0 and 1 based on word overlap."""
    query_words = set(query.lower().split())  # set removes duplicates, order ignored
    doc_words = set(doc_text.lower().split())
    if not query_words:
        return 0
    overlap = len(query_words.intersection(doc_words))
    return overlap / len(query_words)  # fraction of query words found

# ----------
# Step 4: Agentic loop
# ----------

def agentic_retrieval(question, corpus, threshold=0.7, max_rounds=3):
    """Main loop. Returns the final answer or None if not found."""
    round_count = 0
    context_docs = []  # accumulate documents across rounds
    
    while round_count < max_rounds:
        round_count += 1
        # Generate a new query (in a real system the LLM does this)
        # For simplicity, we start with the whole question, then add 'population' or 'capital' as needed
        if round_count == 1:
            query = question  # first guess
        elif round_count == 2:
            query = question + " capital"  # refined query
        else:
            query = question + " population"  # further refinement
        
        print(f"Round {round_count}: Query = '{query}'")
        
        # Retrieve candidate docs
        candidates = search(query, corpus)
        if not candidates:
            print("No results, going to next round")
            continue
        
        # Score and keep top 2 (to avoid overload)
        scored = [(score_relevance(query, doc), doc) for doc in candidates]
        scored.sort(reverse=True)  # highest first
        top_docs = [doc for score, doc in scored[:2]]
        
        # Add to context (avoid duplicates)
        for doc in top_docs:
            if doc not in context_docs:
                context_docs.append(doc)
        
        # Calculate confidence as average score of the top docs
        if scored:
            avg_score = sum(score for score, _ in scored[:2]) / min(2, len(scored))
            print(f"Average score: {avg_score:.2f}")
            
            # Stopping condition
            if avg_score >= threshold:
                print("Confident enough, generating answer...")
                return final_answer(question, context_docs)
        
    print("Max rounds reached.")
    return None

def final_answer(question, context_docs):
    """Dummy answer generator: joins relevant docs."""
    # In a real system, the LLM would generate a sentence. Here we just join.
    return " ".join(context_docs)

# ----------
# Run it
# ----------
question = "What is the capital of the country where the Nile starts?"
answer = agentic_retrieval(question, corpus)
print("Final answer:", answer)

```


## The same thing, with a framework

In a real framework like LangChain, you would use an `AgentExecutor` with a `Retriever` tool. The pseudocode would look like:

```python
from langchain.agents import AgentExecutor, create_react_agent
from langchain.tools import Tool

retriever = vector_store.as_retriever(search_kwargs={"k": 3})

def retrieve_function(query: str) -> str:
    docs = retriever.get_relevant_documents(query)
    return "\n".join([d.page_content for d in docs])

tools = [Tool(name="Retrieve", func=retrieve_function, description="Search the knowledge base")]

agent = create_react_agent(llm, tools, prompt)
agent_executor = AgentExecutor(agent=agent, tools=tools, max_iterations=3)
result = agent_executor.invoke({"input": "What is the capital of the country where the Nile starts?"})
```

The `max_iterations` parameter is the loop bound setting `M` in the formula. Underneath, it loops, the LLM outputs a 'Action' that calls the tool, then evaluates the result, and decides the next 'Action' or 'Final Answer'.


## What the code does

The code implements a crude agentic loop. The `search` function is a basic keyword matcher. `score_relevance` computes a simple overlap score (fraction of query words found). The `agentic_retrieval` function runs a while loop: it generates a query (simulated by rules), retrieves, scores, and computes an average score as confidence. If the confidence is above a threshold, it stops and generates an answer; otherwise, it tries again with a modified query. This demonstrates the core idea: the loop, the evaluation, and the stopping condition. The framework version using LangChain shows how a production system would express this using an AgentExecutor with a retriever tool, and importantly, the `max_iterations` parameter enforces the loop bound, preventing infinite loops.


## Common misconceptions

**Agentic retrieval is just a fancier way to search** — It's basically the same as doing a few searches in a row manually; the model just automates it.

It's not just automation. The model makes decisions about what to search for next based on the current results. That's a loop with feedback, not a fixed sequence. Also, it involves a stopping rule—knowing when to stop and answer—which is a decision, not an action. In fact: Agentic retrieval is a decision loop. The model doesn't just execute a list of queries; it evaluates the results of each query and decides what to do next, including whether it is done.

**Loop bounds are a safety net, not a requirement** — The model is smart enough to know when to stop; we shouldn't limit it with a max iterations count.

The model can get stuck in a loop, repeatedly rephrasing the same query because it is not understanding the results. Without a bound, this can lead to huge costs and long delays, and the user will get frustrated. In fact: Loop bounds are essential to ensure predictable cost and latency. A real system always has a maximum number of iterations (e.g., 5) and a fallback answer if no confident answer is found.

**The first query is what matters most** — If the first search doesn't find the answer, the system is broken and will never find it.

The whole point of agentic retrieval is to recover from a poor first query. The model can reformulate, add terms, or change the search direction based on what it has already retrieved. In fact: Agentic retrieval is specifically designed to handle multi-hop questions and to self-correct. A poor initial query can be fixed in subsequent iterations.

**It answers the question faster** — Agentic retrieval is faster than a single-shot RAG because it's more clever.

On the contrary, it's slower because it may perform multiple retrieval calls. The trade-off is that it can answer more complex questions correctly, but at a higher latency. In fact: Agentic retrieval trades speed for accuracy. It may take more time (multiple search rounds) but can handle multi-hop questions that a single query would fail on.

**The model knows everything; retrieval is just for references** — The language model already has the knowledge, so the retrieval loop is just for show or to provide citations.

The model's parametric memory is limited and outdated. The retrieval step brings in fresh, specific information from a corpus that the model didn't see during training. The loop exists because the model needs this external info to answer correctly. In fact: The external retrieval is the source of truth and freshness. The loop exists to gather this info, not just to show citations.


## In short

Forget everything else and remember this: **agentic retrieval is a loop**. The model doesn't do one search; it does a search, looks at what it got, and decides: 'Do I have enough? If not, I search again with a better query.' This lets it answer complex, multi-hop questions that a single search would fail. It is not a fixed pipeline but a loop with a stop condition—a confidence threshold and a maximum number of rounds—to protect your budget and the user's time. The power lies in the **feedback**, and the discipline lies in the **stop rule**.


## Where this is used

Agentic retrieval is used in many real-world AI systems. **Large language models (LLMs)** with a 'search' tool (like ChatGPT with browsing enabled) use an agentic loop: the model decides to search, sees the results, and then decides to search again if the results are insufficient for a confident answer. **Small language models (SLMs)** on phones or laptops face constraints on model size and context, so agentic retrieval is even more critical—they can't hold as much in memory, so they may need to do multiple smaller searches to build context piece-by-piece, rather than one large search, and they rely on a tight loop bound to conserve battery and compute. **Retrieval-augmented generation (RAG)** systems are the primary home for this: instead of a single query, the system allows the model to issue multi-hop queries. For instance, the user asks 'Who founded the company that owns the brand of the phone in my hand?', which requires first retrieving the phone model, then the company, then the founder. Agentic retrieval orchestrates this chain. **Agentic systems** that plan and call tools use this exact mechanism: the agent's reasoning loop includes 'Retrieve' as one of its tool calls, and it can call it multiple times, evaluating after each call whether to proceed to the next action. **Everyday applied machine learning** for customer support chatbots often uses an agentic retrieval loop, where the bot rephrases the query until it finds the relevant FAQ or knowledge-base article. Practical development: a developer would use a framework like LangChain or LlamaIndex, configuring the retriever, the language model, and setting `max_iterations` to a small number (e.g., 3 or 5). The main risk of bad implementation is **unbounded loops**, which cause unpredictable cost and latency—a runaway chatbot doing dozens of searches per question. Also, if the stopping condition is too lenient (low threshold), the model may generate an answer from insufficient evidence, reducing answer quality.


---

Read this on the web: https://basavapurushottam.com/learn-ai/agentic-retrieval
