The Problem with Standard RAG and How Agentic RAG Solves It
Standard RAG follows a fixed pipeline: embed the query, retrieve top-k chunks, stuff them into a prompt, and generate an answer. This works well when retrieval is successful, but it has no mechanism to detect or recover from retrieval failures. If the retrieved chunks are irrelevant, the model either halluccinates an answer from its training data or generates a response that ignores the context entirely.
Agentic RAG introduces decision-making into the retrieval pipeline. Instead of blindly passing retrieved documents to the generator, an agentic RAG system evaluates the quality of retrieved documents and takes corrective action. If documents are irrelevant, it reformulates the query and tries again. If the question cannot be answered from the knowledge base, it routes to a web search or admits it does not know. If the generated answer contains hallucinations, it detects and corrects them before responding.
This approach is inspired by three research papers: Corrective RAG (CRAG), Self-RAG, and Adaptive RAG. CRAG grades retrieved documents and triggers a web search when retrieval fails. Self-RAG has the model critique its own generation for faithfulness. Adaptive RAG routes queries to different retrieval strategies based on complexity. Combining these ideas in a LangGraph workflow creates a robust system that handles diverse queries gracefully.
The key insight is that evaluation is cheap compared to serving a wrong answer. An extra LLM call to grade retrieval quality costs fractions of a cent but prevents wrong answers that erode user trust. In enterprise applications where accuracy matters, agentic RAG's self-correction mechanisms justify their additional cost many times over.
LangGraph is the ideal framework for agentic RAG because the workflow naturally has cycles (retry after failed retrieval), conditional routing (choose between knowledge base and web search), and multiple evaluation steps that feed back into the pipeline.
Query Routing and Adaptive Retrieval
The first component of agentic RAG is query routing, which analyzes the user's question and decides the best retrieval strategy. Not all questions should go to the vector store. Some are better answered by a web search, some require a SQL query, and some are simple enough that the LLM can answer directly from its training data.
Implement query routing as the entry node of your LangGraph graph. The node calls an LLM with structured output to classify the query into categories: 'vectorstore' for questions about your domain knowledge, 'web_search' for current events or topics outside your knowledge base, 'sql_query' for structured data questions, or 'direct_answer' for simple factual questions the model can answer confidently.
The routing prompt is critical. Provide clear examples of each category and explicit decision criteria. 'Questions about company policies and product documentation should route to vectorstore. Questions about current news, competitor information, or topics from after 2024 should route to web_search. Questions asking for specific numbers from databases should route to sql_query.'
Adaptive retrieval goes further by adjusting retrieval parameters based on query complexity. A simple factual question might need only 2 retrieved chunks with a high similarity threshold. A complex analytical question might need 8-10 chunks with a lower threshold to cast a wider net. A query complexity classifier, implemented as another structured output call, determines these parameters.
Multi-step retrieval handles queries that need information from multiple sources. The question 'How does our pricing compare to industry benchmarks?' requires both internal pricing data (vectorstore) and industry benchmark data (web search). The routing node identifies multi-source queries and triggers parallel retrieval from all relevant sources, merging results before generation.
Each routing decision is logged for monitoring. Track routing accuracy by periodically reviewing a sample of routing decisions against human judgments. Adjust the routing prompt and examples when you discover systematic misroutes.
Document Grading and Retrieval Self-Correction
After retrieval, a grading node evaluates whether the retrieved documents are actually relevant to the query. This is the core of the corrective mechanism. The grader is an LLM call that takes each document and the query and returns a binary relevance judgment: 'yes' this document contains information relevant to the query, or 'no' it does not.
Implement the grader with structured output returning a Pydantic model with a 'relevant' boolean field. Grade each document independently and filter out irrelevant ones. This prevents the generation model from being distracted by off-topic content that happened to have high embedding similarity.
The corrective logic activates when the grading results are poor. If fewer than half the retrieved documents are relevant, the system has three options: reformulate the query using an LLM that rephrases the original question for better retrieval, expand the search to a web search engine that might have broader coverage, or return a message acknowledging that the knowledge base does not contain sufficient information.
Query reformulation is the most common corrective action. The reformulation node takes the original query and the irrelevant documents (which provide negative signal about what not to search for) and generates a rephrased query. Often, rephrasing a question from the user's colloquial language into the terminology used in the documents dramatically improves retrieval.
Implement a maximum retry limit, typically 2-3 reformulation attempts, to prevent infinite loops. If retrieval still fails after retries, the system should gracefully fall back to a web search or an honest 'I could not find relevant information' response.
The grading step adds 100-300ms of latency per retrieval cycle. For most applications this is acceptable, but if latency is critical, you can optimize by grading in parallel, using a smaller model for grading, or implementing a lightweight classifier that approximates the grading without a full LLM call.
Hallucination Detection and Answer Grading
Even with relevant context, the generation model can hallucinate. Hallucination detection is the second line of defense in agentic RAG. After generation, a hallucination checker evaluates whether the generated answer is fully supported by the retrieved context.
The hallucination checker is an LLM call that takes the retrieved context and the generated answer and returns a judgment: is every factual claim in the answer supported by the context? Implement this with structured output returning a 'grounded' boolean and an optional 'unsupported_claims' list that identifies specific claims not found in the context.
When hallucination is detected, the corrective action depends on severity. If the answer is mostly grounded with one unsupported claim, the system can regenerate with a stricter prompt that emphasizes faithfulness. If the answer is largely hallucinated, the system discards it and either retries generation with a different temperature setting or returns only the information that is directly supported by the context.
Answer relevance grading checks whether the answer actually addresses the user's question. An answer might be faithful to the context but miss the point of the question. The relevance grader evaluates whether the answer contains information that helps the user and flags vague or off-topic responses.
The complete agentic RAG loop in LangGraph looks like this: route the query to the appropriate source, retrieve documents, grade the documents for relevance, correct retrieval if needed, generate an answer, check for hallucinations, check for answer relevance, and either return the answer or loop back for correction. Each of these is a node in the graph, with conditional edges implementing the decision logic.
This multi-stage evaluation makes the system dramatically more reliable than standard RAG. In practice, agentic RAG systems achieve 90-95% answer accuracy compared to 70-80% for standard RAG on diverse query sets. The tradeoff is latency (typically 2-4 seconds versus 1-2 seconds) and cost (2-3x more LLM calls), which is worthwhile for applications where accuracy matters.
Implementing the Complete Agentic RAG Graph
Let us walk through the complete LangGraph implementation. The state includes messages, the retrieved documents, the current question (which may be reformulated), generation output, and counters for retry limits.
The graph has seven nodes: route_query (determines retrieval strategy), retrieve (searches the knowledge base), grade_documents (evaluates relevance), web_search (fallback retrieval), generate (produces the answer), check_hallucinations (validates faithfulness), and check_answer_relevance (validates usefulness).
Conditional edges implement the control flow. After route_query, branch to either retrieve or web_search based on the routing decision. After grade_documents, either proceed to generate (if documents are relevant), reformulate and retry (if documents are irrelevant and retries remain), or go to web_search (if retries are exhausted). After check_hallucinations, either proceed to check_answer_relevance (if grounded) or loop back to generate (with a stricter prompt). After check_answer_relevance, either return the answer (if relevant) or loop back to reformulate and try again.
For production deployment, add checkpointing with PostgresSaver so the graph state survives restarts. Implement timeout handling: if any node takes longer than 10 seconds, skip it and proceed with a fallback path. Add comprehensive logging at each node so you can trace any query through the entire pipeline.
Monitor the graph by tracking metrics per node: how often does grading filter out documents, how often does hallucination detection trigger regeneration, and what percentage of queries require web search fallback. These metrics reveal where your knowledge base has gaps and where your retrieval pipeline needs tuning.
The complete implementation typically spans 200-300 lines of Python, which is manageable for a single developer. The graph structure makes it easy to add new capabilities: a new node for SQL retrieval, a new edge for routing to a human expert, or a new evaluation step for fact verification against a trusted source.
Code Example
from langgraph.graph import StateGraph, END
from langchain_openai import ChatOpenAI
from pydantic import BaseModel
class GradeDocuments(BaseModel):
relevant: bool # True if document is relevant to question
def grade_documents(state):
"""Grade retrieved documents for relevance."""
llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)
grader = llm.with_structured_output(GradeDocuments)
filtered = []
for doc in state["documents"]:
result = grader.invoke(
f"Document: {doc}\nQuestion: {state['question']}\n"
"Is this document relevant to the question?"
)
if result.relevant:
filtered.append(doc)
if len(filtered) < 2:
return {"documents": filtered, "needs_web_search": True}
return {"documents": filtered, "needs_web_search": False}