Standard RAG works well for simple Q&A, but it struggles with complex, multi-part questions. For example, if you need to analyze a 100-page financial report for risks and future outlooks, a basic vector search often returns scattered, out-of-context pieces. The answer isn’t just better embeddings. Instead, you need to let the LLM reason, plan, and search in steps. If you want to go beyond basic Q&A bots and tackle real enterprise challenges, learning to build an end-to-end Agentic RAG system is essential.
Today, I’ll show you step by step how to build one using Python and CrewAI.
What is an Agentic RAG System?
A traditional RAG pipeline follows a relatively simple workflow:
→ Load documents
→ Split them into chunks
→ Generate embeddings
→ Store embeddings in a vector database
→ Retrieve relevant chunks
→ Send retrieved context to an LLM
This approach is effective for many uses, but it still depends on a fixed workflow.
An Agentic RAG system adds an autonomous AI agent to the process. Rather than always retrieving documents the same way, the agent decides when to search, which tools to use, how to analyze the information, and how to create a solid response.
In real-world AI systems, this approach makes the application much more reliable because the language model doesn’t have to rely only on its own knowledge.
End-to-End Agentic RAG System
The workflow we’ll build looks like this:
PDF → Document Loader → Text Splitter → Embeddings → Chroma Vector Database → Retriever → CrewAI Tool → AI Agent → Final Response
Each part has its own job, which makes the whole system modular and easy to expand.
Before getting started, you need to install all the required libraries with the following command:
pip install crewai langchain-community langchain-text-splitters langchain-huggingface langchain-chroma chromadb sentence-transformers pypdf
Also make sure to install Ollama on your system and pull the llama model into your directory:
ollama pull llama3.2
Step 1: Initialize the Local LLM
The first step is creating the language model that powers our AI agent:
from crewai import LLM
llm = LLM(
model="ollama/llama3.2",
base_url="http://localhost:11434"
)I use Ollama because it lets me run the Llama model on my own machine without needing outside APIs. When developing, I like local models since they cut API costs, boost privacy, and speed up testing.
For production, you can swap in any CrewAI-supported model, like OpenAI, Gemini, Claude, or Mistral.
Step 2: Load the Report
Next, we load the company’s annual report:
from langchain_community.document_loaders import PyPDFLoader
loader = PyPDFLoader(
"/Users/amankharwal/agentic_rag/annual-report-2025-2026.pdf"
)
documents = loader.load()PyPDFLoader extracts text from every page of the PDF while preserving document structure.
I’ve learned to always check the extracted text before creating embeddings. Many PDFs have formatting problems, scanned images, or broken tables that can hurt retrieval quality.
Step 3: Split Large Documents into Chunks
Large Language Models cannot process hundreds of pages at once. Instead, we divide the report into smaller overlapping chunks:
from langchain_text_splitters import RecursiveCharacterTextSplitter
splitter = RecursiveCharacterTextSplitter(
chunk_size=800,
chunk_overlap=150
)
chunks = splitter.split_documents(documents)I usually use chunk sizes of 500 to 1000 characters for financial reports. This gives enough context and keeps retrieval focused.
Overlap is just as important. Without it, key information near the edges of chunks can get lost.
Step 4: Generate Semantic Embeddings
The next stage converts text into vector representations:
from langchain_huggingface import HuggingFaceEmbeddings
embedding_model = HuggingFaceEmbeddings(
model_name="sentence-transformers/all-MiniLM-L6-v2"
)Embeddings enable semantic search rather than keyword matching.
For example, a query about “company revenue growth” can retrieve sections discussing “increase in sales” even if the exact words don’t appear.
The all-MiniLM-L6-v2 model is one of my favorites because it’s lightweight, fast, and gives strong retrieval results for many business documents.
Step 5: Store Embeddings in ChromaDB
Now we build the vector database:
from langchain_chroma import Chroma
vectorstore = Chroma.from_documents(
documents=chunks,
embedding=embedding_model,
persist_directory="./company_db"
)Chroma stores every embedding, enabling extremely fast similarity searches.
Saving the database locally means you only need to create embeddings once. After that, you can reuse the same vector database every time.
This is especially helpful when you have thousands of documents.
Want to build more real-world GenAI systems? My book Hands-On GenAI, LLMs & AI Agents covers RAG, AI Agents, and production-ready AI projects step by step.
Step 6: Build the Retriever
Next, we create the retriever:
retriever = vectorstore.as_retriever(
search_kwargs={"k": 3}
)The retriever searches the vector database and returns the three most relevant chunks.
Picking the right value for k takes some trial and error. If you use too few documents, you might miss important context. If you use too many, you use more tokens and may get irrelevant information.
For most enterprise uses, I start with values between 3 and 5, then check how well retrieval works.
Step 7: Convert Retrieval into a CrewAI Tool
One of my favorite features in CrewAI is tool integration.
Instead of giving the agent direct access to the vector database, we expose retrieval as a reusable tool:
from crewai.tools import tool
@tool("Financial Report Search")
def search_company_docs(query: str) -> str:
"""
Search the company's annual financial report and return
the most relevant sections related to the user's query.
"""
docs = retriever.invoke(query)
if not docs:
return "No relevant information found."
return "\n\n".join(doc.page_content for doc in docs)This is what gives the system its agentic quality. Instead of always running retrieval before every prompt, the agent chooses when to use the search tool.
This leads to a smarter and more flexible reasoning process.
Step 8: Create the AI Agent
Now we define our CrewAI agent:
from crewai import Agent
assistant = Agent(
role="Financial Report Analysis Expert",
goal="""
Analyze a company's annual financial report to answer questions,
explain financial performance, identify risks, summarize key
sections, and provide evidence-based insights.
""",
backstory="""
You are an experienced financial analyst.
Before answering any question, you search the annual report,
including the financial statements, Management Discussion &
Analysis (MD&A), Notes to Accounts, Risk Factors,
Corporate Governance, and Auditor's Report.
Every answer must be supported by evidence from the report.
""",
tools=[search_company_docs],
llm=llm,
verbose=True
)The role, goal, and backstory help shape how the agent reasons.
I always take time to write these prompts carefully because good prompts make a big difference in how consistently the agent behaves.
Another best practice I use is telling the agent to back up every answer with evidence from the documents, not just its own knowledge.
Step 9: Define the Task
The task tells the agent exactly what it needs to accomplish:
from crewai import Task
task = Task(
description="""
Analyze the company's annual report and answer the following:
- How did the company perform financially this year?
- What were the major business highlights?
- What are the key risks?
- What is management's future outlook?
- Mention important financial numbers whenever available.
Use only the information from the annual report.
""",
expected_output="""
A detailed financial report with headings, bullet points,
important financial metrics, business highlights,
major risks, future outlook, and evidence from
the annual report.
""",
agent=assistant
)Giving a clear expected output makes the responses much better.
Instead of asking for a general summary, we list out headings, financial metrics, risks, business highlights, and future outlook.
This removes confusion and makes the responses more organized.
Step 10: Execute the Crew
Finally, we execute the workflow:
from crewai import Crew
crew = Crew(
agents=[assistant],
tasks=[task],
verbose=True
)
result = crew.kickoff()
print("\n" + "=" * 80)
print("FINAL ANSWER")
print("=" * 80)
print(result)
These are some snapshots of the final output.
CrewAI orchestrates the entire reasoning process.
The agent decides when to use the retrieval tool, collects evidence from the annual report, thinks through the information, and creates a well-supported answer.
This kind of autonomous workflow is what sets an Agentic RAG application apart from a traditional retrieval pipeline.
Final Thoughts
If you’re learning AI engineering now, I suggest moving past basic chatbots and traditional RAG pipelines. Try building systems where agents can search, analyze, and use tools on their own. These are the types of setups driving modern enterprise AI in finance, healthcare, legal tech, customer support, and knowledge management.
The more Agentic RAG apps you build, the more you’ll see that reliable AI doesn’t just come from bigger models. It comes from good system design, strong retrieval pipelines, and agents that know when and how to use the right information.
I hope you liked this article on how to build an end-to-end Agentic RAG system.
For more AI and machine learning tips, follow me on Instagram. My book, Hands-On GenAI, LLMs & AI Agents, can also help you advance your AI career.
