These days, most people prefer to ask questions using LLMs instead of reading long documents. However, just pasting a 100-page PDF into an LLM prompt won’t work. You’ll run into context limits, high API costs, or risk sharing sensitive company data with public models. The standard way to solve this is with RAG (Retrieval-Augmented Generation), which is a core technique in modern AI engineering. In this article, I’ll show you how to build a document Q&A system using Vector Databases and RAG.
Document Q&A System: Getting Started
In this tutorial, we’ll create a pipeline that lets you chat with an Apple Privacy Policy PDF. We’ll use LangChain to manage the workflow, Chroma as our local vector database, and Ollama to run the Llama 3 model on your own machine.
Before we dive into the code, you’ll need to install a few dependencies:
pip install langchain langchain-community langchain-text-splitters langchain-huggingface langchain-ollama chromadb sentence-transformers pypdf
You’ll also need to have Ollama installed to run a local LLM. After installing it, run this command in your terminal:
ollama pull llama3
Now, let’s start building a document Q&A system using Vector Databases and RAG.
Step 1: Document Loading & Splitting
First, we need to load our PDF and split it into smaller parts. LLMs have context windows, which means they can only process a certain amount of text at once. If we give them too much, they might miss important details:
from langchain_community.document_loaders import PyPDFLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
# Load PDF
loader = PyPDFLoader("/Users/amankharwal/document qna/apple-privacy-policy-en-ww.pdf")
documents = loader.load()
# Split into chunks
text_splitter = RecursiveCharacterTextSplitter(
chunk_size=1000,
chunk_overlap=200
)
chunks = text_splitter.split_documents(documents)
print(f"Split document into {len(chunks)} chunks.")Split document into 37 chunks.
Pay attention to chunk_overlap=200. This detail matters because if you simply split a document every 1,000 characters, you could end up cutting a sentence or important idea in half.
Overlapping chunks help keep the context from the end of one chunk at the start of the next, so important information isn’t lost.
Step 2: Creating the Vector Database
Next, we’ll turn our text chunks into embeddings and store them. We’ll use a lightweight, open-source HuggingFace model for the embeddings and Chroma as our database:
from langchain_huggingface import HuggingFaceEmbeddings
from langchain_community.vectorstores import Chroma
# Embeddings
embeddings = HuggingFaceEmbeddings(model_name="all-MiniLM-L6-v2")
# Vector DB
vectorstore = Chroma.from_documents(
documents=chunks,
embedding=embeddings,
persist_directory="./chroma_db"
)
print("Vector database created successfully!")Vector database created successfully!
The all-MiniLM-L6-v2 model works well for local projects. It’s small, fast, and maps the meaning of sentences into high-dimensional vector space effectively.
We save the database to ./chroma_db so you won’t need to recompute the embeddings each time you run the script.
Step 3: Initialize the LLM (Ollama)
Next, we need the part of the system that reads the retrieved text and creates an answer:
from langchain_ollama import OllamaLLM
llm = OllamaLLM(model="llama3")With Ollama, we can run Llama 3 locally. This is a big advantage for enterprise environments.
Your proprietary PDFs never leave your computer; everything stays local instead of being sent to an external API.
Step 4: Creating the RAG Pipeline
This is the most important part of our document Q&A system. Now we’ll connect everything into a working pipeline:
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.runnables import RunnablePassthrough
# Prompt
prompt = ChatPromptTemplate.from_template("""
You are a helpful assistant. Answer the question ONLY using the provided context.
<context>
{context}
</context>
Question: {question}
""")
# Retriever
retriever = vectorstore.as_retriever(search_kwargs={"k": 3})
# Format retrieved docs
def format_docs(docs):
return "\n\n".join(doc.page_content for doc in docs)
# LCEL Chain
rag_chain = (
{
"context": retriever | format_docs,
"question": RunnablePassthrough()
}
| prompt
| llm
)Let’s go over the key parts of the code above:
- The Strict Prompt: We clearly tell the LLM, “Answer the question ONLY using the provided context.” This helps prevent hallucination. We want the LLM to summarize our data, not make things up.
- The Retriever (k=3): When you ask a question, the vector database finds the top three most relevant chunks.
- The Chain (LCEL): LangChain Expression Language (the | symbols) moves data from one step to the next. The user’s question goes in, the retriever grabs the context, both are added to the prompt, and then Llama 3 gets the final prompt.
Step 5: Ask the Question
Once the pipeline is ready, you just need to run it:
question = "According to the document, why user's personal data is used by Apple?"
response = rag_chain.invoke(question)
print("\n--- Answer ---")
print(response)--- Answer ---
According to the document, Apple uses user's personal data for the following purposes:
1. To power our services
2. To process transactions
3. To communicate with you
4. For security and fraud prevention
5. To comply with law
6. With your consent (for other purposes)
Note that these purposes are listed as a valid legal basis for Apple to use user's personal data, along with relying on the user's consent or the fact that the processing is necessary to fulfill a contract with you, protect your vital interests or reasonably be used to identify you.
Behind the scenes, your question is turned into an embedding and compared to the Apple Privacy chunks in Chroma. The top three matches are added to your prompt, and Llama 3 creates a concise, accurate answer based only on that PDF.
Closing Thoughts
That’s the process for building a document Q&A system with Vector Databases and RAG.
Use this pipeline as a starting point. Try changing the chunk size, use a different embedding model, or ask tricky questions. Real confidence in AI engineering comes from understanding how everything works together, especially when things go wrong. Keep building and testing, and you’ll master this ecosystem sooner than you expect.
I hope you enjoyed the article! Follow me on Instagram for more AI and machine learning tips. You can also check out my book, Hands-On GenAI, LLMs & AI Agents, to get career-ready in AI.
