Many learners and enterprise clients often ask me about data privacy. People want to use Large Language Models (LLMs) to search their financial reports, legal contracts, or private journals. However, sending sensitive data to a third-party API online can be a big security risk. That’s why learning to create a ChatGPT for your personal documents that runs only on your own computer is such a valuable skill today.
As an AI/ML engineer, I have built many document-retrieval systems. I quickly learned that using only paid APIs increases costs and limits where and how you can use your applications.
Today, we will build a Local Retrieval-Augmented Generation (RAG) system using only free, open-source tools.
How Local RAG Works
To chat with a document, we use a method called Retrieval-Augmented Generation. Rather than fine-tuning a model, which is costly and hard to update, we give the LLM the information it needs as it works.
Here is exactly how the pipeline functions:
- Ingestion and Chunking: We load a document and break the text into smaller, manageable pieces called chunks. LLMs have context limits, so we can’t process a whole 200-page PDF at once.
- Embedding: We turn these text chunks into dense vectors. This lets the system compare your question and the text based on their meaning.
- Vector Storage: We save these vectors in a local database designed for quick similarity searches.
- Retrieval and Generation: When you ask a question, the system turns your query into a vector, finds the most relevant chunks in the database, and sends those chunks to a local LLM to create an accurate answer.
Let’s go through each step together.
ChatGPT for Your Personal Documents: Building the Pipeline
Before we start coding, make sure you have Ollama installed on your computer. Run ollama pull mistral in your terminal to download the local LLM. You’ll also need to install the required libraries using pip:
pip install langchain langchain-community langchain-text-splitters langchain-huggingface langchain-chroma langchain-ollama chromadb sentence-transformers pypdf
Step 1: Document Loading and Chunking
We start by loading our target PDF and splitting it:
from langchain_community.document_loaders import PyPDFLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
import os
# Load PDF
pdf_path = "annual-report-2025-2026.pdf" # Change if needed
if not os.path.exists(pdf_path):
raise FileNotFoundError(f"PDF not found: {pdf_path}")
loader = PyPDFLoader(pdf_path)
documents = loader.load()
print(f"Loaded {len(documents)} pages.")
# Split into chunks
text_splitter = RecursiveCharacterTextSplitter(
chunk_size=1000,
chunk_overlap=150,
)
chunks = text_splitter.split_documents(documents)
print(f"Created {len(chunks)} chunks.")Beginners often set the chunk size too large or forget to add overlap. Without overlap, you might cut an important sentence in half and lose its meaning.
Looking for more hands-on RAG and AI Agent projects? Check out my book Hands-On GenAI, LLMs & AI Agents.
Step 2: Embeddings and Vector Database
Next, we convert our text chunks into vectors:
from langchain_huggingface import HuggingFaceEmbeddings
from langchain_chroma import Chroma
# Embedding model
embeddings = HuggingFaceEmbeddings(
model_name="sentence-transformers/all-MiniLM-L6-v2"
)
# Create Chroma DB
vector_db = Chroma.from_documents(
documents=chunks,
embedding=embeddings,
persist_directory="./chroma_db",
)
print("Vector database created.")I recommend using a lightweight embedding model for local projects. In this guide, we use all-MiniLM-L6-v2 from HuggingFace. It’s fast, efficient, and works well on a regular CPU. We store these embeddings locally with ChromaDB.
Step 3: Setting Up the Local LLM and Strict Prompting
Now we set up the retriever to get the top 3 most relevant chunks (k=3). We’ll start our local Mistral model using Ollama:
from langchain_ollama import OllamaLLM
from langchain_core.prompts import ChatPromptTemplate
# Retriever
retriever = vector_db.as_retriever(
search_kwargs={"k": 3}
)
# Local LLM
local_llm = OllamaLLM(
model="mistral"
)
# Prompt
prompt = ChatPromptTemplate.from_template(
"""
You are a helpful AI assistant.
Answer ONLY using the provided context.
If the answer is not found in the context, say:
"I couldn't find that information in the document."
Context:
{context}
Question:
{question}
Answer:
"""
)Pay attention to how the prompt is designed. In business settings, hallucination is a serious problem. I always give clear instructions in the prompt, telling the model to say “I couldn’t find that information” instead of making something up.
Step 4: Creating the RAG Chain and Chat Interface
Finally, we tie everything together using LangChain Expression Language (LCEL):
from langchain_core.output_parsers import StrOutputParser
from langchain_core.runnables import RunnablePassthrough
# Helper
def format_docs(docs):
return "\n\n".join(doc.page_content for doc in docs)
# RAG Chain
chain = (
{
"context": retriever | format_docs,
"question": RunnablePassthrough(),
}
| prompt
| local_llm
| StrOutputParser()
)
# Chat Loop
print("\nLocal RAG is ready!")
print("Type 'exit' to quit.\n")
while True:
question = input("You: ")
if question.lower() in ["exit", "quit"]:
break
answer = chain.invoke(question)
print("\nAssistant:")
print(answer)
print("-" * 60)Local RAG is ready!
Type 'exit' to quit.
You: What is this document about
Assistant:
This document appears to be a Business Responsibility and Sustainability Report, which covers various aspects related to the business, including its offerings, business and operations footprint, employees, related parties, Corporate Social Responsibility (CSR), and transparency. The report also includes disclosures related to the management and processes put in place towards adopting the NGRBC Principles and Core Elements. This information is distributed across Sections A, B, and C of the document, with Section B providing indicator-wise disclosures mapped to the nine principles of NGRBC. The report also includes Financial Statements, and it extends from page 136 to page 182.
The helper function turns the retrieved documents into plain text. The chain takes the user’s question, finds the documents, formats them for the prompt, and sends everything to Mistral.
Recommended Courses for Building RAG Applications
If you want to deepen your understanding of Retrieval-Augmented Generation and scale these local projects into enterprise-level systems, there are two programs I highly recommend:
- Retrieval Augmented Generation: This course is perfect for mastering the core mechanics we just covered. It provides hands-on experience with chunking, indexing, and retrieving documents using vector databases to build reliable, production-ready RAG pipelines.
- IBM RAG and Agentic AI Professional Certificate: Once you understand the basics of local RAG, this program helps you take the next step. It focuses on using frameworks like LangChain to implement function calling, integrate vector stores, and build intelligent, context-aware AI applications.
Both of these programs bridge the gap between running local scripts and designing robust AI systems, making them excellent additions to your learning path.
Final Thoughts
Building this local pipeline not only keeps your data safe, but also helps you learn how modern AI systems really work. If you only use managed cloud services, you miss out on understanding the details.
When you manage your own document loaders, try different chunk overlaps, and choose open-source embedding models, you move from just calling APIs to actually building real AI solutions.
I hope you found this article on creating a ChatGPT for your personal documents helpful.
For more tips on AI and machine learning, follow me on Instagram. My book, Hands-On GenAI, LLMs & AI Agents, can also help you grow your AI career.
