I can’t tell you how many times I’ve visited a documentation site, a company blog, or a knowledge base, only to spend 15 minutes searching for one answer. Most sites use basic keyword search, so you type in your question and get a list of links to sift through. It’s frustrating, and as engineers, we should be able to improve this. Today, I’ll show you how to fix this problem by turning any website into an AI chatbot with Retrieval-Augmented Generation (RAG).
How RAG Actually Works
Before we start coding, let’s talk about why we need RAG in the first place.
Large Language Models (LLMs) like LLaMA 3 are powerful, but their knowledge is stuck at a certain point in time. They don’t know anything about your website, private documents, or current events. If you ask a regular LLM about a specific internal company blog post, it will probably make up an answer.
RAG fixes this by changing the process. Instead of depending on the LLM’s memory, here’s what we do:
- Scrape and Chunk: We grab the text from the website and split it into smaller, easy-to-read pieces.
- Embed and Index: We turn those text chunks into mathematical vectors (embeddings) and save them in a database.
- Retrieve: When someone asks a question, we turn it into a vector, search the database for the most similar chunks, and pull them out.
- Generate: We give those specific chunks to the LLM and tell it, “Answer the user’s question using only this text.”
Turning Any Website into an AI Chatbot: Getting Started
Let’s dive in. We’ll use Lilian Weng’s well-known blog post on LLM Agents as our example website. First, make sure you have Ollama installed on your computer. After that, pull the LLaMA 3 model and install the needed Python libraries:
ollama pull llama3pip install -U langchain langchain-community langchain-text-splitters langchain-huggingface langchain-ollama faiss-cpu sentence-transformers beautifulsoup4
Step 1: Loading the Website Data
The first step in any data pipeline is getting the data in:
import os
# Set a user agent so WebBaseLoader does not complain
os.environ["USER_AGENT"] = "Mozilla/5.0 (compatible; WebsiteRAGChatbot/1.0)"
from langchain_community.document_loaders import WebBaseLoader
url = "https://lilianweng.github.io/posts/2023-06-23-agent/"
print("Loading website...")
loader = WebBaseLoader(url)
documents = loader.load()
print(f"Loaded {len(documents)} document(s).")We use LangChain’s WebBaseLoader to scrape the text. Here’s a quick tip: many websites block default Python scrapers, so setting a custom USER_AGENT can help you avoid scraping errors.
Want to build more practical GenAI applications like this? My book, Hands-On GenAI, LLMs & AI Agents, teaches you how to build LLM, RAG, and AI Agent projects step by step.
Step 2: Chunking the Text
You can’t feed a whole website into an LLM at once. It will go over the context window and make the search less accurate. So, we break the text into smaller, manageable pieces.
I strongly suggest using the RecursiveCharacterTextSplitter. It keeps paragraphs and sentences together, which helps keep the meaning of the text better than just cutting every 500 characters:
from langchain_text_splitters import RecursiveCharacterTextSplitter
text_splitter = RecursiveCharacterTextSplitter(
chunk_size=500,
chunk_overlap=50
)
chunks = text_splitter.split_documents(documents)
print(f"Split the website into {len(chunks)} chunks.")Notice the chunk_overlap=50. This way, if an important sentence gets split at the 500-character mark, the context continues into the next chunk.
Step 3: Embeddings and the Vector Database
from langchain_huggingface import HuggingFaceEmbeddings
print("Loading embedding model...")
embeddings = HuggingFaceEmbeddings(
model_name="sentence-transformers/all-MiniLM-L6-v2"
)
print("Embedding model loaded.")
# 4. Create FAISS vector database
from langchain_community.vectorstores import FAISS
print("Creating vector database...")
vector_store = FAISS.from_documents(
documents=chunks,
embedding=embeddings
)
print("Vector database created.")Next, we turn our text chunks into embeddings using a lightweight, open-source model from Hugging Face (all-MiniLM-L6-v2). This model runs fast on a CPU and gives great accuracy for regular text. We then save these vectors in FAISS, a quick, local vector database made by Meta.
Step 4: Setting Up Retrieval and the LLM
retriever = vector_store.as_retriever(
search_kwargs={"k": 3}
)
from langchain_ollama import OllamaLLM
print("Loading Ollama model...")
llm = OllamaLLM(
model="llama3"
)
print("Ollama model loaded.")Now we set up our retriever to get the top 3 most relevant chunks (k=3) for any question. Then, we start our local LLaMA 3 model with Ollama. There are no API keys, no cloud delays, and your data stays private.
Step 5: Prompt Engineering and Formatting
from langchain_core.prompts import ChatPromptTemplate
prompt = ChatPromptTemplate.from_template(
"""
You are a helpful AI assistant that answers questions based
only on the provided website content.
Use the context below to answer the user's question.
If the answer cannot be found in the context, say:
"I couldn't find the answer in the provided website."
Do not make up information.
Context:
{context}
Question:
{question}
Answer:
"""
)
def format_documents(documents):
return "\n\n".join(
document.page_content
for document in documents
)A common mistake in early AI projects is using weak prompts. If you want your chatbot to use only your website, you need to clearly tell it not to use its own internal knowledge. Pay attention to how strict the system instructions are in the prompt.
Step 6: Asking Questions and Generating Answers
user_question = "What are the core components of an AI agent according to this site?"
print("\nSearching the website...\n")
retrieved_documents = retriever.invoke(user_question)
print(f"Retrieved {len(retrieved_documents)} relevant chunks.")
context = format_documents(retrieved_documents)
final_prompt = prompt.invoke(
{
"context": context,
"question": user_question
}
)
print("\nGenerating answer...\n")
response = llm.invoke(final_prompt)
print("=" * 60)
print("CHATBOT ANSWER")
print("=" * 60)
print(response)
print("=" * 60)Searching the website...
Retrieved 3 relevant chunks.
Generating answer...
============================================================
CHATBOT ANSWER
============================================================
According to the provided website context, the core component of a LLM-powered autonomous agent system is:
* Planning: The brain (LLM) of the agent system
* Component One: Planning: A complicated task involves many steps, and the agent needs to know what they are and plan ahead.
* Task Decomposition: Not specified as a separate component, but mentioned as part of planning.
Note that the other components mentioned in the context include "relationships between agents and observations of one agent by another", "environment information" (present in a tree structure), but these are not explicitly listed as core components.
Finally, we bring everything together. We ask a question, get the most relevant text chunks, put them into our prompt, and let LLaMA 3 create a clear, accurate answer.
Recommended Courses for Building AI Chatbots
If you want to go further than this tutorial and learn how to scale these systems for enterprise use, here are two programs I highly recommend:
- Retrieval Augmented Generation: This course is great for learning the main techniques we just covered. You’ll get hands-on practice with chunking, vector databases, and advanced retrieval methods to build solid, production-ready RAG pipelines.
- IBM RAG and Agentic AI Professional Certificate: After you know the basics of local RAG, this certificate helps you go further. It teaches you how to use frameworks like LangChain to connect vector stores, use function calling, and build autonomous, multi-agent AI systems.
Both programs emphasize the rigorous, hands-on software engineering mindset you need to transition from building simple scripts to deploying robust AI systems.
Final Thoughts
When I first got into AI engineering, I thought the key to a great chatbot was using the biggest, most expensive LLM out there. Building systems like this showed me that’s not true.
The real secret is in data quality and how you set up your pipeline. An average model with great, precise context will almost always do better than a huge, advanced model that has to guess.
I hope you enjoyed this article on turning any website into an AI chatbot with RAG.
For more tips on AI and machine learning, follow me on Instagram. My book, Hands-On GenAI, LLMs & AI Agents, can also help you grow your AI career.
