Large Language Models are great at understanding text, but most real-world documents include more than just words. Manuals, research papers, financial reports, medical records, and engineering documents often mix text with diagrams, charts, and images. A Multi-Modal RAG Pipeline is important here because it retrieves both text and visuals, letting an AI model work with both types of information.
Over the past year, I’ve built several Retrieval-Augmented Generation applications. One thing I’ve learned is that most real-world documents are multimodal. Traditional RAG often misses diagrams, which can hold the most important information.
In this tutorial, I’ll walk you through building a simple Multi-Modal RAG pipeline using only free and open-source models.
What is a Multi-Modal RAG Pipeline?
A traditional RAG pipeline follows a straightforward workflow:
- Extract text from documents.
- Convert text into embeddings.
- Store embeddings in a vector database.
- Retrieve relevant chunks.
- Send retrieved context to an LLM.
A Multi-Modal RAG pipeline adds image processing to this workflow. Here’s how the full setup works:
PDF / Documents
│
▼
Extract Text + Images
│
├──────────────┐
▼ ▼
Text Embeddings Image Embeddings
│ │
└──────┬───────┘
▼
Vector Database
│
▼
Similarity Retrieval
│
▼
Vision Language Model (VLM)
│
▼
Final Multimodal Answer
With this approach, the model can answer questions that need it to understand diagrams, screenshots, flowcharts, and illustrations along with the text.
Build a Multi-Modal RAG Pipeline
Step 1: Install Required Libraries
I recommend keeping your setup simple. This example uses only open-source libraries:
pip install pymupdf
pip install pillow
pip install sentence-transformers
pip install chromadb
pip install transformers
pip install torch
Step 2: Extract Text and Images from the PDF
The first step is to process the PDF. Rather than just pulling out the text, we also extract every image inside:
import fitz
doc = fitz.open("manual.pdf")
texts = []
images = []
for page in doc:
texts.append(page.get_text())
for img in page.get_images(full=True):
xref = img[0]
base_image = doc.extract_image(xref)
image_bytes = base_image["image"]
image_ext = base_image["ext"]
filename = f"image_{xref}.{image_ext}"
with open(filename, "wb") as f:
f.write(image_bytes)
images.append(filename)Here’s what’s happening:
- The PDF is opened with PyMuPDF.
- Every page’s text is extracted.
- Every embedded image is saved locally.
- We now have two collections: Page text and Images.
A common mistake is to extract only the text. Many manuals use diagrams to explain key ideas, so leaving out images can really hurt answer quality.
Want to build more real-world RAG and AI Agent systems? My book Hands-On GenAI, LLMs & AI Agents teaches LLMs, RAG, and Agentic AI through practical, hands-on projects.
Step 3: Generate Text Embeddings
The extracted text must now be converted into vectors:
from sentence_transformers import SentenceTransformer
text_model = SentenceTransformer(
"BAAI/bge-small-en-v1.5"
)
text_embeddings = text_model.encode(texts)I picked BAAI/bge-small-en-v1.5 for a few reasons:
- Free
- Lightweight
- Fast
- Produces strong semantic embeddings
Each page of text is turned into a set of numbers that capture its meaning. This lets you search by meaning, not just by keywords.
Step 4: Generate Image Embeddings
Images also need to be turned into vectors so the retrieval system can understand them:
from transformers import CLIPProcessor, CLIPVisionModelWithProjection
from PIL import Image
processor = CLIPProcessor.from_pretrained(
"openai/clip-vit-base-patch32"
)
model = CLIPVisionModelWithProjection.from_pretrained(
"openai/clip-vit-base-patch32"
)
image_vectors = []
for image_path in images:
image = Image.open(image_path).convert("RGB")
inputs = processor(images=image, return_tensors="pt")
outputs = model(**inputs)
features = outputs.image_embeds
image_vectors.append(
features.cpu().detach().numpy()
)CLIP turns each image into a semantic embedding. These vectors capture what’s in the image, so you can find similar images or diagrams later.
While this example just creates image embeddings, a real production pipeline should also save these image vectors in a vector database and keep track of which page each image came from. This makes it easier to pull up the right diagram with its matching text.
Step 5: Store Embeddings in ChromaDB
Now we create a vector database:
import chromadb
client = chromadb.Client()
collection = client.create_collection(
"documents"
)
for i, embedding in enumerate(text_embeddings):
collection.add(
ids=[f"text_{i}"],
embeddings=[embedding.tolist()],
documents=[texts[i]]
)ChromaDB organizes the text embeddings so you can quickly find pages with similar meaning when someone asks a question.
Step 6: Retrieve Relevant Context
Suppose a user asks:
Explain the diagram shown in the retrieved owner’s manual page.
We first convert the query into an embedding and search the vector database:
query = "Explain the diagram shown in the retrieved owner's manual page."
query_embedding = text_model.encode(query)
results = collection.query(
query_embeddings=[query_embedding.tolist()],
n_results=3
)
print(results["documents"])[['All information in this Owner’s Manual is current at the time \nof publication. However, HYUNDAI reserves the right to make \nchanges at any time so that our policy of continual product \nimprovement may be carried out.\nThis manual applies to all models of this vehicle and includes \ndescriptions and explanations of optional as well as standard \nequipment
...
........................................................... 3-3\n* : if equipped\nINTERIOR OVERVIEW\n']]
Instead of looking for exact keywords, the system finds the pages from the manual that are most relevant in meaning.
Step 7: Generate the Final Multimodal Answer
Next, the retrieved text is combined with its matching image and sent to a Vision Language Model:
from transformers import pipeline
pipe = pipeline(
task="image-text-to-text",
model="Qwen/Qwen2.5-VL-3B-Instruct",
device_map="auto"
)
context = results["documents"][0][0]
messages = [
{
"role": "user",
"content": [
{
"type": "image",
"image": "/content/image_1025.jpeg"
},
{
"type": "text",
"text": f"""
You are an expert assistant.
Use the manual context below to explain the image.
Manual Context:
{context}
Question:
Explain this diagram in detail.
"""
}
]
}
]
response = pipe(
messages,
max_new_tokens=512
)
print(response[0]["generated_text"])The Qwen2.5-VL-3B-Instruct model takes in both the document context and the image. This lets it give an explanation that uses both text and visuals. This kind of multimodal reasoning makes the pipeline much more effective than a text-only RAG system.
Final Thoughts
Building a Multi-Modal RAG Pipeline might seem complicated at first. But if you break it down into steps like document processing, embedding generation, vector storage, retrieval, and multimodal reasoning, the whole setup becomes much clearer.
When you combine semantic retrieval with a strong Vision Language Model like Qwen2.5-VL-3B-Instruct, you can create assistants that understand documents more like people do, by looking at both the words and the visuals together.
I hope you found this article on building a Multi-Modal RAG Pipeline helpful.
For more AI and machine learning tips, follow me on Instagram. My book, Hands-On GenAI, LLMs & AI Agents, can also help you advance your AI career.
