If your documents are in French or Spanish and your users ask questions in English, a standard RAG pipeline won’t work well. The usual fix is to translate every document into English before storing it, but that’s costly, slow, and hard to maintain. In this article, I’ll show you how to build a multi-language RAG pipeline with Python.
How Multi-Language RAG Works
A standard RAG pipeline stops working as soon as your users use a different language. The solution is simple, but it’s important to understand how it works.
The pipeline has four stages. First, the user’s query goes through a multilingual embedding model, which is the key step. Unlike standard models that focus on English, multilingual models like paraphrase-multilingual-mpnet-base-v2 are trained on over 50 languages at once. This means that words like “Dog” and “Chien” end up close together in vector space, since the model understands that meaning can cross languages.
Because of this shared vector space, retrieval works no matter the language. An English query can find a relevant French or Japanese document without needing translation; the similarity search handles it. Then, the retrieved text goes to an LLM with a prompt telling it to answer in the user’s language. Modern LLMs can do this kind of translation easily.
So, building a multi-language RAG isn’t about adding a translation step. It’s about picking the right embedding model from the beginning. Once you do that, the rest of the pipeline works just like a standard RAG setup.
If you’re serious about building real-world AI systems like this, my book Hands-On GenAI, LLMs & AI Agents walks you through everything step-by-step.
Building a Multi-Language RAG Pipeline: Step by Step
We’ll build this using LangChain, ChromaDB for our vector store, HuggingFace for multilingual embeddings, and Ollama to run Llama 3 locally as our LLM.
Before you continue, make sure you have Ollama installed and that you’ve downloaded the Llama 3 model with this command:
ollama pull llama3
And install the necessary Python packages:
pip install langchain langchain-huggingface langchain-chroma langchain-ollama
Step 1: The Imports and the Data
First, let’s set up our imports and define our documents. In practice, you’d load these from PDFs, Confluence, or databases. For this tutorial, we’ll use a short list of sample documents that represent an international company’s policies:
from langchain_core.documents import Document
from langchain_core.output_parsers import StrOutputParser
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.runnables import RunnablePassthrough
from langchain_huggingface import HuggingFaceEmbeddings
from langchain_chroma import Chroma
from langchain_ollama import ChatOllama
# Our multilingual knowledge base
documents = [
Document(
page_content="The company policy allows for up to 20 days of paid time off per year. Employees must request time off at least two weeks in advance.",
metadata={"language": "English", "source": "HR_Policy_EN"},
),
Document(
page_content="La politique de l'entreprise accorde jusqu'a 20 jours de conges payes par an. Les employes doivent en faire la demande au moins deux semaines a l'avance.",
metadata={"language": "French", "source": "HR_Policy_FR"},
),
Document(
page_content="El soporte tecnico esta disponible 24/7. Para problemas urgentes, llame al numero de emergencia en lugar de enviar un correo electronico.",
metadata={"language": "Spanish", "source": "IT_Policy_ES"},
),
]Step 2: The Prompt Strategy
The prompt tells the LLM how to behave. We need to make it clear that the context might be in a different language than the question, but the answer must always match the user’s language:
prompt = ChatPromptTemplate.from_template(
"""
You are a helpful assistant.
Use the retrieved context to answer the user's question.
The context may be in a different language than the question.
Answer in the same language as the user's question.
Context:
{context}
Question:
{question}
Answer:
""".strip()
)
# Helper function to combine our retrieved documents into a single string
def format_docs(docs: Iterable[Document]) -> str:
return "\n\n".join(doc.page_content for doc in docs)Step 3: Multilingual Embeddings
This is the most important part of the code. We’re using paraphrase-multilingual-MiniLM-L12-v2 from HuggingFace. It’s a lightweight and efficient model that supports more than 50 languages. This lets our English queries find Spanish and French data:
# A model trained to map different languages to the same vector space
embeddings = HuggingFaceEmbeddings(
model_name="sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2"
)
# Load the documents into a local Chroma vector database
vectorstore = Chroma.from_documents(
documents=documents,
embedding=embeddings,
collection_name="multilingual_hr_docs",
)
# Set up the retriever to fetch the top 2 most relevant chunks
retriever = vectorstore.as_retriever(search_kwargs={"k": 2})Step 4: The LLM and the LCEL Chain
We’ll start our local Llama 3 model using Ollama. By setting the temperature to 0, we make sure the answers are factual and based only on our documents, not on creative guesses:
# Initialize local LLM
llm = ChatOllama(model="llama3", temperature=0)
# Build the pipeline
rag_chain = (
{"context": retriever | format_docs, "question": RunnablePassthrough()}
| prompt
| llm
| StrOutputParser()
)Finally, we bring everything together with LangChain Expression Language (LCEL). This gives us a clear and readable pipeline: the user’s question goes to the retriever, the documents are formatted, sent to the prompt, passed to the LLM, and then returned as a string.
Step 5: Testing the Pipeline
Let’s try it out with two cross-language examples:
- An English user asking about a policy that only exists in Spanish.
- A French user asking about an HR policy.
# Test 1: English query hitting Spanish context
query_en = "How do I handle urgent IT issues?"
print(f"User: {query_en}")
print(f"AI: {rag_chain.invoke(query_en)}")
print("-" * 50)
# Test 2: French query hitting French/English context
query_fr = "Combien de jours de conges payes puis-je prendre ?"
print(f"User: {query_fr}")
print(f"AI: {rag_chain.invoke(query_fr)}")User: How do I handle urgent IT issues?
AI: According to the context, for urgent IT issues, you should call the emergency number instead of sending an email.
--------------------------------------------------
User: Combien de jours de conges payes puis-je prendre ?
AI: Selon la politique de l'entreprise, vous pouvez prendre jusqu'à 20 jours de congés payés par an.
Closing Thoughts
That’s how you can build a multi-language RAG pipeline with Python.
This pipeline shows that good AI architecture often depends on the strengths of the models themselves. By picking the right embedding model, we removed the need for a translation layer.
Thank you for reading. For additional AI and machine learning insights, follow me on Instagram. You may also find my book, Hands-On GenAI, LLMs & AI Agents, helpful for advancing your AI career.
