Many teams working with Large Language Models quickly encounter challenges: models may generate inaccurate information, lack access to company-specific data, and become costly when processing large document archives. A reliable RAG cheat sheet is now essential.
In mentoring engineers moving into Generative AI, I have observed that developers often overcomplicate Retrieval-Augmented Generation by focusing on complex frameworks and overlooking retrieval fundamentals. If the retrieval mechanism provides irrelevant context, even the best model will produce inaccurate answers.
This article provides a clear, end-to-end overview of RAG architecture, using only open-source, local tools that run on your machine without requiring paid APIs.
RAG Cheat Sheet
1. Document Ingestion and Text Extraction
RAG starts by parsing unstructured files into clean, raw text. The quality of this text directly affects downstream embeddings. Poor formatting, extraneous headers and footers, or corrupted Unicode can reduce retrieval accuracy.
For local PDF and document processing, open-source libraries such as pypdf, pdfplumber, or unstructured efficiently handle parsing without the need for external cloud APIs:
from pypdf import PdfReader
def extract_text(pdf_path: str) -> str:
reader = PdfReader(pdf_path)
text = ""
for page in reader.pages:
page_text = page.extract_text()
if page_text:
text += page_text + "\n"
return text2. Chunking Strategies
Embedding models have token limits, and large documents can dilute semantic meaning. Therefore, it is necessary to divide text into manageable chunks:
- Fixed-size with overlap: Divides text by a set number of characters or tokens (for example, 500 characters with a 50-character overlap). Overlap helps preserve sentence continuity.
- Recursive character splitting: This standard approach splits text hierarchically by paragraphs (\n\n), newlines (\n), and spaces, only using character cutoffs as a last resort to maintain semantic boundaries.
- Semantic or document-aware chunking: Divides text by Markdown headers or section titles. This method is essential for technical documentation and structured reports.
def recursive_chunking(text: str, chunk_size: int = 500, overlap: int = 50) -> list[str]:
chunks = []
start = 0
while start < len(text):
end = start + chunk_size
chunks.append(text[start:end])
start += chunk_size - overlap
return chunks3. Embedding Models
Embeddings transform text chunks into dense numerical vectors that capture semantic meaning. Commercial APIs are not required for production-quality vector representations; open-source models from Hugging Face deliver strong retrieval performance locally.
The sentence-transformers library supports lightweight, high-performance models such as all-MiniLM-L6-v2 or BAAI/bge-small-en-v1.5, which can run directly on a CPU or local GPU:
from sentence_transformers import SentenceTransformer
# Free, open-source, runs locally on CPU/GPU
embedding_model = SentenceTransformer("all-MiniLM-L6-v2")
chunks = ["Retrieval-Augmented Generation improves accuracy.", "Vector databases store embeddings."]
embeddings = embedding_model.encode(chunks)4. Vector Databases
Vector databases index dense representations and perform fast similarity searches, typically using cosine similarity or Euclidean distance, across millions of vectors.
For lightweight local development and embedded production, ChromaDB operates in-memory or on local disk without requiring a server cluster:
import chromadb
# Initialize local Chroma client
client = chromadb.Client()
collection = client.get_or_create_collection(name="local_rag_store")
# Store embeddings alongside chunk metadata
collection.add(
documents=chunks,
embeddings=embeddings.tolist(),
ids=[f"chunk_{i}" for i in range(len(chunks))]
)Preparing for AI/ML Interviews?
Understanding RAG is important, but articulating chunking strategies, embedding models, and retrieval trade-offs in an interview requires additional preparation. I recommend Cracking Your First AI/ML Interview as a helpful resource.
5. Semantic vs Hybrid Search
Dense semantic search retrieves text based on meaning and is effective for conversational queries. However, it may not perform well with exact keyword matches, such as product serial numbers, legal clauses, or code identifiers.
Semantic Search finds documents conceptually similar to the query.
Keyword Search (BM25) finds documents containing the exact tokens.
Hybrid search combines dense semantic retrieval with sparse keyword matching, such as BM25, using algorithms like Reciprocal Rank Fusion (RRF). In production, hybrid search is typically the best option to address cases that pure vector search may miss.
6. Reranking
Retrieving top-k results from a vector database is efficient, but vector distance alone may lack precision. A cross-encoder reranker evaluates the user query with each candidate chunk and reorders them by relevance.
Incorporating a lightweight local cross-encoder model, such as cross-encoder/ms-marco-MiniLM-L-6-v2, to refine the top 20 vector matches to the best 3 context chunks can significantly improve answer precision:
from sentence_transformers import CrossEncoder
reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
query = "How to reduce hallucinations in LLMs?"
retrieved_chunks = [
"RAG grounds LLM outputs using retrieved document contexts.",
"Data preprocessing cleans up bad character encoding in texts."
]
scores = reranker.predict([(query, chunk) for chunk in retrieved_chunks])
ranked_chunks = [chunk for _, chunk in sorted(zip(scores, retrieved_chunks), reverse=True)]7. Prompt Construction and Local Inference
The final step is to ground the model. Create a structured prompt that includes the reranked chunks and instructs the model to answer only using the provided evidence.
For local inference, open-source models such as Llama 3, Mistral, or Qwen can be run locally using Ollama or transformers:
def build_rag_prompt(query: str, context_chunks: list[str]) -> str:
context = "\n---\n".join(context_chunks)
prompt = f"""You are a precise technical assistant. Answer the question using ONLY the provided context below.
If the answer cannot be determined strictly from the context, state clearly that you do not know.
Context:
{context}
Question: {query}
Answer:"""
return promptThe Takeaway
When engineers encounter difficulties with RAG implementations, the problem is rarely the generation model. Issues typically arise upstream, such as noisy PDF parsing, poorly segmented chunks, or exclusive reliance on semantic search when exact keywords are needed.
RAG should not be viewed as a simple wrapper for an LLM; it is an information retrieval discipline. Focus on understanding how documents are chunked, indexed, and reranked. Mastering each stage of the retrieval pipeline enables you to build reliable systems that perform well in production.
Thank you for reading this article on the RAG cheat sheet. For more insights on AI and machine learning, please follow me on Instagram.
