Deploying a RAG-based app or AI agent on a production Linux server can be tricky. Fixing one small issue might cause several new ones. Setting up custom evals for your LLM pipeline is the best way to make sure your generative AI apps stay reliable.
Today, I’ll show you how I build these evaluation systems. I spend a lot of time mentoring data scientists and writing about AI engineering, so I focus on practical problem-solving. We won’t cover abstract theories here. Instead, I’ll walk you through how we actually test language models in practice.
The Core Components of an Evaluation System
To build a reliable pipeline, you need to focus on three main parts:
- The Golden Dataset: This is your ground truth. It’s a carefully chosen list of input prompts, the context given to the model, and the ideal responses. In my projects, I usually start with 50 to 100 diverse test cases. You don’t need thousands of examples; you just need high-quality, representative data.
- Evaluation Criteria: Decide what makes a “good” answer. I usually look at two things: Answer Relevance (did the model answer the user’s question, or did it go off-topic?) and Faithfulness (did the model make things up, or did it stick to the provided context?).
- The Evaluators: This is how we score the criteria. We use both traditional mathematical metrics and newer LLM-as-a-judge methods.
Setting Up Custom Evals for an LLM Pipeline
Let’s look at how I put this into practice in my daily workflow. For this setup, we’ll use sentence-transformers for semantic similarity (a traditional metric) and Ollama, running a local model like Llama 3 or Mistral, as our LLM judge.
For that, you need to prepare the environment with:
pip install sentence-transformers scikit-learn numpy langchain-community ollama
You’ll also need to install Ollama separately. Once it’s installed, run:
ollama pull llama3
Step 1: Defining the Golden Dataset
First, we need our test cases. In a real application, you would load this from a CSV or a database, but for clarity, let’s define a small dictionary of test cases representing a simple RAG (Retrieval-Augmented Generation) scenario:
# Our Golden Dataset
eval_dataset = [
{
"query": "What is the return policy?",
"context": "Items can be returned within 30 days of purchase with a receipt.",
"golden_answer": "You can return items within 30 days if you have the receipt.",
"model_output": "Our return policy allows returns up to 30 days from purchase, provided you show the receipt."
# (Imagine this output was generated by your main LLM pipeline)
},
{
"query": "Do you offer free shipping?",
"context": "Shipping is a flat rate of $5 for all orders under $50. Orders over $50 ship free.",
"golden_answer": "Free shipping is available for orders over $50.",
"model_output": "Yes, we always offer free shipping on all orders."
# (This is a hallucination/error we want our eval to catch)
}
]Step 2: Traditional Metrics (Semantic Similarity)
Metrics like ROUGE or BLEU are popular, but I rarely use them for conversational AI because they penalize creative phrasing by only counting matching words. Instead, Semantic Similarity is very useful. We turn texts into vector embeddings and measure the distance between them. If the cosine similarity score is close to 1.0, the meanings match:
from sentence_transformers import SentenceTransformer
from sklearn.metrics.pairwise import cosine_similarity
import numpy as np
# Load a small, fast open-source embedding model
embedding_model = SentenceTransformer('all-MiniLM-L6-v2')
def calculate_semantic_similarity(golden, output):
"""Calculates cosine similarity between expected and actual output."""
embeddings = embedding_model.encode([golden, output])
# Compare the two vectors
similarity = cosine_similarity([embeddings[0]], [embeddings[1]])[0][0]
return round(similarity, 4)
# Test it
for data in eval_dataset:
score = calculate_semantic_similarity(data["golden_answer"], data["model_output"])
print(f"Similarity Score: {score}")Step 3: LLM-as-a-Judge
Semantic similarity tells us if the “vibe” is right, but it can miss subtle factual errors. To catch hallucinations, we need an LLM to judge the output. I use langchain_community.llms with Ollama to run a free, local model that grades our pipeline.
The secret to a good LLM judge is an extremely strict, well-formatted prompt:
from langchain_community.llms import Ollama
import re
# Initialize a local open-source model via Ollama (e.g., Llama 3)
# Make sure Ollama is installed and running on your machine
judge_llm = Ollama(model="llama3")
def evaluate_with_llm_judge(query, context, output):
"""Uses a local LLM to score the output for faithfulness and relevance."""
judge_prompt = f"""
You are an impartial evaluator grading an AI assistant's response.
User Query: {query}
Provided Context: {context}
AI Output: {output}
Task:
Rate the AI Output on a scale of 1 to 5 based on whether it is factually correct according to the Provided Context.
- 5: Perfect, factually accurate based ONLY on context.
- 1: Completely incorrect or hallucinates information outside the context.
Provide your evaluation in this format exactly:
SCORE: [Your number]
REASONING: [Brief explanation]
"""
evaluation = judge_llm.invoke(judge_prompt)
# Extract the numerical score using regex
match = re.search(r"SCORE:\s*(\d)", evaluation)
score = int(match.group(1)) if match else 0
return score, evaluation
# Run the judge on our dataset
for data in eval_dataset:
score, reason = evaluate_with_llm_judge(data["query"], data["context"], data["model_output"])
print(f"\nQuery: {data['query']}")
print(f"Judge Score: {score}/5")Query: What is the return policy?
Judge Score: 5/5
Query: Do you offer free shipping?
Judge Score: 1/5
To make this a real pipeline, you combine these functions into one script. Whenever you change your system prompt, chunking strategy, or model settings, you run this script. If your average semantic similarity drops below 0.85 or your judge score averages below 4.5, your pipeline fails the test. This way, you’ll know right away if an update broke something before your users notice.
If you want to go deeper into LLMs and AI agents, I’d recommend my book, Hands-on GenAI, LLMs and AI Agents, where I cover how to build practical applications with them.
The Takeaway
Building custom evals for an LLM pipeline isn’t just a technical task. It marks a fundamental shift in how you think as an AI engineer.
When you first start learning about generative AI, you focus on making the model do something impressive. You want to see the magic. But as you move from learning to building real, production-ready systems, your focus needs to shift to reliability, consistency, and solid problem-solving.
I hope you enjoyed this article on setting up custom evals for an LLM pipeline. For more tips on AI and machine learning, you can follow me on Instagram.
