AI agents may perform well in one instance and fail in the next, such as by selecting the wrong tool, misinterpreting instructions, returning invalid output, or failing to recover from errors. Therefore, learning to evaluate AI agents in Python is essential for building reliable applications.
In this tutorial, we’ll develop a practical, free evaluation framework in Python. We will create a simple agent interface, define test cases, score agent behavior, save results to CSV, and compare agent versions without using paid APIs or platforms.
Evaluate an AI Agent with Python
What should you evaluate?
Begin by evaluating five key dimensions:
- Task completion: Did the agent achieve the requested outcome?
- Tool selection: Did it choose the appropriate tool?
- Output correctness: Is the result accurate and properly structured?
- Failure recovery: Can it handle errors and retry appropriately?
- Consistency: Does it produce reliable results across repeated runs?
Not every task requires an LLM judge. Whenever possible, use deterministic checks first: compare expected values, validate JSON schemas, inspect tool-call logs, and verify required actions occurred.
Create test cases with expected outcomes
Begin with a small evaluation dataset that reflects real tasks the agent must perform. Include straightforward requests, incorrect inputs, tool failures, and multi-step tasks.
For example, a research agent should search a knowledge base, while a calculation agent should use a calculator rather than guess.
Install the free dependencies:
pip install pandas pytest
Create eval_cases.py:
TEST_CASES = [
{
"id": "calc_01",
"task": "Calculate 25 * 4",
"expected_answer": "100",
"expected_tools": ["calculator"],
},
{
"id": "search_01",
"task": "Find the refund policy",
"expected_answer": "30 days",
"expected_tools": ["search"],
},
{
"id": "unknown_01",
"task": "Find an unavailable document",
"expected_answer": "not found",
"expected_tools": ["search"],
},
]Define success criteria before running the agent. Deciding what counts as success after seeing the output introduces bias into the evaluation.
Build a reusable evaluation framework
The evaluation framework should provide a consistent interface for calling the agent and capturing its behavior. Each run should return the final answer, tools used, and whether an error occurred before recovery.
For this example, assume the agent adapter returns these fields. The same interface can be connected to CrewAI, LangGraph, or a custom Python agent.
Create evaluate_agent.py:
import time
import pandas as pd
from eval_cases import TEST_CASES
def evaluate(agent, version, repetitions=3):
results = []
for case in TEST_CASES:
for run in range(repetitions):
start = time.perf_counter()
try:
output = agent.run(case["task"])
error = False
except Exception:
output = {
"answer": "",
"tools": [],
"recovered": False
}
error = True
latency = time.perf_counter() - start
answer = output.get("answer", "").lower()
tools = output.get("tools", [])
task_score = float(
case["expected_answer"].lower() in answer
)
tool_score = float(
tools == case["expected_tools"]
)
recovery_score = float(
output.get("recovered", False)
) if not error else 0.0
results.append({
"version": version,
"case_id": case["id"],
"run": run + 1,
"task_score": task_score,
"tool_score": tool_score,
"recovery_score": recovery_score,
"latency_seconds": latency,
"error": error,
"answer": answer
})
return pd.DataFrame(results)This code demonstrates the evaluation harness, not a full agent implementation. The agent must implement run() and return the specified fields. Test cases should include expected answers and tool traces that match the application.
Measure output correctness and failure recovery
The framework above uses exact, deterministic checks, which are well suited for numeric results, required fields, known tool calls, and predefined error scenarios.
For free-form answers, substring matching is insufficient. For example, an answer may explain the correct refund policy without using the exact expected phrase.
Use task-specific checks where possible:
- Numeric outputs: compare values with a tolerance.
- JSON responses: validate the schema and required fields.
- Research tasks: verify claims against trusted sources.
- Tool selection: compare actual tool calls with expected calls.
- Failure recovery: deliberately make a tool fail and check whether the agent retries or returns a safe error.
Open-source DeepEval provides additional metrics for task completion, tool correctness, and agent trajectories. Its documentation distinguishes end-to-end evaluation from evaluating individual components.
You can explore DeepEval later; however, some of its LLM-as-a-judge metrics require a configured judge model. For a fully local, no-paid-API workflow, use deterministic checks or a locally hosted judge model.
Preparing for AI/ML Interviews?
Building reliable AI agents is a valuable skill, but you should also be able to explain your evaluation methods, debugging decisions, and technical trade-offs in interviews. Cracking Your First AI/ML Interview is a resource I’d recommend as you prepare for your first AI/ML role.
Test consistency across repeated runs
An agent that succeeds once may still be unreliable. Run each test multiple times and calculate the success rate:
from evaluate_agent import evaluate
from my_agent import agent_v1, agent_v2
results_v1 = evaluate(agent_v1, "v1", repetitions=5)
results_v2 = evaluate(agent_v2, "v2", repetitions=5)
results = pd.concat([results_v1, results_v2])
results.to_csv("agent_evaluation.csv", index=False)
summary = results.groupby("version").agg(
task_completion=("task_score", "mean"),
tool_accuracy=("tool_score", "mean"),
recovery_rate=("recovery_score", "mean"),
average_latency=("latency_seconds", "mean"),
error_rate=("error", "mean")
)
print(summary.round(3))The code assumes agent_v1 and agent_v2 are implemented and Pandas is imported. Each adapter should provide the same run() interface.
A higher task-completion rate is valuable, but do not evaluate performance in isolation. An agent that completes tasks reliably but makes unnecessary tool calls or takes significantly longer may not be the better choice.
Compare agent versions with meaningful metrics
Track the following metrics for each version:
Be precise about the denominator. For recovery rate, count only runs where a recoverable failure was deliberately injected. Otherwise, the metric may be misleading.
Tool accuracy should also consider tool arguments and call order, not just tool names, when these details affect correctness. DeepEval’s tool correctness metrics support these detailed comparisons.
Automate evaluation with pytest
Once the evaluation logic is functional, integrate it into the development workflow:
# test_agent.py
import pytest
@pytest.mark.parametrize(
"task,expected",
[
("Calculate 25 * 4", "100"),
("Find the refund policy", "30 days"),
],
)
def test_agent_answers(task, expected):
result = agent_v1.run(task)
assert expected.lower() in result["answer"].lower()Run the tests using:
pytest -v
Pytest supports parameterized tests, making it straightforward to expand the evaluation dataset as new failure cases arise.
For nondeterministic agents, keep broader repeated-run evaluations separate from fast unit tests. Run these evaluations before releasing a new agent version or changing its tools and prompts.
The Takeaway
A paid evaluation platform is not required to test AI agents effectively. Python, Pandas, pytest, and a well-designed set of test cases provide a strong baseline.
Begin by defining success, capturing the agent’s tool calls and outputs, testing failure scenarios, and repeating each task. Compare versions using consistent metrics rather than selecting the agent with the most impressive single response.
A reliable AI agent does not only succeed in a demonstration. It consistently performs well across realistic tasks, unexpected failures, and repeated runs.
Thank you for reading this article on evaluating AI agents with Python. For more insights on AI and machine learning, please follow me on Instagram.
