While setting up a complex RAG-based app and several AI agents on a Linux server for a client, I noticed my data preprocessing pipeline was slowing everything down. I kept having to write custom imputation code for each new data source. So, I changed my approach and built an AI agent team to automate data cleaning. In this article, I’ll show you how I designed this workflow.
Why a Team of Agents?
When people first try using LLMs for data tasks, they often make a big mistake. They give a raw CSV file to a single language model and ask it to “clean the data.”
This approach almost always fails. Large Language Models work by making predictions, not by following strict rules. If you let them change your datasets directly, they might make up values, remove important outliers, or quietly break your schema. You should never let an LLM change production data directly.
Instead, it’s better to use a multi-agent system that keeps rule-based code separate from tasks that need flexible reasoning. By creating a team of specialized agents, we can give each one a clear job:
- The Profiler Agent (Deterministic): Uses Python and Pandas to extract metadata, calculate missing value percentages, and identify data types.
- The Schema Validator (Deterministic): Uses SQL-like constraints to ensure columns match the expected format and domain logic.
- The Anomaly Detector (Deterministic): Uses statistical methods to flag extreme outliers.
- The LLM Strategist (Probabilistic): Reviews the reports from the first three agents and formulates a specific, rule-based cleaning strategy.
- The Executor (Deterministic): Applies the LLM’s recommended strategy using strict, safe Python operations.
With this setup, the LLM acts as the manager making decisions, while reliable Python code does the main work.
AI Agent Team to Automate Data Cleaning
Let’s see how this works in practice. We’ll use the well-known Titanic dataset from the seaborn library because it has real-world issues like missing ages, inconsistent categories, and outliers. For the LLM Strategist, we’ll use Ollama with a local model like Llama 3, so there’s no extra cost.
Before starting, check that Ollama is installed on your system. After that, download the Llama 3 model:
ollama pull llama3
Next, install these libraries:
pip install pandas seaborn numpy requests
First, set up your environment and load the data:
import pandas as pd
import seaborn as sns
import numpy as np
# Load the dataset
df = sns.load_dataset('titanic')
print(f"Original Data Shape: {df.shape}")Step 1: The Profiler and Schema Agents
Rather than asking an LLM to find missing values, write a rule-based Python agent to create a detailed report:
def profiler_agent(data):
"""Generates a statistical profile of the dataset."""
profile = {}
profile['missing_values'] = data.isnull().sum()[data.isnull().sum() > 0].to_dict()
profile['data_types'] = data.dtypes.astype(str).to_dict()
# Simple schema validation logic
schema_warnings = []
if data['age'].min() < 0 or data['age'].max() > 110:
schema_warnings.append("Age column out of realistic human bounds.")
profile['schema_warnings'] = schema_warnings
return profile
def anomaly_detector_agent(data, column):
"""Detects outliers using the Interquartile Range (IQR) method."""
if pd.api.types.is_numeric_dtype(data[column]):
Q1 = data[column].quantile(0.25)
Q3 = data[column].quantile(0.75)
IQR = Q3 - Q1
lower_bound = Q1 - 1.5 * IQR
upper_bound = Q3 + 1.5 * IQR
anomalies = data[(data[column] < lower_bound) | (data[column] > upper_bound)]
return len(anomalies)
return 0
# Generate the context for the LLM
data_profile = profiler_agent(df)
fare_anomalies = anomaly_detector_agent(df, 'fare')
data_profile['anomalies'] = {'fare_outliers_count': fare_anomalies}
print(data_profile)This approach saves token space and makes sure the results are accurate.
Step 2: The LLM Strategist
Next, feed this rule-based report into your local LLM:
import requests
import json
def llm_strategist_agent(profile_context):
"""Uses a local Ollama instance to decide on a cleaning strategy."""
prompt = f"""
You are an expert Data Engineer. Review the following dataset profile:
{json.dumps(profile_context, indent=2)}
Formulate a deterministic cleaning strategy.
Respond ONLY with a JSON object containing column names as keys, and the imputation/cleaning method as values.
Use methods like 'median', 'mode', 'drop_column', or 'cap_outliers'.
"""
# Call local Ollama API (assuming Llama 3 is running locally on port 11434)
# Using a try-except block to simulate the API response if Ollama is not active on the reader's machine
try:
response = requests.post('http://localhost:11434/api/generate',
json={'model': 'llama3', 'prompt': prompt, 'stream': False})
strategy = json.loads(response.json()['response'])
except:
# Fallback simulated strategy based on the profile
strategy = {
"age": "median",
"deck": "drop_column",
"embarked": "mode"
}
return strategy
cleaning_strategy = llm_strategist_agent(data_profile)
print("LLM Recommended Strategy:", cleaning_strategy)The prompt is set up so the LLM only outputs a structured list of actions, not changes to the data itself.
Step 3: The Executor Agent
Finally, the executor agent takes the JSON strategy and runs rule-based pandas operations:
def executor_agent(data, strategy):
"""Executes the LLM's strategy deterministically."""
df_cleaned = data.copy()
for column, action in strategy.items():
if column not in df_cleaned.columns:
continue
if action == 'median':
df_cleaned[column] = df_cleaned[column].fillna(df_cleaned[column].median())
elif action == 'mode':
mode_val = df_cleaned[column].mode()[0]
df_cleaned[column] = df_cleaned[column].fillna(mode_val)
elif action == 'drop_column':
df_cleaned = df_cleaned.drop(columns=[column])
return df_cleaned
final_df = executor_agent(df, cleaning_strategy)
print(f"Cleaned Data Shape: {final_df.shape}")
print(f"Remaining Missing Values:\n{final_df.isnull().sum()[final_df.isnull().sum() > 0]}")This keeps the pipeline strong, easy to track, and safe for production use.
If you want to go deeper into LLMs and AI agents, I’d recommend Hands-on GenAI, LLMs and AI Agents. It’s a practical resource for understanding these systems and building them through hands-on projects.
In real projects, you’d expand the executor agent to handle more complex SQL checks. This way, any changes keep referential integrity before sending the data back to the warehouse.
The Takeaway
A mistake I often see is learners trying to make LLMs do everything, from reading the CSV to changing the cells. This method doesn’t have the careful checks needed for enterprise-level work.
When you build an AI agent team, you’re thinking like an AI Engineer. You combine the reliability of traditional programming (Python, SQL, Pandas) with the flexible thinking of a language model.
I hope you found this article on building an AI agent team for data cleaning helpful. For more AI and machine learning tips, feel free to follow me on Instagram.
