From my experience with machine learning projects, I’ve realized that getting a good validation score is just the beginning. After deployment, the data a model sees can change. Prediction patterns might shift, error rates can go up, and the model may become less reliable over time. That’s why we can build a simple ML model monitoring dashboard in Python to keep track of what happens after deployment.
For this task, I’m using the Wine dataset that comes with scikit-learn. It’s a multiclass classification dataset with 178 samples and 13 numeric features.
The goal is to monitor:
- Model accuracy
- Error rate
- Prediction distribution
- Feature drift
- Prediction confidence
- Prediction latency
- Alerts when metrics move outside expected ranges
ML Model Monitoring Dashboard With Python: Getting Started
I want the workflow to resemble a simple production monitoring system:
Training data → ML model → Predictions → Monitoring metrics → Streamlit dashboard → Alerts
First, I’ll train a classification model and set a baseline for how it behaves. Then I’ll simulate a new production batch where the feature distribution is different.
This is important because I don’t want the dashboard to show only static metrics. I want to show what happens when the new data starts to look different from the data the model was trained on.
Before getting started, make sure to install the necessary libraries:
pip install pandas numpy scikit-learn scipy streamlit
I’ll use scikit-learn for the model and dataset, Pandas and NumPy for data processing, SciPy to test for data drift, and Streamlit to build the dashboard.
Step 1: Train the Model
I start by loading the Wine dataset and splitting it into training and test sets:
import pandas as pd
import numpy as np
from sklearn.datasets import load_wine
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
wine = load_wine()
X = pd.DataFrame(
wine.data,
columns=wine.feature_names
)
y = pd.Series(wine.target, name="target")
X_train, X_test, y_train, y_test = train_test_split(
X,
y,
test_size=0.3,
random_state=42,
stratify=y
)
model = RandomForestClassifier(
n_estimators=200,
random_state=42
)
model.fit(X_train, y_train)The model gives me a baseline for comparison. I don’t need a complex model for this. The goal isn’t to optimize the classifier, but to understand how to monitor it after deployment.
Step 2: Create a Production-Like Batch
In a real application, new predictions would come in all the time. For this task, I’ll simulate that using the test data.
I’ll also make a small change to the distribution of some features so the monitoring system has something to pick up:
production_data = X_test.copy()
production_labels = y_test.copy()
production_data["alcohol"] *= 1.15
production_data["color_intensity"] *= 1.20
production_data["proline"] *= 1.10This isn’t intended to represent a realistic business transformation. It’s simply a controlled way to demonstrate data drift.
That distinction matters when building a portfolio project: we want to be clear about what is simulated and what reflects actual production behaviour.
Building Projects for AI/ML Interviews?
If you’re building projects like this for your portfolio, remember to prepare for interviews as well. I recommend Cracking Your First AI/ML Interview if you’re getting ready for your first AI/ML interview.
Step 3: Track Model Performance
First, I want to know whether the model is still making correct predictions:
from sklearn.metrics import accuracy_score
predictions = model.predict(production_data)
accuracy = accuracy_score(
production_labels,
predictions
)
error_rate = 1 - accuracy
print("Accuracy:", accuracy)
print("Error Rate:", error_rate)Accuracy and error rate are helpful when we have ground-truth labels. But in real production, labels might not show up until hours or days later. So we also need to track metrics we can measure right away.
Step 4: Monitor Prediction Distributions
A model can start acting differently even before I have enough labels to check its accuracy in production.
So I also want to see how the prediction distribution changes:
prediction_distribution = (
pd.Series(predictions)
.value_counts(normalize=True)
.sort_index()
)
print(prediction_distribution)If the model previously predicted class 0 for 30% of requests but suddenly starts predicting it for 70%, we will need to investigate.
A change like this doesn’t always mean the model is broken. The real-world data might have changed. Still, it’s a useful signal.
Step 5: Detect Feature Drift
For feature drift, I’ll compare the training distribution with the production distribution using the Kolmogorov-Smirnov test:
from scipy.stats import ks_2samp
drift_results = []
for column in X_train.columns:
statistic, p_value = ks_2samp(
X_train[column],
production_data[column]
)
drift_results.append({
"feature": column,
"ks_statistic": statistic,
"p_value": p_value,
"drift": p_value < 0.05
})
drift_df = pd.DataFrame(drift_results)
print(drift_df)Here, I’m using p_value < 0.05 as a simple threshold for demonstration.
Don’t treat that threshold as a rule for every production system. In real projects, you should consider things like sample size, business impact, feature importance, historical changes, and the cost of false alerts.
Step 6: Monitor Prediction Confidence
For a classification model, I also want to understand how confident the model is:
probabilities = model.predict_proba(
production_data
)
confidence = probabilities.max(axis=1)
average_confidence = confidence.mean()
low_confidence_rate = (
confidence < 0.60
).mean()This gives us two useful signals:
- Average confidence tells how confident the model is overall.
- The low-confidence rate tells how often the model is uncertain.
If both of these start to move in the wrong direction, it’s another sign we should check the incoming data.
Step 7: Track Prediction Latency
Model monitoring isn’t just about accuracy. In production, prediction speed is important too.
We can measure prediction latency with Python:
import time
start = time.perf_counter()
predictions = model.predict(
production_data
)
latency = time.perf_counter() - start
average_latency = latency / len(production_data)
print("Average prediction latency:", average_latency)For real applications, we should track percentiles like p50, p95, and p99 instead of just looking at the average.
Step 8: Build the Monitoring Dashboard
Now we can bring everything together with Streamlit:
import streamlit as st
st.title("ML Model Monitoring Dashboard")
st.metric(
"Model Accuracy",
f"{accuracy:.2%}"
)
st.metric(
"Error Rate",
f"{error_rate:.2%}"
)
st.metric(
"Average Confidence",
f"{average_confidence:.2%}"
)
st.metric(
"Low Confidence Rate",
f"{low_confidence_rate:.2%}"
)
st.subheader("Prediction Distribution")
st.bar_chart(
prediction_distribution
)
st.subheader("Feature Drift")
st.dataframe(
drift_df
)
st.subheader("Drifted Features")
st.write(
drift_df[drift_df["drift"]]
)Save everything in app.py and run:
streamlit run app.py
Now we have a simple dashboard that shows the model’s current behaviour, so we don’t have to check each prediction by hand.
Step 9: Add Alerts
The last thing I’d add is an alerting layer. For example, I can flag the model if accuracy drops below a certain point, too many predictions have low confidence, or several key features show drift:
alerts = []
if accuracy < 0.85:
alerts.append(
"Accuracy has dropped below the expected threshold."
)
if low_confidence_rate > 0.20:
alerts.append(
"High percentage of low-confidence predictions."
)
if drift_df["drift"].sum() >= 3:
alerts.append(
"Multiple features show signs of distribution drift."
)
for alert in alerts:
st.warning(alert)In a real system, you could connect these alerts to Slack, email, or an incident management tool.
The Takeaway
Building the model is just one part of machine learning engineering. Once a model starts getting real-world data, we need to check if the data still looks familiar, if predictions make sense, if errors are going up, and if the model is staying within expected limits.
That’s why I suggest building this as an ML portfolio project. It shows more than just model training; it shows you understand what happens after deployment.
For me, that’s the biggest lesson from building an ML monitoring dashboard with Python: a model isn’t truly production-ready just because it’s accurate today. I also need a system that lets me know when tomorrow’s predictions start to change.
I hope you liked this article on building an ML model monitoring dashboard in Python to track what happens after deployment. For more tips on AI and machine learning, feel free to follow me on Instagram.
