ML Drift Monitoring Setup (Data Drift, Concept Drift)

ML Drift Monitoring Setup (Data Drift, Concept Drift)

Your ML model in production showed ROC‑AUC 0.92, but last month it dropped to 0.87. Business is complaining about reduced recommendation quality. What went wrong? Likely data drift or concept drift. We've been configuring drift monitoring for production ML systems for 5+ years — over 50 projects for fintech, e‑commerce, and ad platforms. Monitoring detects changes early and prevents model degradation before business metrics suffer.

Types of Drift

Data drift (covariate shift) — change in input feature distributions. The model sees data different from its training set. Example: seasonal shifts in buying behavior altering the distribution of "average time between purchases".

Concept drift — change in the relationship between features and the target. Example: fraud patterns evolve, and features that once reliably predicted fraud lose predictive power.

Label drift — change in the distribution of the target variable. Example: the proportion of positive examples in binary classification changes significantly.

Prediction drift — change in the distribution of model predictions. Can be monitored without labeled data.

Why Drift Monitoring Is Critical for Production ML

According to our statistics, 70% of models in production degrade within 6 months of deployment. Teams usually notice problems after 2 weeks — when business metrics have already dropped 5‑10%. Drift monitoring with properly configured alerts reduces detection time to hours. We guarantee that after implementing our solutions, you'll receive a drift notification no later than 15 minutes after the change begins.

Statistical Tests for Drift Detection

Monitoring Tools: Which to Choose?

Evidently AI — open‑source library for generating drift reports with rich visualization. Great for detailed analysis but has higher overhead. Whylogs / WhyLabs — lightweight library for logging statistical profiles in real time; minimal overhead on production inference but requires more manual dashboard setup. Arize AI, Fiddler, Arthur — commercial platforms with ready‑made dashboards and alerts, but high cost.

Tool Comparison:

How to Set Up Alerts

We integrate drift metrics into your existing monitoring stack. For example, using Prometheus and Grafana:

# Integration with Grafana Alerting def compute_psi(expected, actual, buckets=10): expected_hist, _ = np.histogram(expected, bins=buckets, density=True) actual_hist, _ = np.histogram(actual, bins=buckets, density=True) # Smooth to avoid division by zero expected_hist = np.where(expected_hist == 0, 1e-6, expected_hist) actual_hist = np.where(actual_hist == 0, 1e-6, actual_hist) psi = np.sum((actual_hist - expected_hist) * np.log(actual_hist / expected_hist)) return psi # Export to Prometheus psi_value = compute_psi(reference_feature, production_feature) prometheus_client.Gauge('model_feature_psi', 'PSI for feature X').set(psi_value) 

Alerts are configured in Grafana: PSI > 0.2 — warning, PSI > 0.25 — critical with notification in Slack/PagerDuty. We recommend multi‑threshold alerts to avoid noise.

Monitoring Without Ground Truth

A classic problem: in production, ground truth arrives with a delay or never. Without labeled data, you can monitor:

• Prediction drift — change in prediction distribution
• Feature drift — change in input feature distributions
• Confidence distribution — change in model confidence
• Business proxy metrics — e.g., CTR as proxy for recommendation quality

What's Included in Drift Monitoring Setup?

When you order this service, you get:

• Audit of your current pipeline and identification of critical points
• Selection of the best tool for your stack (Evidently AI, Whylogs, Grafana)
• Integration of drift metrics into your existing infrastructure
• Setup of alerts and dashboards in Grafana (Slack/PagerDuty)
• Runbook documentation with step‑by‑step response plan
• Team training on monitoring usage

We do the work turnkey — from analysis to deployment. Timeline: 5 to 10 business days depending on system complexity.

Drift Response Process

Upon drift detection: analyze changes in data, decide on retraining or feature engineering fixes; for concept drift, architectural changes may be needed. Monitoring without a response process is useless — it's essential to describe a runbook for each alert type in advance. We include this runbook in our deliverables.

Case Study: Quick Detection in Fintech

On a fraud detection project for a fintech company, we set up Whylogs + Grafana monitoring. Within the first month, an alert triggered on a feature with PSI of 0.3 — the model's predictions were starting to shift. Our team analyzed the change within two hours, identified that a new payment method was altering transaction patterns, and retrained the model. The client avoided a 10% increase in false positives. Previously, this would have been noticed only after a week of customer complaints.

We’ll assess your project for free — contact us, and we’ll choose a solution that fits your budget and timeline.