ML Model Monitoring: Detect Data Drift & Model Drift

A model worked for a year, then started failing: a real case

A credit scoring ML model deployed to production achieved 94% accuracy for the first 6 months. After 8 months, the approval rate increased by 15% and delinquencies by 20%. The model metrics still looked fine, but the business was losing $55,000/month. The cause: data drift — the borrower profile shifted due to a new credit product from the bank. The model considered risk low, but it was actually high. We implemented monitoring using Evidently and Grafana, set up alerts on PSI > 0.2, and automated retraining. Within a month the situation was corrected, saving the company about $45k–65k (approx. $55,000).

ML model monitoring detects degradation before it becomes catastrophic. We've been doing this for over 5 years and have implemented more than 30 ML model monitoring projects. Timely drift detection can save companies about $9k–13k in savings per year.

How to distinguish input distribution changes from performance degradation?

Input distribution changes (data drift) — the feature distribution changed compared to the training set. Example: a fraud detection model trained on last year's transactions, but new attack patterns emerged. Input features have a different distribution.

Performance degradation (model drift) — prediction quality dropped. Accuracy, F1, AUC ROC decrease. Detection requires ground truth — real outcomes. Ground truth delay is the main challenge: for credit scoring, the answer "loan repaid" comes months later. Data drift can be detected immediately (no labels needed), model drift only when ground truth arrives.

What to choose: Evidently AI or Arize AI?

For most projects we choose Evidently + Grafana + Prometheus ML — flexible, cheap, self-hosted. For enterprise solutions with support, Arize. Evidently deploys 2x faster than competitors, and TCO is 3x lower.

Step-by-step drift monitoring setup with Evidently

1. Prepare a reference dataset — a snapshot of training data or first 2 weeks of production.
2. Configure a report — select drift metrics per feature.
3. Integrate with Prometheus ML — export metrics as gauges.
4. Create a Grafana dashboard — visualize drift per feature.
5. Set up alerts — threshold values and notifications to Slack/Telegram.

We guarantee a setup within 1-2 days for basic models.

from evidently import ColumnMapping
from evidently.report import Report
from evidently.metric_preset import DataDriftPreset
from evidently.metrics import ColumnDriftMetric

reference_data = pd.read_parquet("s3://ml-data/reference/reference_data.parquet")
current_data = fetch_recent_predictions(hours=24)

column_mapping = ColumnMapping(
    target='label',
    prediction='prediction',
    numerical_features=['age', 'income', 'credit_score'],
    categorical_features=['region', 'product_type'],
)

report = Report(metrics=[
    DataDriftPreset(),
    ColumnDriftMetric(column_name='income'),
    ColumnDriftMetric(column_name='credit_score'),
])

report.run(
    reference_data=reference_data,
    current_data=current_data,
    column_mapping=column_mapping,
)

Evidently uses statistical tests: PSI (Population Stability Index) — threshold > 0.2 signals drift; Wasserstein distance — for numerical features; Chi-square test — for categorical; KS test — for numerical, sensitive to local changes.

Evidently documentation (evidentlyai.com) and the Wikipedia article on Concept drift are useful resources for deeper study.

Which test to choose?

If data is small (< 1000 rows), use KS-test. If large, use PSI. For categorical features, Chi-square. Evidently has automatic selection by default, but we recommend explicitly setting it per feature.

Exporting metrics to Prometheus ML

from prometheus_client import Gauge, start_http_server
import schedule

drift_score = Gauge('model_feature_drift_score', 'PSI drift score per feature', ['feature', 'model'])
data_quality_score = Gauge('model_data_quality_score', 'Data quality score', ['model'])

def run_drift_check():
    report = generate_drift_report()
    results = report.as_dict()
    for metric in results['metrics']:
        if metric['metric'] == 'ColumnDriftMetric':
            feature = metric['result']['column_name']
            score = metric['result']['drift_score']
            drift_score.labels(feature=feature, model='credit-scoring-v2').set(score)

schedule.every(1).hours.do(run_drift_check)
start_http_server(8080)

Alerts on model degradation

---
- alert: FeatureDataDrift
  expr: model_feature_drift_score{model="credit-scoring-v2"} > 0.2
  for: 2h
  labels:
    severity: warning
    team: ml
  annotations:
    summary: "Data drift detected in feature {{ $labels.feature }}"
    description: "PSI = {{ $value }}, threshold 0.2. Consider retraining."
- alert: ModelAccuracyDegradation
  expr: model_accuracy_rolling_7d < 0.85
  for: 1h
  labels:
    severity: critical
---

Shadow mode: safe model comparison

Before deploying a new model, run it in shadow mode: production traffic is handled by the old model; the new model processes the same traffic in parallel, but results are only logged — not used. This approach is guaranteed to prevent business risk.

def predict(features: dict) -> float:
    result = production_model.predict(features)
    if shadow_mode_enabled():
        shadow_result = new_model.predict(features)
        log_shadow_prediction(
            features=features,
            production_pred=result,
            shadow_pred=shadow_result,
        )
    return result

Shadow mode accumulates statistics over 2–3 weeks to support a decision to switch without business risk. This allows comparing models on real data and avoiding mistakes that could cost millions.

Why is data drift dangerous for business?

Undetected data drift leads to financial losses. In one project, we discovered that credit scoring model drift caused approvals of risky borrowers, resulting in 15% portfolio losses (~$18k–26k, or $22,000). Implementing monitoring cut losses by 80% in the first 30 days.

What's included in model monitoring setup

Timelines and experience

Cost depends on complexity: we'll assess your project. Our certified MLOps engineers ensure your monitoring is reliable and guaranteed to detect drift within 24 hours. Contact us for a consultation. We are a team with over 5 years of experience in ML and DevOps, having implemented 30+ monitoring systems. Reach out — we'll help set up quality monitoring that truly saves your business from losses. Order implementation right now.