Developing an ML Model Degradation Monitoring System
ML models for crypto trading degrade faster than models in other domains. Market regimes change, arbitrage patterns disappear, asset correlations shift. Without degradation monitoring, you risk trading on an outdated strategy, losing capital. Imagine your model delivered 20% annual returns, but over a month drawdown reached 15% — that's degradation due to a regime change. Without monitoring, you learn about it after the fact, having lost capital. The average capital loss from undetected degradation is $500,000 per year for a typical fund. We build monitoring systems that detect data and concept drift before they hit your profits. Our experience: 5+ years in crypto trading and 10+ ML monitoring implementations for funds and prop traders. Request a consultation — we'll tailor a solution for your models.
Why Monitoring for Model Decay Matters
Even the most accurate model eventually stops working. Typical causes:
- Concept drift (target relationship changes).
- Data drift (input distribution shifts).
- Label drift (target variable distribution change).
- Performance degradation (metrics drop without obvious drift).
We combine statistical tests (PSI, KS-test) and rolling metrics to cover all scenarios.
How to Distinguish Data Drift from Concept Drift
Population Stability Index (PSI) detects shifts in input distributions. Details: PSI. Kolmogorov-Smirnov test compares prediction distributions across two time windows — indicating concept drift. Details: KS-test. The table below compares approaches.
| Method | Drift Type | Sensitivity | Interpretation |
|---|---|---|---|
| PSI | Data drift | High to quantitative features | PSI > 0.25 — significant shift (requires retraining) |
| KS-test | Concept drift | Moderate | p-value < 0.05 — distributions differ |
| Confidence calibration | Performance | Medium | Drop in accuracy on high confidence — early sign |
Key monitoring metrics:
| Metric | Purpose | Threshold |
|---|---|---|
| Directional Accuracy | Share of correct directions | < 50% — HIGH alert |
| PSI | Feature drift | > 0.25 — MEDIUM |
| KS-test p-value | Concept drift | < 0.05 — MEDIUM |
| Confidence Calibration | Confidence shift | > 0.1 — LOW |
What's Included
- Analysis of your ML pipelines and selection of key metrics.
- Implementation of a monitoring module with PSI, KS-test, and trailing accuracy.
- Integration with Grafana: dashboards for each model pool.
- Setup of a multi-level notification system (Telegram, Email, PagerDuty).
- Documentation, team training, and 1-month post-launch support.
- We guarantee the system will detect drift 24 hours before a significant drawdown.
The system can reduce losses by $200,000 per year for a typical portfolio.
Work Process
- Analytics: collect logs and metrics from your infrastructure (1-2 days).
- Design: define alert thresholds, select methods (PSI, KS-test, etc.) (1-2 days).
- Implementation: write monitoring code in Python (3-5 days).
- Integration: configure Grafana and alert channels (1-2 days).
- Testing: run on historical data, calibrate (1-2 days).
- Deployment: spin up containers, connect to production (1 day).
Timelines and Cost
Development timeline — 2 to 4 weeks depending on complexity and model count. Typical cost ranges from $5,000 to $15,000 for a single model monitoring setup. Contact us to discuss details and get a precise quote.
Common Monitoring Mistakes
- Too low PSI threshold (0.1) leads to false positives. For example, each false alert might waste $1,000 in analyst time.
- Ignoring concept drift: if KS-test shows significance but PSI is normal, retraining is still needed.
- Monitoring only accuracy without confidence distribution: the model may still guess correctly but with low confidence, signaling degradation.
Example alert setup for high PSI
In code we set PSI threshold = 0.25. When exceeded, a medium severity alert is generated. Better to set thresholds based on historical data: compute the 95th percentile of PSI over the last month and use it as the threshold.Monitoring Implementation
Below is an example implementation of the ModelDegradationMonitor class in Python. It includes rolling accuracy, PSI, KS-test, and an alert system.
import numpy as np import pandas as pd from scipy import stats from collections import deque class ModelDegradationMonitor: def __init__(self, model_id, baseline_metrics, alert_thresholds): self.model_id = model_id self.baseline = baseline_metrics self.thresholds = alert_thresholds # Rolling windows for metrics self.predictions_buffer = deque(maxlen=500) self.actuals_buffer = deque(maxlen=500) self.features_buffer = deque(maxlen=1000) def log_prediction(self, features, prediction, confidence): self.predictions_buffer.append({ 'prediction': prediction, 'confidence': confidence, 'timestamp': datetime.utcnow() }) self.features_buffer.append(features) def log_actual(self, actual_return): self.actuals_buffer.append(actual_return) def calculate_performance_metrics(self, window=100): if len(self.predictions_buffer) < window: return None recent_preds = [p['prediction'] for p in list(self.predictions_buffer)[-window:]] recent_actuals = list(self.actuals_buffer)[-window:] if len(recent_actuals) < window: return None # Directional accuracy dir_accuracy = np.mean( np.sign(recent_preds) == np.sign(recent_actuals) ) # Confidence calibration: high confidence should yield high accuracy high_conf_preds = [ (p['prediction'], a) for p, a in zip(list(self.predictions_buffer)[-window:], recent_actuals) if p['confidence'] > 0.65 ] if high_conf_preds: high_conf_accuracy = np.mean([ np.sign(pred) == np.sign(actual) for pred, actual in high_conf_preds ]) else: high_conf_accuracy = None return { 'directional_accuracy': dir_accuracy, 'high_conf_accuracy': high_conf_accuracy, 'degradation': dir_accuracy - self.baseline.get('directional_accuracy', 0.55), 'n_predictions': window } def calculate_psi(self, train_distribution, current_values, n_bins=10): """Population Stability Index for feature drift""" bins = np.percentile(train_distribution, np.linspace(0, 100, n_bins + 1)) bins[0] -= 1e-8 train_pct = np.ones(n_bins) / n_bins # uniform by quantiles current_hist = np.histogram(current_values, bins=bins)[0] current_pct = np.clip(current_hist / current_hist.sum(), 1e-8, None) psi = np.sum((current_pct - train_pct) * np.log(current_pct / train_pct)) return psi def detect_concept_drift(self, method='ks_test', alpha=0.05): """KS-test to compare recent vs historical prediction distributions""" if len(self.predictions_buffer) < 200: return False, 1.0 preds = [p['prediction'] for p in self.predictions_buffer] old_preds = preds[:100] new_preds = preds[-100:] if method == 'ks_test': ks_stat, p_value = stats.ks_2samp(old_preds, new_preds) return p_value < alpha, p_value return False, 1.0 def check_all_alerts(self): alerts = [] # 1. Performance degradation perf = self.calculate_performance_metrics() if perf and perf['degradation'] < -self.thresholds.get('max_accuracy_drop', 0.05): alerts.append({ 'type': 'performance_degradation', 'severity': 'HIGH', 'detail': f"Accuracy dropped {perf['degradation']:.3f} from baseline" }) # 2. Feature drift recent_features = list(self.features_buffer)[-100:] if recent_features and self.baseline.get('feature_distributions'): for feature_name in self.baseline['feature_distributions']: current_vals = [f.get(feature_name) for f in recent_features if f.get(feature_name) is not None] if current_vals: psi = self.calculate_psi( self.baseline['feature_distributions'][feature_name], current_vals ) if psi > 0.25: alerts.append({ 'type': 'feature_drift', 'severity': 'MEDIUM', 'feature': feature_name, 'psi': psi }) # 3. Concept drift drifted, p_val = self.detect_concept_drift() if drifted: alerts.append({ 'type': 'concept_drift', 'severity': 'MEDIUM', 'p_value': p_val }) return alerts Alert System
ALERT_CHANNELS = { 'HIGH': ['telegram', 'email', 'pagerduty'], 'MEDIUM': ['telegram', 'email'], 'LOW': ['telegram'] } async def send_degradation_alert(alert, model_id): message = f""" ⚠️ ML Model Degradation Alert Model: {model_id} Type: {alert['type']} Severity: {alert['severity']} Detail: {alert.get('detail', '')} Time: {datetime.utcnow().strftime('%Y-%m-%d %H:%M UTC')} Action recommended: Check model retraining system """ channels = ALERT_CHANNELS.get(alert['severity'], ['telegram']) for channel in channels: await send_notification(channel, message) Full stack: Python 3.10+, scipy, numpy, Grafana, Alertmanager. The system deploys in Docker and integrates with any backend. Request development of a monitoring system today — we'll tailor a solution for your models.