Monitoring ML Model Performance in Production
Imagine your ML recommendation model drives 15% of revenue. Suddenly CTR drops by 40%, but you only notice it a day later via business reports. Cause: data drift after a new UI release. Without monitoring, engineers spend hours debugging, and revenue bleeds every minute. We've solved such cases dozens of times over 10+ years working with production ML. In another project, inference latency jumped from 50 ms to 1.2 s due to batch size increase — the alert fired in 2 minutes, and we rolled back the config.
Model monitoring is more than an AUC dashboard. It's a system that catches degradation early: latency spikes, embedding drift, batch timeouts. A comprehensive approach covers infrastructure, data, model, and business metrics. Contact us for an audit of your current monitoring — we'll suggest concrete improvements.
Why ML Model Monitoring Is Critical for Business
Production models are affected by hundreds of factors: input distribution shift, load spikes, feature pipeline bugs, stale embeddings. Without a metric system, you're blind. Example from our practice: a retailer's scoring model suddenly gave 90% positive predictions — an ETL bug, but it was only detected after 3 days when approval rate skyrocketed. With monitoring, such issues don't happen.
Monitoring Levels and Their Metrics
We define three metric layers. The table below shows example thresholds:
| Level | Example Metrics | Critical Thresholds |
|---|---|---|
| Infrastructure | latency (p50/p95/p99), throughput, GPU utilization | p99 > 2s, GPU util < 30% |
| Data and Model | prediction distribution, PSI, KS-test | PSI > 0.2, KS > 0.1 |
| Business | CTR, conversion, revenue impact | Deviation > 5% from baseline |
Level 1 — Infrastructure: latency (p50, p95, p99), throughput (RPS), error rate (5xx, timeouts), GPU utilization (more important than CPU during inference), queue depth when batching. These metrics signal problems first.
Level 2 — Data and Model: feature statistics (mean, std, null rate), prediction distribution (score histogram), confidence distribution, data drift (KS-test, PSI). Drift is the primary cause of silent degradation. According to Data drift, drift control is a key MLOps element.
Level 3 — Business Metrics: proxy metrics (CTR, conversion, engagement) before ground truth arrives, downstream KPIs (revenue impact, churn rate). They reflect the model's real value.
Setting Up Alerting for Production ML
Alerts should be multi-level. Here's a typical scheme:
| Level | Metric | Threshold | Channel |
|---|---|---|---|
| Warning | data drift PSI | > 0.15 | Slack |
| Warning | latency p99 | > 500 ms | Slack |
| Critical | error rate | > 1% | PagerDuty |
| Critical | latency p99 | > 2 s | PagerDuty |
| Fatal | service unavailable | - | phone on-call |
With such alerting, average detection time drops to 5–10 minutes vs. hours manually. In our projects, we guarantee that critical metrics reach the on-call engineer within 30 seconds.
Monitoring Stack
Prometheus + Grafana is the standard for infrastructure metrics. ML-specific metrics are exported via prometheus_client:
Example code for metric export
from prometheus_client import Histogram, Counter, Gauge REQUEST_LATENCY = Histogram( 'ml_inference_latency_seconds', 'Inference request latency', buckets=[0.01, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5] ) PREDICTION_DISTRIBUTION = Histogram( 'ml_prediction_score', 'Distribution of model prediction scores', buckets=[0.1 * i for i in range(11)] ) @REQUEST_LATENCY.time() def predict(features): score = model.predict_proba(features)[0][1] PREDICTION_DISTRIBUTION.observe(score) return score Evidently + Grafana — for drift monitoring with visualization. Evidently detects drift 10x faster compared to manual histogram analysis. Integration with Prometheus allows real-time dashboards.
OpenTelemetry — a standardized way to instrument tracing, metrics, and logs. Especially useful in microservice architectures where inference is one of many services.
Best Practices for Prediction Logging
To compute quality metrics later (when ground truth becomes available), log request-prediction pairs with a unique ID:
import uuid def predict_and_log(request_features): prediction_id = str(uuid.uuid4()) prediction = model.predict(request_features) # Log to ClickHouse/BigQuery/Kafka prediction_store.log({ 'prediction_id': prediction_id, 'timestamp': datetime.utcnow(), 'features': request_features.to_dict(), 'prediction': float(prediction), 'model_version': MODEL_VERSION }) return prediction, prediction_id When ground truth becomes known (e.g., user made a purchase or not), it is recorded with the same prediction_id, and the system computes actual quality metrics.
Dashboards
Recommended Grafana dashboard structure:
- Operational Overview — latency, throughput, error rate in real time
- Model Health — prediction distribution, feature statistics, drift metrics
- Business Impact — proxy metrics and downstream KPIs
- Model Comparison — compare current and previous version during canary deployment
Each dashboard has deployment annotations — this helps correlate metric changes with releases.
What's Included in Monitoring Setup
When you order production ML monitoring from us, we:
- Analyze current metrics, model, and infrastructure
- Design a metric system (infrastructure + ML + business)
- Implement metric collection (Prometheus client, Evidently, OpenTelemetry)
- Set up Grafana dashboards (4+ panels)
- Configure alerts with escalation (Slack/PagerDuty/phone)
- Train your team on the system
- Provide documentation and access
Timeline: 5 to 15 business days depending on pipeline complexity. Pricing is determined individually after an audit — contact us to evaluate your project.
Our engineers hold AWS ML certifications and have implemented monitoring for 40+ production models in retail and fintech. We guarantee SLA on alert reaction time — under 30 seconds to the on-call engineer. Order an audit of your monitoring system — we'll find weak spots in 2 days.