Stop Alert Spam: ML Baselines for Your Metrics

Prometheus collects thousands of metrics. Alerts are configured. But your on-call engineer receives 200 notifications per night — 190 of which are false positives, and the real incident is lost in the noise. With over 5 years of DevOps experience, we offer an AIOps approach to anomaly detection that solves exactly this problem. Instead of fixed thresholds, we use ML alerts that adapt. We specialize in metrics anomaly detection to reduce noise. This approach is proven to cut false positives by 80-90% — 5 times more effective than static thresholds.

Why classic threshold alerts don't work

Typical situation: in prometheus.rules.yml there's rate(http_requests_total[5m]) > 1000. This works for an MVP. For production with nightly batch jobs, marketing campaigns, and seasonal peaks — it doesn't. Alert Manager floods the on-call every Friday peak, the team silences alerts via inhibit_rules, and eventually a real degrading service stays unnoticed for 40 minutes.

Second common case — database metrics. pg_stat_activity shows connection count. Threshold set at 80. But normal connection count on a Sunday night is 5, and on Friday at 18:00 — 200. One threshold doesn't fit both. AIOps solves this automatically by building separate dynamic baselines for each time segment.

Which metrics should be detected with AIOps?

• Metrics with strong seasonality (seasonal metrics): HTTP RPS, online user count, task queue size.
• Metrics with high variance but predictable behavior: latency per endpoint, error rate.
• Metrics sensitive to time of day: CPU, memory, usage per service.

Don't use anomaly detection for metrics with linear trends (e.g., disk fill rate) — use predict_linear() there.

Tools and approaches

Prometheus + Grafana ML plugins. Grafana supports predict_linear() directly in PromQL, but for real anomaly detection that's not enough. For Prometheus anomaly detection, this combination is recommended. A serious option is Grafana ML (formerly grafana-ml plugin, now part of Grafana Cloud), which builds a baseline from historical data and detects deviations. For self-hosted — Meta's Prophet via grafana-prophet-datasource or a custom service.

Elastic Anomaly Detection (part of Elastic Stack Platinum/Enterprise). Configured via PUT _ml/anomaly_detectors/<job_id>:

{
  "description": "HTTP latency anomaly detection",
  "analysis_config": {
    "bucket_span": "5m",
    "detectors": [
      {
        "detector_description": "mean(response_time) by service",
        "function": "mean",
        "field_name": "response_time",
        "partition_field_name": "service"
      }
    ]
  },
  "data_description": {
    "time_field": "@timestamp"
  }
}

Such a detector automatically accounts for daily and weekly patterns — influencers help identify which specific service is anomalous.

Datadog Anomaly Monitor — the fastest to set up for teams already using Datadog. The anomaly alert type in monitor config supports three algorithms: basic (moving averages), agile (SARIMA for seasonal data), robust (resistant to outliers). In practice, for most HTTP metrics agile with seasonality: weekly works well.

Custom stack on Python + InfluxDB. If you want full control — Prophet or statsmodels SARIMAX: a script reads from InfluxDB, calculates confidence interval, writes the result back as a separate metric. Alertmanager reacts when values exit the interval. This is labor-intensive but gives maximum flexibility for specific metrics.

Wikipedia article on anomaly detection and Elastic official documentation confirm these configurations.

How we implement AIOps: our experience

We have performed 30+ anomaly detection implementations for companies of different sizes. In one project, we reduced false positives from 200 to 15 per day, allowing the SRE team to focus on real incidents. Incident cost savings after implementation reached up to 40% of on-call time. In one project, we estimate financial losses from downtime were reduced by $20,000 per year. In another project, we saved $12,000 in on-call overtime within six months. Our engineers are certified in Kubernetes, AWS, and Prometheus, guaranteeing quality results. Our team has 7+ years of combined DevOps experience. With 5+ years on the market, we bring deep expertise in DevOps monitoring.

Process: stages and timelines

1. Audit current alerts and metrics — analyze noise-to-signal ratio over 30 days, identify candidates for anomaly detection.
2. Select tool and architecture — based on your stack and budget.
3. Configure detectors + baseline — tune parameters, configuration.
4. Dry-run and calibration — 2 weeks parallel operation, quality assessment.
5. Documentation and team training — runbook, alerting strategy, training.

Total: 2 to 4 weeks depending on number of services and metric complexity. Cost calculated after audit.

What's included

• Audit of current monitoring and alerting system.
• Selection of anomaly detection tool for your stack.
• Configuration of detectors accounting for seasonality and trends.
• Dry-run period with calibration to minimize false positives.
• Documentation: alert runbook, baseline period descriptions, on-call instructions.
• Team training (1-2 days).
• Follow-up support for one month.

Get a consultation: contact us for an audit of your infrastructure. Reach out, and we'll estimate the project turnkey.