Metrics-Based Alerting Configuration for Web Applications

Alerting without a well-thought-out methodology turns into noise: 200 notifications per night, half of which are 'resolved' in 2 minutes. The team stops responding—and that's when the real problem hits. On one project, we got 150 alerts in an hour due to an incorrectly set CPU threshold. After implementing the burn rate methodology, there were only 5. The goal of configuration is alerts only for situations requiring human action. Properly configured alerting is not just notification; it is a system that allows you to identify problems before they affect users. We have been doing this for over 5 years and have set up monitoring for 50+ web applications—from startups to enterprise. Below is an engineering approach without fluff.

What Metrics to Monitor First?

Start with the four golden signals described in Site Reliability Engineering: latency, traffic, errors, saturation. For a web application, latency (P95), errors (5xx), traffic (RPS), and saturation (CPU, memory) are sufficient. Additionally, queue length, SSL certificate, and business metrics (conversion, registrations). SLO (Service Level Objective) is the target availability level, e.g., 99.9% uptime. SLI (Service Level Indicator) is the actual metric we measure. Alerts with burn rate allow quick response when deviation from SLO threatens the budget.

Principles of Effective Alerting

Alert on symptoms, not causes. An alert like 'Site is unavailable to users' is more important than 'CPU > 80%'. High CPU is a cause that may not affect users. This principle is described in the Google SRE Book.

The rule of four golden signals: latency, traffic, errors, saturation. Start with the first three. Burn rate instead of thresholds: 'Error rate > 5% for 5 minutes' is better than '1 error per 1 minute'. Burn rate shows how fast you are consuming the SLO error budget and allows early detection of anomalies.

Why Burn Rate is Better Than Thresholds?

Threshold-based alerts produce many false positives. Example: an error on one out of a hundred requests is 1%, but if it lasts an hour, the error budget (SLO 99.9%) will be exhausted in 4 days. Burn rate = 10. An alert with this value will fire in 5 minutes, not in an hour. This reduces noise by 10 times.

How to Avoid Noisy Alerts?

Use burn rate, grouping, and deduplication in Alertmanager. Configure group_wait (30s), group_interval (5m), repeat_interval (4h). Alerts should be on symptoms, not causes. If CPU > 80% does not affect users, it is not an alert.

Stack Configuration: Prometheus + Alertmanager + Grafana

---
services:
  prometheus:
    image: prom/prometheus:v2.51.0
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - ./rules:/etc/prometheus/rules
      - prometheus_data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.retention.time=30d'
    ports:
      - "9090:9090"
  alertmanager:
    image: prom/alertmanager:v0.27.0
    volumes:
      - ./alertmanager.yml:/etc/alertmanager/alertmanager.yml
    ports:
      - "9093:9093"
  grafana:
    image: grafana/grafana:11.0.0
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin
    volumes:
      - grafana_data:/var/lib/grafana
    ports:
      - "3000:3000"
volumes:
  prometheus_data:
  grafana_data:

Application Metrics (Laravel) and Configuration

For Laravel, install the package spatie/laravel-prometheus and register custom metrics:

// app/Providers/AppServiceProvider.php
use Prometheus\CollectorRegistry;

public function boot(): void
{
    $registry = app(CollectorRegistry::class);

    // Counter — number of HTTP requests
    $httpRequests = $registry->getOrRegisterCounter(
        'app',
        'http_requests_total',
        'Total HTTP requests',
        ['method', 'route', 'status']
    );

    // Histogram — response time
    $httpDuration = $registry->getOrRegisterHistogram(
        'app',
        'http_request_duration_seconds',
        'HTTP request duration',
        ['method', 'route'],
        [0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0]
    );

    // Gauge — queue length
    $queueSize = $registry->getOrRegisterGauge(
        'app',
        'queue_size',
        'Current queue size',
        ['queue']
    );
}

Prometheus configuration:

# prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s
alerting:
  alertmanagers:
    - static_configs:
        - targets: ['alertmanager:9093']
rule_files:
  - 'rules/*.yml'
scrape_configs:
  - job_name: 'web-app'
    static_configs:
      - targets: ['app:9000']
    metrics_path: /metrics

Example Alerting Rules

# rules/web-app.yml
groups:
  - name: web-app
    rules:
      # High error rate (5xx)
      - alert: HighErrorRate
        expr: |
          sum(rate(app_http_requests_total{status=~"5.."}[5m])) / sum(rate(app_http_requests_total[5m])) > 0.05
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Error rate {{ $value | humanizePercentage }}"
          description: "5xx rate exceeded 5% for 2 minutes"
      # Slow responses (P95 > 2 seconds)
      - alert: HighLatencyP95
        expr: |
          histogram_quantile(0.95, sum by (le) (rate(app_http_request_duration_seconds_bucket[5m])) ) > 2
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "P95 latency {{ $value | humanizeDuration }}"
      # Traffic drop (anomalous fall)
      - alert: TrafficDrop
        expr: |
          sum(rate(app_http_requests_total[5m])) < 0.1 and sum(rate(app_http_requests_total[1h] offset 1h)) > 1
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Traffic almost zero — possible outage"
      # Large queue backlog
      - alert: QueueBacklog
        expr: app_queue_size{queue="default"} > 1000
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Queue backlog: {{ $value }} jobs"
      # SSL certificate expiring soon
      - alert: SSLCertExpiringSoon
        expr: probe_ssl_earliest_cert_expiry - time() < 14 * 24 * 3600
        for: 1h
        labels:
          severity: warning
        annotations:
          summary: "SSL cert expires in {{ $value | humanizeDuration }}"

Routing and Deduplication in Alertmanager

# alertmanager.yml
global:
  resolve_timeout: 5m
route:
  group_by: ['alertname', 'severity']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  receiver: 'telegram-critical'
  routes:
    - match:
        severity: critical
      receiver: 'telegram-critical'
      continue: false
    - match:
        severity: warning
      receiver: 'telegram-warning'
      group_interval: 15m
      repeat_interval: 12h
receivers:
  - name: 'telegram-critical'
    telegram_configs:
      - bot_token: 'your_bot_token'
        chat_id: -1001234567890
        message: |
          🔴 *{{ .CommonLabels.alertname }}*
          {{ range .Alerts }}
          {{ .Annotations.summary }}
          {{ if .Annotations.description }}{{ .Annotations.description }}{{ end }}
          {{ end }}
  - name: 'telegram-warning'
    telegram_configs:
      - bot_token: 'your_bot_token'
        chat_id: -1001234567890
        message: |
          ⚠️ *{{ .CommonLabels.alertname }}*
          {{ range .Alerts }}{{ .Annotations.summary }}{{ end }}
inhibit_rules:
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['alertname']

Comparison of Prometheus and Cloud Services

Prometheus + Alertmanager provides flexibility and control. Unlike CloudWatch or Stackdriver, you are not tied to a vendor and the cost is significantly lower at scale. On one project, we reduced monitoring costs by 3 times by moving from Datadog to a self-built stack. Such a configuration pays for itself in a few months by reducing infrastructure costs.

Work Process and Indicative Timelines

Basic setup (8–12 rules) takes 1–2 business days. If custom metrics and complex routing are needed, up to 4 days.

What’s Included

• Audit of current application metrics (APM, logs, infrastructure)
• Deployment of Prometheus + Alertmanager + Grafana
• Creation of 8–12 alert rules with burn rate
• Integration with Telegram, Slack, or email
• Basic Grafana dashboards (error rate, latency, RPS, queue)
• Documentation and team training
• Result guarantee and 1 month support

If you have a similar task, contact us for a consultation. We will assess the scope of work within 1 day. Order monitoring setup to get rid of noise and sleep soundly. Get a consultation on alerting configuration.