Metrics-Based Alerting Configuration for Web Applications

Alerting without a well-thought-out methodology turns into noise: 200 notifications per night, half of which are 'resolved' in 2 minutes. The team stops responding—and that's when the real problem hits. On one project, we got 150 alerts in an hour due to an incorrectly set CPU threshold. After implementing the burn rate methodology, there were only 5. The goal of configuration is alerts only for situations requiring human action. Properly configured alerting is not just notification; it is a system that allows you to identify problems before they affect users. We have been doing this for over 5 years and have set up monitoring for 50+ web applications—from startups to enterprise. Below is an engineering approach without fluff.

What Metrics to Monitor First?

Start with the four golden signals described in Site Reliability Engineering: latency, traffic, errors, saturation. For a web application, latency (P95), errors (5xx), traffic (RPS), and saturation (CPU, memory) are sufficient. Additionally, queue length, SSL certificate, and business metrics (conversion, registrations). SLO (Service Level Objective) is the target availability level, e.g., 99.9% uptime. SLI (Service Level Indicator) is the actual metric we measure. Alerts with burn rate allow quick response when deviation from SLO threatens the budget.

Principles of Effective Alerting

Alert on symptoms, not causes. An alert like 'Site is unavailable to users' is more important than 'CPU > 80%'. High CPU is a cause that may not affect users. This principle is described in the Google SRE Book.

The rule of four golden signals: latency, traffic, errors, saturation. Start with the first three. Burn rate instead of thresholds: 'Error rate > 5% for 5 minutes' is better than '1 error per 1 minute'. Burn rate shows how fast you are consuming the SLO error budget and allows early detection of anomalies.

Why Burn Rate is Better Than Thresholds?

Threshold-based alerts produce many false positives. Example: an error on one out of a hundred requests is 1%, but if it lasts an hour, the error budget (SLO 99.9%) will be exhausted in 4 days. Burn rate = 10. An alert with this value will fire in 5 minutes, not in an hour. This reduces noise by 10 times.

How to Avoid Noisy Alerts?

Use burn rate, grouping, and deduplication in Alertmanager. Configure group_wait (30s), group_interval (5m), repeat_interval (4h). Alerts should be on symptoms, not causes. If CPU > 80% does not affect users, it is not an alert.

Stack Configuration: Prometheus + Alertmanager + Grafana

services: prometheus: image: prom/prometheus:v2.51.0 volumes: - ./prometheus.yml:/etc/prometheus/prometheus.yml - ./rules:/etc/prometheus/rules - prometheus_data:/prometheus command: - '--config.file=/etc/prometheus/prometheus.yml' - '--storage.tsdb.retention.time=30d' ports: - "9090:9090" alertmanager: image: prom/alertmanager:v0.27.0 volumes: - ./alertmanager.yml:/etc/alertmanager/alertmanager.yml ports: - "9093:9093" grafana: image: grafana/grafana:11.0.0 environment: - GF_SECURITY_ADMIN_PASSWORD=admin volumes: - grafana_data:/var/lib/grafana ports: - "3000:3000" volumes: prometheus_data: grafana_data: 

Application Metrics (Laravel) and Configuration

For Laravel, install the package spatie/laravel-prometheus and register custom metrics:

// app/Providers/AppServiceProvider.php use Prometheus\CollectorRegistry; public function boot(): void { $registry = app(CollectorRegistry::class); // Counter — number of HTTP requests $httpRequests = $registry->getOrRegisterCounter( 'app', 'http_requests_total', 'Total HTTP requests', ['method', 'route', 'status'] ); // Histogram — response time $httpDuration = $registry->getOrRegisterHistogram( 'app', 'http_request_duration_seconds', 'HTTP request duration', ['method', 'route'], [0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0] ); // Gauge — queue length $queueSize = $registry->getOrRegisterGauge( 'app', 'queue_size', 'Current queue size', ['queue'] ); } 

Prometheus configuration:

# prometheus.yml global: scrape_interval: 15s evaluation_interval: 15s alerting: alertmanagers: - static_configs: - targets: ['alertmanager:9093'] rule_files: - 'rules/*.yml' scrape_configs: - job_name: 'web-app' static_configs: - targets: ['app:9000'] metrics_path: /metrics 

Example Alerting Rules

# rules/web-app.yml groups: - name: web-app rules: # High error rate (5xx) - alert: HighErrorRate expr: | sum(rate(app_http_requests_total{status=~"5.."}[5m])) / sum(rate(app_http_requests_total[5m])) > 0.05 for: 2m labels: severity: critical annotations: summary: "Error rate {{ $value | humanizePercentage }}" description: "5xx rate exceeded 5% for 2 minutes" # Slow responses (P95 > 2 seconds) - alert: HighLatencyP95 expr: | histogram_quantile(0.95, sum by (le) (rate(app_http_request_duration_seconds_bucket[5m])) ) > 2 for: 5m labels: severity: warning annotations: summary: "P95 latency {{ $value | humanizeDuration }}" # Traffic drop (anomalous fall) - alert: TrafficDrop expr: | sum(rate(app_http_requests_total[5m])) < 0.1 and sum(rate(app_http_requests_total[1h] offset 1h)) > 1 for: 5m labels: severity: critical annotations: summary: "Traffic almost zero — possible outage" # Large queue backlog - alert: QueueBacklog expr: app_queue_size{queue="default"} > 1000 for: 10m labels: severity: warning annotations: summary: "Queue backlog: {{ $value }} jobs" # SSL certificate expiring soon - alert: SSLCertExpiringSoon expr: probe_ssl_earliest_cert_expiry - time() < 14 * 24 * 3600 for: 1h labels: severity: warning annotations: summary: "SSL cert expires in {{ $value | humanizeDuration }}" 

Routing and Deduplication in Alertmanager

# alertmanager.yml global: resolve_timeout: 5m route: group_by: ['alertname', 'severity'] group_wait: 30s group_interval: 5m repeat_interval: 4h receiver: 'telegram-critical' routes: - match: severity: critical receiver: 'telegram-critical' continue: false - match: severity: warning receiver: 'telegram-warning' group_interval: 15m repeat_interval: 12h receivers: - name: 'telegram-critical' telegram_configs: - bot_token: 'your_bot_token' chat_id: -1001234567890 message: | 🔴 *{{ .CommonLabels.alertname }}* {{ range .Alerts }} {{ .Annotations.summary }} {{ if .Annotations.description }}{{ .Annotations.description }}{{ end }} {{ end }} - name: 'telegram-warning' telegram_configs: - bot_token: 'your_bot_token' chat_id: -1001234567890 message: | ⚠️ *{{ .CommonLabels.alertname }}* {{ range .Alerts }}{{ .Annotations.summary }}{{ end }} inhibit_rules: - source_match: severity: 'critical' target_match: severity: 'warning' equal: ['alertname'] 

Comparison of Prometheus and Cloud Services

Prometheus + Alertmanager provides flexibility and control. Unlike CloudWatch or Stackdriver, you are not tied to a vendor and the cost is significantly lower at scale. On one project, we reduced monitoring costs by 3 times by moving from Datadog to a self-built stack. Such a configuration pays for itself in a few months by reducing infrastructure costs.

Work Process and Indicative Timelines

Basic setup (8–12 rules) takes 1–2 business days. If custom metrics and complex routing are needed, up to 4 days.

What’s Included

• Audit of current application metrics (APM, logs, infrastructure)
• Deployment of Prometheus + Alertmanager + Grafana
• Creation of 8–12 alert rules with burn rate
• Integration with Telegram, Slack, or email
• Basic Grafana dashboards (error rate, latency, RPS, queue)
• Documentation and team training
• Result guarantee and 1 month support

If you have a similar task, contact us for a consultation. We will assess the scope of work within 1 day. Order monitoring setup to get rid of noise and sleep soundly. Get a free consultation on alerting configuration.