Configuring Grafana Alerting Rules for Automated Notifications

We encounter this constantly: when alerts are numerous, the on-call engineer stops reacting to them. Alert fatigue is not a metaphor. Real situation: 300 firing alerts in a Slack channel, 280 of which are chronic flapping of a single NodeMemoryPressure on a loaded node that "always does that." As a result, a critical alert about a database crash drowns in this noise. Our experience shows: proper alerting configuration saves up to 70% of on-call time and eliminates false night calls.

Unified Alerting vs Legacy

Since Grafana 9 and newer, the classic approach with Alertmanager rules in Prometheus and notifications via Grafana has been split—there is now Grafana Unified Alerting, which manages alerts from multiple data sources (Prometheus, Loki, InfluxDB) in a single interface. This is convenient but creates confusion: if the project already has Alertmanager configured with routes, inhibit rules, and silences, duplicating the logic in Grafana Alerting is unnecessary.

The rule is simple: if Prometheus Alertmanager is already the routing hub, use Grafana only for alert visualization, write rules in PrometheusRule CRD (for Kubernetes) or in prometheus.yml. If Grafana is the single entry point and there is no separate Alertmanager, then use Grafana Unified Alerting with contact points.

How to Design Rules Without Flapping

Flapping—an alert constantly switching from firing to resolved and back. It's treated in two ways.

First—the for parameter in the rule. for: 5m means the condition must be true continuously for 5 minutes before sending a notification. For CPU alerts, this is critical: a 10-second spike should not wake an engineer at night.

---
groups:
  - name: node.alerts
    rules:
      - alert: NodeCPUHighUsage
        expr: |
          ( 1 - avg by (instance) ( rate(node_cpu_seconds_total{mode="idle"}[5m]) ) ) > 0.85
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "CPU > 85% on {{ $labels.instance }} for 10 minutes"
          runbook_url: ""

Second—proper thresholds. 85% CPU—warning, 95%—critical. Do not use one alert at 80% with severity=critical. Gradation is needed so the on-call knows: warning can be reviewed in the morning, critical requires immediate call.

Why is Flapping Dangerous?

Flapping clogs notification channels, reduces on-call attention, and masks real incidents. With frequent status changes, the engineer stops trusting alerts, leading to missed critical events. The for parameter is the only way to cut noise without losing sensitivity.

How to Configure Inhibit Rules in Alertmanager?

In Alertmanager, configure inhibit rules: if NodeDown is firing, all alerts from that node (pod, service, container) are suppressed. Otherwise, one down node leads to an avalanche of 50 child alerts.

---
inhibit_rules:
  - source_match:
      alertname: NodeDown
    target_match_re:
      alertname: "Pod.*|Container.*|Service.*"
    equal:
      - instance

Routing: critical alerts go to PagerDuty with escalation policy, warning go to Slack channel #alerts-warning, info go to a separate low-priority channel. The on-call sees only what requires immediate action.

Contact Points in Grafana Unified Alerting

If using Grafana Alerting, configure contact points via provisioning, not the UI:

---
# /etc/grafana/provisioning/alerting/contact-points.yaml
apiVersion: 1
contactPoints:
  - orgId: 1
    name: pagerduty-critical
    receivers:
      - uid: pd-critical-uid
        type: pagerduty
        settings:
          integrationKey: "${PAGERDUTY_INTEGRATION_KEY}"
          severity: critical

Secrets—through environment variables, not hardcoded. Grafana reads them at startup.

Comparison of Grafana Unified Alerting and Prometheus Alertmanager

Evaluation Groups and Their Impact on Performance

All alert rules are grouped with evaluationInterval. Default is 1 minute. For critical infrastructure, 30 seconds can be used, but this increases load on Prometheus/Grafana. For 99% of rules, one minute is a normal interval. Faster is needed only for SLA-critical services with tight RTO.

How to Create an Alert Rule in Unified Alerting (Step-by-Step)

1. Go to Alerting → Alert rules
2. Click New alert rule
3. Select datasource (Prometheus, Loki)
4. Write expression, set for and severity
5. Configure annotations (summary, description, runbook_url)
6. Select contact point
7. Save

What Is Included in Alerting Configuration Work

• Audit of current alerts and data sources
• Rule design with grouping and thresholds
• Configuration of contact points (Slack, PagerDuty, Telegram)
• Creation of inhibit rules and routing
• Runbook documentation with automatic links
• Training for on-call staff on alert handling
• Post-release support for 2 weeks

Timelines

Auditing existing alerts and designing the correct structure takes more time than the configuration itself. A typical project: 2–4 days for inventory and rule design, 1–2 days for implementation and testing, 1 day for runbook documentation with links from alert annotations.

Our team specialists hold Kubernetes certifications and have infrastructure experience of 5+ years. We guarantee reduced alert noise and clear notification routing. Order turnkey alerting configuration—we will audit, design, and implement the notification system. Contact us to evaluate your project.