Your team is drowning in a sea of alerts: 200+ notifications per hour, half are duplicates, the rest are warnings requiring no action. Duty engineers burn out, real incidents get lost, and MTTR grows. We configure PagerDuty so the problem is solved in 2–4 days. Incident management stops being a headache.
One of our projects: Prometheus generated 200+ alerts per hour. After configuring PagerDuty with Event Intelligence, the number of incidents dropped to 5–7 per day. MTTR fell from 45 to 8 minutes — 5 times faster. The average cost of a minute of downtime for e-commerce is high, so the savings were substantial. Over 5+ years, we have completed more than 50 such integrations for teams of various sizes. We guarantee stable system operation after deployment.
PagerDuty Architecture: Key Components
Services — logical units (backend API, payment service, database). Each service has its own escalation policy and on-call schedule.
Integrations — event sources: Prometheus/Alertmanager, Datadog, CloudWatch, Grafana, Uptime Robot, custom webhooks. Each integration generates a unique endpoint key.
Escalation Policies — rules: who gets the alert, after how many minutes escalation, where to escalate.
Schedules — on-call schedules with rotations.
How to Connect Prometheus Alertmanager? (Detailed Case Study from Our Practice)
This is the most common scenario. We proceed step by step:
- Create a service in PagerDuty and add a Prometheus integration. Get the routing_key.
- In the Alertmanager config, set up a receiver with this key. Be sure to specify description and severity.
- Configure grouping by alertname and cluster so related alerts don't spawn multiple incidents.
- Verify that a test alert arrives in the correct service and escalates according to the policy.
Example receiver configuration:
# alertmanager.yml route: group_by: ['alertname', 'cluster'] group_wait: 30s group_interval: 5m repeat_interval: 4h receiver: 'pagerduty-critical' routes: - match: severity: critical receiver: 'pagerduty-critical' - match: severity: warning receiver: 'slack-warnings' receivers: - name: 'pagerduty-critical' pagerduty_configs: - routing_key: '<PAGERDUTY_INTEGRATION_KEY>' description: '{{ range .Alerts }}{{ .Annotations.summary }}{{ end }}' severity: '{{ .CommonLabels.severity }}' details: firing: '{{ template "pagerduty.default.instances" .Alerts.Firing }}' In one project, we configured grouping by cluster and alertname. This reduced the number of created incidents by 70%.
Comparison of Integration Methods
| Source | Integration Type | Complexity | Notes |
|---|---|---|---|
| Prometheus Alertmanager | Webhook (HTTP) | Low | Supports grouping and templates |
| Datadog | Native API | Medium | Requires enabling in Datadog, convenient via tags |
| CloudWatch | SNS → Lambda → Webhook | High | No direct integration, needs middleware |
| Grafana | Webhook | Low | Supports custom payload |
| Custom webhook | HTTP | Low | Maximum flexibility |
How PagerDuty Helps Tame Alert Noise?
PagerDuty Event Intelligence (available in paid plans) automatically suppresses noise. It includes three mechanisms:
- Alert Grouping: related alerts are merged into one incident. When a DB fails, you won't get 50 alerts from all services — just one.
- Intelligent Alert Grouping: ML model based on historical patterns groups similar alerts.
- Suppression Rules: temporary suppression of alerts during planned maintenance.
According to PagerDuty documentation, noise reduction cuts notifications by up to 90%. In one project, we achieved an 85% reduction.
Advantages of PagerDuty over Email Notifications
Email cannot group alerts, has no escalations, and does not provide MTTR statistics. PagerDuty delivers critical notifications 5 times faster: push notifications arrive in seconds, while email can be delayed by minutes. Moreover, PagerDuty automatically keeps an incident timeline, which aids postmortems.
Table: Key Metrics Before and After PagerDuty Deployment
| Metric | Before | After |
|---|---|---|
| Number of alerts per day | 500+ | 10–15 |
| MTTR | 45 min | 8 min |
| False positive rate | 80% | 5% |
| Annual savings | — | up to 2 million RUB |
| Team satisfaction | low | high |
Automation with Webhooks and Runbooks
PagerDuty Webhooks send events when an incident is created, updated, or resolved. Example handler in Python:
@app.route('/pd-webhook', methods=['POST']) def pagerduty_webhook(): data = request.json event_type = data['event']['event_type'] incident = data['event']['data'] if event_type == 'incident.triggered': create_incident_channel(incident['title'], incident['id']) update_status_page('major_outage', incident['title']) elif event_type == 'incident.resolved': archive_incident_channel(incident['id']) update_status_page('operational', '') return '', 200 Runbook Automation (formerly Rundeck) allows automatic actions on alert: restart service, clean disk, scale. If a script fixes the problem, the incident closes automatically without waking the on-call engineer.
Additional details on Datadog integration
Datadog has a direct integration with PagerDuty via API. Setup takes about an hour: add PagerDuty as an integration in Datadog, attach to the desired monitors, and configure tags. After that, alerts from Datadog will automatically create incidents in PagerDuty.What Does Our Work Include?
- Audit of current monitoring and incident management processes.
- Designing service structure, escalation policies, and schedules.
- Configuring integrations with Prometheus, Datadog, CloudWatch, Grafana, and others.
- Setting up webhooks and automation with Jira/Slack.
- Testing and team training.
- Handover of documentation and access rights.
- Stable operation guarantee and post-deployment support.
- Our engineers have PagerDuty certifications and experience with over 50 integrations.
Get a consultation to optimize your incident management. Contact us for an audit of your system — we will evaluate your infrastructure and suggest an optimal configuration in 1–2 days. Order a turnkey PagerDuty integration and get a 3–5x MTTR reduction.