OpsGenie integration for incident management

OpsGenie Integration for Incident Management

OpsGenie (Atlassian) — PagerDuty alternative with more flexible pricing and native Atlassian ecosystem integration (Jira, Confluence). For teams already using Jira Service Management, OpsGenie is included in Advanced/Premium plans at no extra cost.

Key OpsGenie Concepts

Teams — groups of engineers. Alerts route to teams, not individuals.

On-Call Schedules — duty schedules with rotation, override, temporary exceptions.

Escalation Policies — notification order on no response.

Routing Rules — conditional alert routing: by tags, source, time of day.

Alert Policies — alert transformation rules: noise suppression, auto-close, re-routing.

Connecting Alert Sources

Prometheus Alertmanager:

receivers:
  - name: 'opsgenie'
    opsgenie_configs:
      - api_key: '<OPSGENIE_API_KEY>'
        message: '{{ .CommonAnnotations.summary }}'
        description: '{{ .CommonAnnotations.description }}'
        priority: |
          {{- if eq .CommonLabels.severity "critical" -}}P1
          {{- else if eq .CommonLabels.severity "warning" -}}P3
          {{- else -}}P5{{- end -}}
        tags: '{{ .CommonLabels.alertname }},{{ .CommonLabels.cluster }}'

Grafana → OpsGenie: In Grafana alert channel select OpsGenie, enter API key. Notification contains panel screenshot.

Custom webhook (any source):

curl -X POST https://api.opsgenie.com/v2/alerts \
  -H "Authorization: GenieKey $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "message": "High error rate on payment service",
    "alias": "payment-high-error-rate",
    "priority": "P1",
    "tags": ["payment", "production"],
    "details": {"error_rate": "5.2%", "threshold": "1%"}
  }'

Smart Alert Routing

Routing rules allow directing alerts based on context:

IF alert.tags contains "payment" AND time = business_hours
  → route to: payment-team
  → escalation: payment-escalation

IF alert.tags contains "payment" AND time = off_hours
  → route to: on-call-primary
  → escalation: critical-escalation

IF alert.priority = "P5"
  → create ticket only, no notification

Heartbeat Monitoring

OpsGenie Heartbeats — monitoring regular processes (cron jobs, batch tasks). If cron doesn't send heartbeat at expected time — alert:

import requests

def send_heartbeat(heartbeat_name: str):
    """Call after each successful cron job execution"""
    requests.get(
        f"https://api.opsgenie.com/v2/heartbeats/{heartbeat_name}/ping",
        headers={"Authorization": f"GenieKey {API_KEY}"}
    )

Configuration: period 1 hour, grace period 10 minutes. If cron didn't ping 70 minutes — alert on-call.

Jira Service Management Integration

On JSM Advanced/Premium plan — OpsGenie built-in. Alert in OpsGenie → automatically Issue in JSM. On resolve OpsGenie → Issue closes. Two-way sync: comments and status update in both tools.

For standalone Jira (Software):

• OpsGenie → Jira integration: creates ticket on incident
• Severity field → Jira priority mapping
• Assignee → by on-call schedule

Maintenance Windows

Planned maintenance — alert suppression for period:

import requests, datetime

def create_maintenance(name: str, start: datetime, end: datetime, services: list):
    requests.post(
        "https://api.opsgenie.com/v1/maintenance",
        headers={"Authorization": f"GenieKey {API_KEY}"},
        json={
            "description": name,
            "time": {
                "type": "schedule",
                "startDate": start.isoformat(),
                "endDate": end.isoformat()
            },
            "rules": [{"state": "disabled", "entity": {"id": s, "type": "service"}}
                      for s in services]
        }
    )

OpsGenie Mobile App

Critical for on-call: notifications via push → call → SMS in configurable order. Ability to acknowledge and escalate directly from app without laptop.

Integration Timeframes

• Basic setup (teams + schedules + escalation) — 1 day
• Connecting monitoring (Prometheus, Grafana, CloudWatch) — 1 day
• Routing rules + alert policies — 1 day
• Heartbeats + maintenance windows — 0.5 day
• Jira integration — 0.5-1 day