Configure Grafana OnCall for Incident Escalation

Imagine production goes down at 3 AM. Alerts flood a general Slack channel with 15 people. Everyone has Do Not Disturb enabled, some are asleep, others on vacation. An hour later it turns out the on-call person was yesterday's shift. This is common if on-call scheduling isn't automated. Grafana OnCall is an on-call management system that handles schedule rotations, incident escalation, and ensures every critical alert reaches the responsible person. Our team at our team has deployed OnCall in 50+ projects and knows how to configure it for any infrastructure — from a single service to a thousand. With OnCall you stop losing alerts and forgetting who's on duty. The system automatically assigns the responsible person, forcefully escalates if no response, and rotates schedules. Contact us — we'll tailor a configuration to your scale and set it up turnkey.

What is Grafana OnCall?

Grafana OnCall is an open-source on-call management system integrated into the Grafana ecosystem. It's available as a managed service in Grafana Cloud or for self-hosted deployment via Docker or Kubernetes. The source code is on GitHub. Grafana OnCall GitHub repository

Key entities: Integration, Escalation Chain, On-Call Schedule, and Notification Policy. Each entity is configured through the web interface or API.

How Grafana OnCall accelerates incident response

A properly configured escalation chain reduces response time from 15–20 minutes to 2–3 minutes — that's 5x faster than manual processes. How does it work? An alert reaches the on-call person; if not acknowledged within 5 minutes, escalation moves to the next in chain. After another 10 minutes, the entire team gets notified in Slack; after 15, the CTO receives a phone call. Each step is independently configurable.

Compare notification channels by speed and reliability:

Voice calls are the most reliable channel: they reach 99% of cases, even when the phone is silent.

Deployment and configuration

Self-hosted deployment

For self-hosted we use Docker Compose. Example configuration:

---
services:
  oncall:
    image: grafana/oncall:latest
    ports:
      - "8080:8080"
    environment:
      DATABASE_TYPE: postgresql
      DATABASE_HOST: postgres
      DATABASE_NAME: oncall
      DATABASE_USER: oncall
      DATABASE_PASSWORD: secret
      BROKER_TYPE: redis
      REDIS_URI: redis://redis:6379/0
      SECRET_KEY: "change-me-in-production"
      GRAFANA_API_URL: http://grafana:3000
      GRAFANA_TOKEN: "glsa_serviceaccount_token"
    depends_on:
      - postgres
      - redis
  celery:
    image: grafana/oncall:latest
    command: celery
    environment: *oncall-env
    depends_on:
      - oncall
---

Celery is mandatory: it processes asynchronous tasks (sending notifications, escalation timers). After deployment, enable the plugin in Grafana: Administration → Plugins → Grafana OnCall → Enable.

Compare self-hosted vs Grafana Cloud:

Creating an on-call schedule

Schedules are configured via UI or API. A typical schedule is weekly rotations:

1. Go to On-Call Schedules → Create schedule.
2. Choose type: Web (UI-based), iCal (Google Calendar/Outlook integration), or API.
3. Add rotation members.
4. Configure time slots.

For a 4-person team on a weekly rotation, each person rotates every week. The schedule automatically accounts for holidays and time zones. It can be exported as an iCal URL — team members subscribe in their calendars.

Building escalation chains

This is the heart of the system. The chain is assembled via a drag-and-drop constructor in the UI:

• Step 1: Notify on-call person from schedule (timeout 10 minutes).
• Step 2: Notify entire team in Slack (timeout 15 minutes).
• Step 3: Notify CTO personally (timeout 5 minutes).
• Step 4: Repeat steps 1–3 until acknowledged.

Each step can notify a specific user, a group, pause, call a webhook, or create an incident in Jira.

User notification policies

Each user configures their notification methods. Example default policy:

1. Slack DM — immediately.
2. Wait 5 minutes.
3. Voice call.

For important incidents you can set an immediate call. Grafana OnCall supports Slack, Telegram, Email, SMS, voice calls (via Twilio or AWS SNS), and mobile push. For night shifts we strongly recommend voice calls — they bypass Do Not Disturb.

The mobile app (iOS/Android) sends Critical Alert push notifications that also ignore silent mode.

Integration with Grafana Alerting

Create a Contact Point of type Grafana OnCall:

Name: oncall-backend
Integration: Grafana OnCall
OnCall integration URL: http://oncall:8080/integrations/v1/grafana_alerting/abc123/

Configure alert routing by labels in Grafana Alerting:

- receiver: oncall-backend matchers: - team = backend - severity =~ critical|high 

Now all critical alerts from backend services automatically go to OnCall and trigger the escalation chain.

What's included in our work?

We deliver a complete set of results:

• Documentation on deployment and configuration (including escalation schemes and notification policies).
• Access to infrastructure (self-hosted or Grafana Cloud) with role model.
• Team training on managing schedules, chains, and notifications.
• Post-production support for 2 weeks.

Process and guarantees

1. Audit current on-call and notification processes.
2. Deploy OnCall (self-hosted or connect to Grafana Cloud).
3. Configure schedules, escalation chains, integrations.
4. Test complete flow: from alert to acknowledgement and escalation.
5. Document configurations and policies.
6. Train team on managing on-call.
7. Post-production support for two weeks.

Certified engineers with experience in Grafana, Kubernetes, and cloud providers. We guarantee every critical alert reaches the responsible person and escalation works correctly. We've implemented projects for companies handling up to 100,000 alerts per day.

Timelines and cost

Basic setup (one cluster, one schedule, one escalation chain) takes from 2 days. Full project with multiple team integrations and custom policies takes up to 5 days. Cost is calculated individually. Get a consultation — we'll assess your project and recommend the optimal configuration.