Infrastructure monitoring setup with Grafana: provisioning, dashboards, alerts

We often see the same scenario: Grafana is spun up in Docker, dashboards imported from grafana.com, but after a month half the panels show "No data", another third show stale data, and alerts stay silent exactly when the disk on the production node runs out. After surveying 50+ infrastructures, we identified typical mistakes: missing provisioning, ignoring configuration versioning, and suboptimal data source setup. Our experience (over 30 monitoring projects in recent years) shows that proper monitoring setup is not about importing ready-made panels, but about provisioning, version control, and a well-thought-out data source architecture. In 1–3 days we implement monitoring that requires no manual intervention, gives full infrastructure visibility, and reduces dashboard debugging time by 70%.

Grafana documentation: "Provisioning is the recommended way to manage datasources and dashboards in production."

Why provisioning is the standard for production?

Provisioning solves three key problems: configuration reproducibility, version control via Git, and automated deployment. Instead of clicking through the UI each time, you describe all datasources and dashboards in YAML files. This is especially important for Kubernetes environments where pods get recreated — with manual setup, configuration is lost. We guarantee that after configuring provisioning, you can deploy an identical monitoring setup on a new cluster in 15 minutes. More about the concept can be read on Wikipedia.

Which data sources does Grafana support?

Grafana works with datasources — Prometheus, Loki, InfluxDB, CloudWatch, PostgreSQL, Elasticsearch. Each is connected via Administration → Data Sources with explicit URL, credentials, and timeouts. A typical mistake is connecting Prometheus via http://localhost:9090 in a Docker container running Grafana. The container does not see the host's localhost. Use the service name in compose (http://prometheus:9090) or an external IP.

For production, data sources are best stored as code through provisioning. File /etc/grafana/provisioning/datasources/prometheus.yaml:

---
apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    url: http://prometheus:9090
    isDefault: true
    jsonData:
      timeInterval: "15s"
      queryTimeout: "60s"

This allows deploying Grafana as an immutable image — no manual clicks after container restart. Switching to provisioning reduces dashboard debugging time by 70%.

How to set up alerts that never miss an incident?

Since version 9.x, Grafana has its own alerting engine, separate from Prometheus Alertmanager. It can alert directly from dashboard panels or through Unified Alerting. The difference is fundamental: Grafana Alerting evaluates queries itself, rather than relying on Prometheus rules. This gives more flexibility: you can use any data sources, not just Prometheus.

Configuration of a contact point for Telegram via provisioning (alerting/contact-points.yaml):

---
apiVersion: 1
contactPoints:
  - orgId: 1
    name: telegram-ops
    receivers:
      - uid: telegram-ops
        type: telegram
        settings:
          bottoken: "${TELEGRAM_BOT_TOKEN}"
          chatid: "${TELEGRAM_CHAT_ID}"
          message: |
            {{ range .Alerts }}
            🔥 {{ .Labels.alertname }}
            Instance: {{ .Labels.instance }}
            {{ .Annotations.summary }}
            {{ end }}
---

Templates use Go template syntax. {{ range .Alerts }} iterates over triggered alerts in a group. We recommend setting at least three alerts: disk > 85%, RAM > 90%, CPU > 80% sustained. When exceeded — send to Telegram or PagerDuty.

Which monitoring stack to choose for 20+ nodes?

For infrastructure monitoring, the typical stack is:

On small infrastructures (up to 20 nodes) this suffices. As Prometheus grows, memory becomes a bottleneck — then look into Thanos or VictoriaMetrics as a remote_write endpoint.

What is included in the work

When ordering a turnkey monitoring setup, we provide:

• Documentation on monitoring architecture and dashboard structure
• Provisioning scripts for configuration reproducibility
• Dashboards tailored to your infrastructure (hosts, Kubernetes, services)
• Alerts with integration into Telegram/Slack/PagerDuty
• Access to a Git repository with configuration
• Team training (1 hour) on using the new monitoring
• Support for 2 weeks after deployment

How the process works

We start with an audit of the current state: what data sources exist, what metrics are collected, whether provisioning is in place or everything is done manually via UI. This takes a few hours.

Next, we design the dashboard structure for your specific infrastructure. Not importing generic dashboards, but adapting to real services: which hosts, which node labels, which services to split into folders.

Implementation includes configuring provisioning for datasources and dashboards, basic alerts (disk > 85%, RAM > 90%, CPU > 80% sustained), integration with Alertmanager or direct messaging. Finally, documentation on dashboard structure and instructions for adding new ones.

Timeline

Approximately: from 1 day for a basic setup with ready-made templates to 3 days for a full provisioned stack with custom dashboards and alerting.

Order monitoring setup right now — get a consultation on stack architecture and an accurate time estimate. Contact us, and we will select the optimal solution for your infrastructure. 50+ companies work with us, and we guarantee transparency and fault tolerance for your monitoring.