Imagine your online store goes down, but the default Grafana dashboard shows only green graphs, hiding that Redis is full or N+1 queries in the database are killing P95. We've encountered this dozens of times. A custom dashboard solves this by designing panels for real incident scenarios. We develop dashboards that answer three key questions in seconds: is the service alive, where is the bottleneck, what to do? Experience shows a custom dashboard is 5x faster than default in incident search—MTTR drops from 40 to 15 minutes. Our track record: over 50 dashboards for projects of varying complexity, from startups to enterprise.
Why Default Dashboards Don't Solve Your Problems?
Community dashboards suffer from two problems: information noise and lack of context. CPU and memory panels per host are not service monitoring but hardware metrics. Your SRE doesn't look at CPU unless there's an incident. They need: 5xx errors, response latency, and status of external dependencies. Default dashboards don't provide this. For example, one of our clients—a food delivery service—after implementing a custom dashboard reduced average recovery time (MTTR) from 40 to 15 minutes.
How to Build a Dashboard That Answers Real Questions?
We use a metric pyramid: at the top—availability and errors, below—performance, then resources. Each panel is actionable. For instance, a metric "CPU 67%" is useless. We add a trend, target threshold, and scaling alert. This way the engineer doesn't guess but makes a decision. As a result, the team saves up to 8 hours per week on problem finding.
| Component | Default Dashboard | Custom Dashboard |
|---|---|---|
| Number of panels | 20+ (noise) | 5-7 (only what's needed) |
| Answer to "What to do?" | No | Yes: alert, trend, threshold |
| Problem search time | >10 minutes | <2 minutes |
We embed dashboard variables: $__timeRange, $__interval, plus environment and instance variables. This allows viewing staging and production without cloning.
Which Metrics to Include in a Dashboard for Rapid Problem Finding?
It's important to select metrics that reflect user experience and service health. We recommend an SLI/SLO approach: define service level indicators (error rate, latency, throughput) and target values. The dashboard must include:
- Error Rate (5xx)—error percentage. Threshold: <1% for critical services.
- P95 Latency—delay for 95% of requests. Threshold depends on SLA.
- RPS—requests per second, needed for understanding peaks.
- Uptime—availability, checked via synthetic monitoring.
- Database metrics: active connections, query latency, slow queries.
- Cache metrics: hit rate, memory usage, evictions.
Example Dashboard Structure for a Web Application
Row 1: Service Health (big stat panels) [Error Rate %] [P95 Latency ms] [Uptime %] [Active Users] Row 2: Traffic & Performance [RPS - time] [Response time P50/P95/P99 - time] [HTTP status breakdown] Row 3: Infrastructure [CPU % per host] [Memory % per host] [Disk I/O] [Network I/O] Row 4: Database [DB Connections active/max] [Query latency P95] [Slow queries count] Row 5: Cache [Redis hit rate %] [Redis memory usage] [Evictions per sec] Show example PromQL queries for key metrics
| Metric | Query |
|---|---|
| Error Rate | sum(rate(http_requests_total{status=~"5..", job="app"}[5m])) / sum(rate(http_requests_total{job="app"}[5m])) * 100 |
| P95 Latency | histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket{job="app"}[5m])) by (le)) |
| Active DB Connections | pg_stat_activity_count{datname="mydb", state="active"} |
| Redis Hit Rate | rate(redis_keyspace_hits_total[5m]) / (rate(redis_keyspace_hits_total[5m]) + rate(redis_keyspace_misses_total[5m])) * 100 |
Dashboard as Code: Dashboards in Git
Storing dashboards in UI chaos leads to losses. We use Dashboard as Code via Grafonnet or Terraform Grafana provider. Example in Jsonnet:
local grafana = import 'grafonnet/grafana.libsonnet'; local dashboard = grafana.dashboard; local graphPanel = grafana.graphPanel; dashboard.new( 'Application Overview', time_from='now-1h', refresh='30s', ) .addPanel( graphPanel.new( 'Error Rate', datasource='Prometheus', ) .addTarget( grafana.prometheus.target( 'sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) * 100', legendFormat='Error Rate %' ) ), gridPos={ x: 0, y: 0, w: 12, h: 8 } ) This approach enables versioning, auditing, and reproducibility. Deployment annotations are added via CI/CD—each release is marked on graphs. A custom dashboard reduces downtime root cause search time by 5x, saving the team up to 8 hours per week.
Process
- Audit current infrastructure and metrics
- Design panels using a top-down approach
- Build dashboards in Grafana with variables and annotations
- Implement Dashboard as Code (Jsonnet/Terraform)
- Documentation and team training
- Guarantee: free adjustments within 30 days
Estimated Timelines
| Dashboard Type | Timeline |
|---|---|
| Basic (error rate, latency, traffic) | 1-2 days |
| Full (all application layers) | 3-5 days |
| Dashboard as Code + git workflow | 1-2 days |
| Deployment annotations | 1 day |
Experience and Guarantees
We have developed over 50 dashboards for projects of varying complexity—from startups to enterprise. Our engineers are certified in Grafana and experienced with Prometheus, VictoriaMetrics, InfluxDB. We guarantee that every dashboard answers three questions: "Is the service alive? Where is the problem? What to do?"
Order custom dashboard development—from design to team training. Get a consultation for your project. Contact us for a preliminary assessment.