Custom Grafana Dashboards for Application Monitoring

Custom Grafana Dashboards for Applications

We often encounter a situation: infrastructure metrics are fine, but users complain about slow API performance. Without an application dashboard, you have to grep logs and hope for the best. With a properly configured dashboard — 2 minutes for diagnostics. Our experience shows that for a service with 50,000 RPS, a well-designed dashboard reduces MTTR from 30 minutes to 3-5.

An infrastructure dashboard shows hardware state: CPU, memory, disk. An application dashboard displays metrics important for developers: request count, error percentage, latency per endpoint, connection pool states. Let's examine how we create such turnkey dashboards for microservices.

Why the Infrastructure Dashboard Is Not Enough

A typical example: CPU at 30%, memory free, but p95 latency spikes to 2 seconds. The infrastructure dashboard won't show that the problem is a slow SQL query or a connection pool lock. Only an application dashboard with business logic metrics allows pinpointing the problem in minutes.

What Metrics Need to Be Exported?

The application must export metrics itself. For Go use prometheus/client_golang, for Python — prometheus_client, for Node.js — prom-client, for Java — Micrometer with Prometheus registry. Without instrumentation, no data; without data, no dashboard.

Minimal set of metrics every HTTP application should export:

var (
	httpRequestsTotal = prometheus.NewCounterVec(
		prometheus.CounterOpts{
			Name: "http_requests_total",
			Help: "Total number of HTTP requests",
		},
		[]string{"method", "path", "status_code"},
	)
	httpRequestDuration = prometheus.NewHistogramVec(
		prometheus.HistogramOpts{
			Name:    "http_request_duration_seconds",
			Help:    "HTTP request duration in seconds",
			Buckets: []float64{0.01, 0.05, 0.1, 0.5, 1, 2, 5},
		},
		[]string{"method", "path", "status_code"},
	)
)

These metrics form the core of an application dashboard.

Key Panels of the Application Dashboard

Request rate — number of requests per second, split by status code. Quickly spot error spikes:

sum by (status_code) ( rate(http_requests_total{service="$service"}[5m]) ) 

Error rate — percentage of 5xx relative to total traffic. Often displayed as a stat panel with threshold coloring: green up to 1%, yellow up to 5%, red above:

sum(rate(http_requests_total{service="$service", status_code=~"5.."}[5m])) / sum(rate(http_requests_total{service="$service"}[5m])) * 100 

Latency percentiles — p50, p95, p99. Average latency is almost always misleading. p99 is what the unluckiest 1% of users experience:

histogram_quantile(0.99, sum by (le, method) ( rate(http_request_duration_seconds_bucket{service="$service"}[5m]) ) ) 

Saturation — resource states of the application: connection pool size, goroutines, request queue. Without this, you may see normal latency until the service collapses under load.

Typical SLO thresholds:

Drill-down Structure

A good set of dashboards for a service is built in three levels.

Overview — state of all service instances on one screen. If something is red, you know where to look next.

Instance detail — one instance with full metrics: request rate, latency breakdown by endpoint, error rate, JVM heap (if Java), goroutines (if Go), DB connection pool usage.

Endpoint detail — a specific HTTP endpoint. Latency distribution (histogram heatmap), payload size of requests/responses, downstream dependencies. This level is needed when debugging a specific slow operation.

Transition between levels via data link — clicking an instance in the overview opens instance detail with the correct variables. This is not set automatically; it must be configured manually via Data links in panel settings.

Deployment Annotations

Deployment of the application should be visible on the dashboard. Annotation via Grafana Annotations API or Prometheus event:

curl -X POST http://grafana:3000/api/annotations \
  -H "Content-Type: application/json" \
  -d '{ "time": '"$(date +%s000)"', "tags": ["deployment", '"$SERVICE_NAME"'], "text": "Deploy '"$VERSION"' to production" }'

In Grafana, this displays as a vertical line on all time series panels. If after a deployment p99 latency increased, it's visible immediately, without manually correlating deployment time with the graph.

Dashboard Development Process

We start with metric inventory: what the application exports, which labels are available, whether SLOs need to be reflected. Without understanding the available metrics, the dashboard won't work.

Next, the dashboard structure is agreed with the development team: they know best which endpoints are critical, which downstream dependencies matter, and which metrics they look at when debugging.

Implementation: panels, variables, data links between levels, threshold colors, units of measurement (not "value 0.045", but "45 ms"). Finally, provisioning JSON in git and documentation on adding new metrics.

What Is Included in the Work (Deliverables)

• Audit of existing metrics and exporters
• Dashboard structure design (mockup)
• Implementation of panels, variables, data links
• Setup of deployment annotations
• Operations documentation (README, provisioning JSON)
• Team training (1 hour)
• Result guarantee: we don't hand over the dashboard until all metrics are confirmed correct

Timeframe: Dashboard for one service with three levels of detail — 2-3 days. Standardized template for a typical microservice — 1-2 days, then adaptation for specific services is faster.

How to Speed Up Dashboard Development with Templates?

We have prepared templates for typical microservices (Go, Python, Java). They cover 80% of common metrics. Adaptation for specific requirements takes just one day. Over 5 years of experience in DevOps allows us to guarantee that the dashboard will follow best practices of Prometheus and Grafana.

Want the same dashboard for your service? Contact us — we'll evaluate your project in one day and propose a suitable solution.