Custom Metrics and Alerts (Prometheus / CloudWatch)

Imagine your online store losing orders, but CPU and memory metrics are normal. Where to look? We faced a case where a client complained about slow cart page loading. After implementing custom metrics, we discovered the payment gateway response time exceeded 5 seconds under load. Standard metrics didn't show this. Custom metrics are the only way to see real application behavior. Without them, you're blind: an incident can cost up to 10% of revenue per hour of downtime. This article covers which metrics to customize, how to set them up in Prometheus and CloudWatch, and how long integration takes. With over 5 years of monitoring experience and 50+ projects, we confirm: custom metrics save millions on incidents.

Which metrics to customize first?

Business metrics: orders created per minute, checkout funnel conversion, active user sessions. Technical metrics: task queue size, cache hit rate, specific operation execution time, error count by type. External dependencies: latency to third-party APIs, payment gateway availability, integration status. For example, conversion dropped from 3% to 1% — we would track the conversion_rate metric and alert. For quick issue identification, we also monitor the 99th percentile of response time and database error count (N+1 query).

How to set up custom metrics in Prometheus?

For instrumenting Python (FastAPI), use the prometheus_client library:

from prometheus_client import Counter, Histogram, Gauge from prometheus_fastapi_instrumentator import Instrumentator # Счётчик order_counter = Counter( 'orders_created_total', 'Total orders created', ['status', 'payment_method'] ) # Гистограмма (для percentile) checkout_duration = Histogram( 'checkout_duration_seconds', 'Time spent in checkout process', buckets=[0.1, 0.5, 1.0, 2.0, 5.0, 10.0] ) # Gauge (текущее значение) queue_size = Gauge( 'task_queue_size', 'Current size of processing queue' ) # Использование в коде async def create_order(order_data: dict): with checkout_duration.time(): result = await process_order(order_data) order_counter.labels( status=result.status, payment_method=order_data['payment_method'] ).inc() return result 

Node.js (prom-client):

const client = require('prom-client') const httpDuration = new client.Histogram({ name: 'http_request_duration_ms', help: 'Duration of HTTP requests in ms', labelNames: ['method', 'route', 'code'], buckets: [1, 5, 15, 50, 100, 200, 500, 1000, 2000] }) app.use((req, res, next) => { const end = httpDuration.startTimer() res.on('finish', () => { end({ method: req.method, route: req.route?.path, code: res.statusCode }) }) next() }) 

After adding metrics, remember to expose them via the /metrics endpoint and configure scraping in prometheus.yml.

Prometheus vs CloudWatch: what to choose?

Prometheus allows collecting metrics at up to 10 ms intervals, 20 times faster than CloudWatch. However, CloudWatch is more convenient for AWS environments. For detailed study, refer to Prometheus documentation and CloudWatch documentation.

Prometheus Rules: Recording and Alerting

For fast dashboards and timely notifications, we configure recording and alerting rules. Recording rules precompute complex expressions (e.g., job:request_errors:rate5m), speeding up Grafana queries. Alerting rules trigger notifications when thresholds are exceeded. Example configuration:

groups: - name: app_slo interval: 30s rules: # Recording rule: precomputed error metric - record: job:request_errors:rate5m expr: rate(http_requests_total{status=~"5.."}[5m]) # Alert: high error rate - alert: HighErrorRate expr: job:request_errors:rate5m > 0.05 for: 2m labels: severity: critical annotations: summary: "Error rate {{ $value | humanizePercentage }}" 

Alertmanager then sends notifications to Slack, Telegram, or PagerDuty. This allows response in minutes, not hours.

How to set up CloudWatch Custom Metrics?

import boto3 cw = boto3.client('cloudwatch') def put_metric(name: str, value: float, unit: str = 'Count', dimensions: dict = None): metric_data = { 'MetricName': name, 'Value': value, 'Unit': unit } if dimensions: metric_data['Dimensions'] = [ {'Name': k, 'Value': v} for k, v in dimensions.items() ] cw.put_metric_data( Namespace='MyApp/Business', MetricData=[metric_data] ) # Usage put_metric('OrdersCreated', 1, 'Count', {'Environment': 'production'}) put_metric('CheckoutDuration', 0.85, 'Seconds', {'PaymentMethod': 'card'}) put_metric('QueueDepth', queue.size(), 'Count') 

CloudWatch Alarm on a custom metric:

resource "aws_cloudwatch_metric_alarm" "queue_depth" { alarm_name = "high-queue-depth" comparison_operator = "GreaterThanThreshold" evaluation_periods = 3 metric_name = "QueueDepth" namespace = "MyApp/Business" period = 60 statistic = "Maximum" threshold = 1000 alarm_description = "Task queue is backed up" dimensions = { Environment = "production" } alarm_actions = [aws_sns_topic.alerts.arn] ok_actions = [aws_sns_topic.alerts.arn] } 

What's included in the work?

How we set up monitoring in 3-7 days

Our process: audit current metrics, select tools, write metric code, configure alerts, test, document, and train the team. You receive Grafana dashboards (or CloudWatch Dashboard), notifications in Slack/Telegram/email, and full documentation. We guarantee 99.9% SLA on correct metric operation. We can integrate with existing systems (PagerDuty, Opsgenie) if needed.

Step-by-step: adding your first custom metric

1. Install the prometheus_client library (Python) or prom-client (Node.js).
2. Create a metric of the required type (Counter, Histogram, Gauge).
3. Instrument the code: add metric calls at key points.
4. Expose the metric (e.g., via /metrics endpoint).
5. Configure metric scraping in Prometheus (add a job in prometheus.yml).

Checklist for setup:

• Prometheus installed or AWS CloudWatch access
• Instrumentation libraries
• Access to application code
• Alertmanager or SNS topic configured

Why custom metrics save budget

Standard metrics (CPU, RAM) don't show business indicators. Without custom metrics, you spend hours searching for nonexistent issues. One incident detected an hour late can cost more than a year of monitoring service. Custom metrics reduce mean time to detect (MTTD) from hours to minutes, reducing financial losses.

Order custom metric setup — gain full control over your application's performance and business indicators. Contact us for a free consultation — we'll evaluate your project in one day. Over 50 monitoring projects and 5+ years of experience guarantee results.