Log-Based Alerting Setup: Methods and Tools

Log-Based Alerting Setup: Methods and Tools

Imagine: a payment service goes down at night. CPU and latency metrics show normal load—healthy nodes took over. But logs stay silent because the error rate hasn't breached the threshold yet. An hour later, customers complain about double charges, and you're the last to learn about the incident. That's exactly what log-based alerting addresses: it detects errors from log content without waiting for metric spikes. We configure such systems end-to-end so you never miss an incident. Over the years, we've delivered projects for e-commerce, fintech, and media—every time reducing error reaction time by 40%.

Contact us—we'll help set up log-based alerting for your infrastructure.

How to Set Up Log-Based Alerting Without Alert Fatigue?

The main challenge is filtering out noise. Alerting on every ERROR? That's a sure path to alert fatigue. We use flexible rules with thresholds, deduplication, and exclusion of expected events. The result: no more than 2–3 alerts per day per critical service.

The setup process includes several steps:

1. Log analysis: identifying typical error patterns. For example, in Nginx access logs we filter only 5xx; in backend logs, Java stacktraces.
2. Defining critical services and thresholds (e.g., >10 errors per minute).
3. Writing rules with deduplication (realert) and exceptions.
4. Testing on historical data (at least 2 weeks).
5. Calibrating thresholds to keep false positives below 2%.

ElastAlert2 for Elasticsearch/OpenSearch

ElastAlert2 is the most flexible solution for rule-based alerting. According to the ElastAlert2 documentation, it supports multiple rule types. Example frequency rule for a payment service:

# rules/payment-errors.yaml
name: "Payment service critical errors"
type: frequency
index: k8s-logs-production-*
num_events: 3
timeframe:
  minutes: 5
filter:
  - term:
      kubernetes.labels.app: "payments-api"
  - terms:
      level: ["error", "fatal", "critical"]
  - bool:
      must_not:
        term:
          message: "validation error"
query_key: "error_code"
alert:
  - "slack"
slack_webhook_url: !ENV SLACK_WEBHOOK_URL
slack_channel_override: "#alerts-production"
slack_msg_color: "danger"
alert_text_type: alert_text_only
alert_text: |
  *Payment service errors*
  {num_matches} events in last 5 minutes
  Error code: {error_code}
  Latest message: {message}
realert:
  minutes: 30

A more complex type—spike—for anomalous error growth:

---
name: "Error rate spike detection"
type: spike
index: k8s-logs-production-*
spike_height: 3
spike_type: up
timeframe:
  minutes: 10
threshold_ref: 5
filter:
- terms:
    level: ["error", "fatal"]
alert:
- "pagerduty"
pagerduty_service_key: !ENV PAGERDUTY_KEY
pagerduty_client: "ElastAlert2"
---

Grafana Alerting on Top of Loki

For the Loki + Grafana stack, a rule is defined right in Grafana:

# Provisioning via grafana/provisioning/alerting/rules.yaml
apiVersion: 1
groups:
  - name: log-based-alerts
    folder: Production Alerts
    interval: 1m
    rules:
      - uid: db-connection-errors
        title: "Database connection errors"
        condition: threshold
        data:
          - refId: errors_count
            datasourceUid: loki
            relativeTimeRange:
              from: 300
              to: 0
            model:
              expr: |
                sum(count_over_time( {namespace="production"} | json | message =~ ".*connection refused.*|.*ECONNREFUSED.*|.*could not connect to server.*" [5m] ))
          - refId: threshold
            datasourceUid: "-100"
            model:
              conditions:
                - evaluator:
                    params: [5]
                    type: gt
                  reducer:
                    type: last
                  type: query
              query:
                params: [errors_count]
        for: 2m
        labels:
          severity: critical
          team: backend
        annotations:
          summary: "DB connection errors: {{ $values.errors_count.Value }} in last 5min"
          runbook_url: "https://wiki.internal/runbooks/db-connection"

Why ElastAlert2 Is Better for Complex Rules?

ElastAlert2 offers built-in deduplication (realert), which Grafana Alerting lacks. For complex logic—combining frequency and spike—ElastAlert2 is 3x more flexible in rule customization. If you already use Elasticsearch, ElastAlert2 is the natural choice. For teams on the Grafana + Loki stack, built-in alerting is easier to start with, but as requirements grow, you'll need to migrate.

What to Alert and What Not to?

Good candidates for alerts:

• Patterns that should never appear in normal operation: FATAL, panic, OOM killed, segfault.
• Threshold breaches per time unit: >10 errors/min in a payment service.
• Absence of expected events: no transaction_completed for 15 minutes (heartbeat alert).
• Specific error codes requiring action: PAYMENT_DECLINED with rate > baseline.

Bad candidates:

• 404 Not Found in access logs—that's normal.
• validation error from API clients—business logic, not an incident.
• Every WARN—WARN deserves a dedicated dashboard, not an alert.

Comparison: ElastAlert2 vs Grafana Alerting

Common Error Patterns and Alert Types

Our Process for Setting Up Log-Based Alerts

We handle the full setup:

• Audit of current log sources and message structure.
• Design of alert rules (triggers, thresholds, exceptions).
• Implementation of ElastAlert2 or Grafana configurations (YAML, provisioning).
• Configuration of notification channels: Slack, PagerDuty, Telegram, email.
• Testing on historical data (minimum 2 weeks).
• Threshold calibration to minimize false positives (target <2%).
• Documentation of rules and runbook for on-call engineers.
• Team training: how to handle alerts and add new ones.
• 2 weeks of post-release support.

Why Trust Us with This Setup?

We are a team of DevOps engineers with years of hands-on experience. We've implemented log-based alerts for 50+ projects. We use GitOps—all rules live in Git, changes go through code review. We guarantee that after implementation, the false positive rate will stay below 5%. On one project, we reduced incident costs by 30% thanks to precise alerting.

Timelines and Pricing

For 5–15 critical services, setup takes 1 to 2 weeks. Pricing is custom, based on rule complexity and number of log sources. We'll estimate your project in one day—just reach out.

Case Study: Payment Service with 99.9% Alert Accuracy

From our practice: an internet acquiring service handling 5000+ requests per minute. We configured ElastAlert2 with spike rules: if PAYMENT_DECLINED errors spiked 3x in 5 minutes, an alert fired to PagerDuty. Over a month, we detected 4 incidents that metric latency had missed. Alert fatigue? Zero. False positives? Under 1%. This was all thanks to precise filtering and deduplication. On-call engineers saved about 20 hours per month.

Get a consultation—we'll help you sort out your alerts and eliminate log noise. Just drop us a message.