AIOps for Log Analysis: Automatic Log Anomaly Detection

AIOps for Log Analysis: Automatic Log Anomaly Detection

Elastic reports 100 GB of logs per day, Alertmanager is silent — no thresholds set. The on-call engineer checks Kibana manually every two hours. An incident is noticed 40 minutes after users have already contacted support. This is a classic scenario: lots of logs, little signal, people get tired. Our certified DevOps and SRE engineers implement turnkey AIOps solutions so you only receive alerts that truly require a reaction. We estimate your project within one day.

AIOps for log analysis is not about replacing people. It's about filtering noise: out of 10,000 events per minute, automation picks the 3 worth looking at right now. We guarantee a 40% MTTR reduction within the first month after deployment.

What ML Actually Does in Logs — and Where Not to Trust It

Modern approaches to automatic log analysis fall into two classes: template-based (pattern parsing, drain algorithm) and embedding-based (vectorization via LLM or BERT-like models). Both work, but in different scenarios.

Drain3 is a Python library implementing a log parsing tree algorithm. It takes a raw log line and outputs a template: Connection to <IP>:<PORT> failed after <NUM> retries. This is useful for grouping: a thousand lines about different IPs become one template with a counter. An anomaly occurs when the counter for a specific template spikes or a new template appears that didn't exist before.

from drain3 import TemplateMiner
from drain3.template_miner_config import TemplateMinerConfig

config = TemplateMinerConfig()
config.load("drain3.ini")
template_miner = TemplateMiner(config=config)

with open("app.log") as f:
    for line in f:
        result = template_miner.add_log_message(line.strip())
        if result["change_type"] == "cluster_created":
            print(f"NEW PATTERN: {result['template_mined']}")

# drain3.ini [DRAIN] sim_th = 0.4 depth = 4 max_children = 100 max_clusters = 1024 

A new pattern in production logs is a signal. Not always an incident, but always a reason to look.

The embedding approach is needed where log text is semantically diverse and templates work poorly. For example, Python tracebacks or free-text error messages from third-party APIs. Here, sentence-transformers are used for vectorization and FAISS or pgvector for anomaly detection based on cosine distance from known "normal" logs.

But embedding models require GPU or significant CPU resources, and they are less often deployed in production-grade setups. For most tasks, Drain3 plus statistical methods suffice.

How Drain3 Helps in Log Analysis

Drain3 parses logs, groups them into clusters by template, and tracks the appearance of new ones. If a cluster with a connection error appeared once an hour and now ten times per minute — that's an anomaly. We configure automatic alerting based on this change. For production, we use Drain3 in conjunction with Prometheus and Alertmanager.

Why Trust Embedding Approaches?

Embedding approaches (BERT, OpenAI Ada) handle unstructured text better, e.g., tracebacks. They convert strings into vectors, and an anomaly is an outlier in vector space. However, they require more resources and fine-tuning time. For most logs with clear patterns (Nginx, PostgreSQL), Drain3 is more efficient — faster and no GPU needed. Comparison below.

Drain3 vs Embedding: When to Choose What

Stack for Production Deployment

Loki + LogQL + Grafana Alerting

If Loki is already in use, the first step to "smart" alerts is metrics from logs via log_range_queries. For instance, count the rate of ERROR logs:

sum(rate({app="payment-service"} |= "level=error" [5m])) by (app) 

An alert on a sudden spike is already the first level of AIOps without ML. Anomaly via Z-score:

( sum(rate({app="payment-service"} |= "error" [5m])) - avg_over_time(sum(rate({app="payment-service"} |= "error" [5m]))[1h:5m]) ) / stddev_over_time(sum(rate({app="payment-service"} |= "error" [5m]))[1h:5m]) > 3 

This calculates Z-score on the fly: if the current error rate deviates from the hourly average by more than 3 standard deviations — alert.

OpenSearch + Anomaly Detection

OpenSearch (formerly Open Distro for Elasticsearch) has a built-in Anomaly Detection plugin based on Random Cut Forest (RCF). Configuration via API:

PUT _plugins/_anomaly_detection/detectors
{
  "name": "error_rate_detector",
  "description": "Detects anomalies in error log rate",
  "time_field": "@timestamp",
  "indices": ["app-logs-*"],
  "feature_attributes": [
    {
      "feature_name": "error_count",
      "feature_enabled": true,
      "aggregation_query": {
        "error_count": {
          "filter": {
            "term": {
              "level": "error"
            }
          }
        }
      }
    }
  ],
  "detection_interval": {
    "period": {
      "interval": 5,
      "unit": "Minutes"
    }
  },
  "window_delay": {
    "period": {
      "interval": 1,
      "unit": "Minutes"
    }
  }
}

RCF works well on time series with clear seasonality (night/day, weekday/weekend) — it automatically adapts to patterns without manual threshold tuning.

Wazuh for Security Log Analysis

For security events (syslog, Windows Event Log, auditd), Wazuh combines signature-based rules and ML models for behavioral analysis. After agent installation, it automatically parses /var/log/auth.log, /var/log/syslog, and Windows system journals, correlates events, and creates alerts with severity levels.

Integration with Alertmanager and On-Call

ML detection is useless without proper alert routing. A typical mistake: all anomalies land in a single Slack channel; after a week the team ignores them. We configure Alertmanager with a routing tree:

# alertmanager.yml
route:
  group_by: ['alertname', 'service']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  receiver: 'slack-default'
  routes:
    - match:
        severity: critical
        category: security
      receiver: 'pagerduty-security'
    - match:
        severity: critical
      receiver: 'pagerduty-oncall'
    - match:
        anomaly_type: new_log_pattern
      receiver: 'slack-dev-alerts'
      continue: true
receivers:
  - name: 'pagerduty-oncall'
    pagerduty_configs:
      - routing_key: '${PAGERDUTY_KEY}'
        description: '{{ .CommonAnnotations.summary }}'

New log patterns go to the dev channel (maybe just a new feature), critical metric anomalies go to PagerDuty.

Implementation Process and Timelines

We start with an audit: how many logs per day, which systems generate them, whether Elasticsearch/OpenSearch or Loki is already present. The choice of tool depends on this — there's no point in dragging Wazuh if ELK is already in place.

A minimum viable option — Z-score alerting on top of existing Loki in 2-3 days. A full anomaly detector on OpenSearch with RCF configured and integration into PagerDuty — 2 weeks. Implementation of an embedding model for semantic analysis of tracebacks with fine-tuning for your specific stack — 4-6 weeks, this is a project with an ML engineer.

An important point: the first 2-4 weeks after launching the detector will see many false positives — alerts for "normal" anomalies like deployments or scheduled maintenance. Time is needed to set up suppress rules and train the model on historical traffic. This is normal; do not disable the system in the first week.

What's Included

Order turnkey AIOps for logs: we select the optimal stack, configure anomaly detection and integration with your notification system. Contact us via email or messengers — we estimate your project within one business day. Quality guarantee on all work.