ELK Stack Logging Setup for Web Application

Setting Up Logging (ELK Stack) for Your Web Application

ELK (Elasticsearch + Logstash + Kibana) or the modern variant with Beats is a mature platform for centralized logging. It handles volumes from gigabytes per day to terabytes, but requires careful tuning of indexes, resources, and retention policies. Without this, the cluster degrades quickly.

Choosing a Schema: ELK vs EFK

Classic stack: Filebeat → Logstash → Elasticsearch → Kibana

Simplified for small volumes: Filebeat → Elasticsearch (without Logstash, parsing via ingest pipelines directly in ES)

For Kubernetes: Fluent Bit → Elasticsearch → Kibana (Fluent Bit is lighter than Filebeat, native k8s metadata integration)

The choice depends on transformation complexity: if complex enrichment, grok parsing, and routing to multiple destinations are needed — Logstash is essential. If logs are structured (JSON) and simple delivery is all that's needed — ingest pipelines suffice.

Deployment via Docker Compose

For dev/staging environments:

# docker-compose.yml
version: '3.8'
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.13.0
    environment:
      - discovery.type=single-node
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=false
      - ELASTIC_PASSWORD=changeme
      - "ES_JAVA_OPTS=-Xms2g -Xmx2g"
    volumes:
      - esdata:/usr/share/elasticsearch/data
    ports:
      - "9200:9200"
    ulimits:
      memlock:
        soft: -1
        hard: -1

  kibana:
    image: docker.elastic.co/kibana/kibana:8.13.0
    environment:
      - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
      - ELASTICSEARCH_USERNAME=kibana_system
      - ELASTICSEARCH_PASSWORD=changeme
    ports:
      - "5601:5601"
    depends_on:
      - elasticsearch

  logstash:
    image: docker.elastic.co/logstash/logstash:8.13.0
    volumes:
      - ./logstash/pipeline:/usr/share/logstash/pipeline
      - ./logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml
    ports:
      - "5044:5044"   # Beats input
      - "5000:5000"   # TCP input
    depends_on:
      - elasticsearch

volumes:
  esdata:

Logstash Pipeline

Configuration for parsing Nginx access logs and application JSON logs:

# logstash/pipeline/main.conf
input {
  beats {
    port => 5044
  }
  tcp {
    port => 5000
    codec => json_lines
  }
}

filter {
  if [fields][log_type] == "nginx_access" {
    grok {
      match => {
        "message" => '%{IPORHOST:client_ip} - %{DATA:user} \[%{HTTPDATE:timestamp}\] "%{WORD:method} %{DATA:request} HTTP/%{NUMBER:http_version}" %{NUMBER:status_code:int} %{NUMBER:bytes_sent:int} "%{DATA:referrer}" "%{DATA:user_agent}" %{NUMBER:request_time:float}'
      }
    }
    date {
      match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"]
      target => "@timestamp"
    }
    geoip {
      source => "client_ip"
      target => "geoip"
    }
    useragent {
      source => "user_agent"
      target => "ua"
    }
    mutate {
      remove_field => ["message", "timestamp"]
    }
  }

  if [fields][log_type] == "app_json" {
    json {
      source => "message"
      target => "app"
    }
    mutate {
      remove_field => ["message"]
    }
  }
}

output {
  if [fields][log_type] == "nginx_access" {
    elasticsearch {
      hosts => ["http://elasticsearch:9200"]
      user => "elastic"
      password => "changeme"
      index => "nginx-access-%{+YYYY.MM.dd}"
    }
  } else {
    elasticsearch {
      hosts => ["http://elasticsearch:9200"]
      user => "elastic"
      password => "changeme"
      index => "app-logs-%{+YYYY.MM.dd}"
    }
  }
}

Filebeat on Application Servers

# /etc/filebeat/filebeat.yml
filebeat.inputs:
  - type: log
    enabled: true
    paths:
      - /var/log/nginx/access.log
    fields:
      log_type: nginx_access
    fields_under_root: false
    multiline:
      # Nginx access log — single line, multiline not needed

  - type: log
    enabled: true
    paths:
      - /var/www/app/storage/logs/laravel.log
    fields:
      log_type: app_json
    multiline:
      pattern: '^\['
      negate: true
      match: after
      max_lines: 50

output.logstash:
  hosts: ["logstash-server:5044"]
  ssl.enabled: false

# Host metadata
processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_fields:
      target: ''
      fields:
        environment: production
        service: web-app

Sending Application Logs Directly

For Laravel — via custom Monolog handler:

// config/logging.php
'channels' => [
    'logstash' => [
        'driver' => 'custom',
        'via' => App\Logging\LogstashLogger::class,
        'host' => env('LOGSTASH_HOST', 'logstash'),
        'port' => env('LOGSTASH_PORT', 5000),
        'level' => 'debug',
    ],
    'stack' => [
        'driver' => 'stack',
        'channels' => ['daily', 'logstash'],
    ],
],

// app/Logging/LogstashLogger.php
namespace App\Logging;

use Monolog\Logger;
use Monolog\Handler\SocketHandler;
use Monolog\Formatter\JsonFormatter;

class LogstashLogger
{
    public function __invoke(array $config): Logger
    {
        $handler = new SocketHandler(
            "tcp://{$config['host']}:{$config['port']}"
        );
        $handler->setFormatter(new JsonFormatter());

        return new Logger('app', [$handler]);
    }
}

Now each Log::error(...) sends structured JSON directly to Logstash.

Index Lifecycle Management (ILM)

Without ILM, indexes grow uncontrollably and fill the disk. Policy for application logs:

PUT _ilm/policy/app-logs-policy
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_size": "5gb",
            "max_age": "1d"
          },
          "set_priority": { "priority": 100 }
        }
      },
      "warm": {
        "min_age": "3d",
        "actions": {
          "shrink": { "number_of_shards": 1 },
          "forcemerge": { "max_num_segments": 1 },
          "set_priority": { "priority": 50 }
        }
      },
      "cold": {
        "min_age": "30d",
        "actions": {
          "freeze": {},
          "set_priority": { "priority": 0 }
        }
      },
      "delete": {
        "min_age": "90d",
        "actions": { "delete": {} }
      }
    }
  }
}

Index template ties the policy to a pattern:

PUT _index_template/app-logs-template
{
  "index_patterns": ["app-logs-*"],
  "template": {
    "settings": {
      "number_of_shards": 2,
      "number_of_replicas": 1,
      "index.lifecycle.name": "app-logs-policy",
      "index.lifecycle.rollover_alias": "app-logs"
    }
  }
}

Kibana: Basic Setup

After first launch:

1. Stack Management → Index Patterns → create pattern app-logs-* with @timestamp field
2. Discover → select pattern → verify data is flowing
3. Dashboards → create dashboard with widgets:
   • Top error messages (Terms aggregation by app.message.keyword)
   • HTTP status distribution (Pie chart by status_code)
   • Request rate (Date histogram by @timestamp)
   • Error rate by service (Bar chart filtered by app.level: error)

Saved searches for quick access:

• app.level: error AND environment: production
• status_code >= 500
• request_time > 3

Elasticsearch Performance

Critical settings for production:

# /etc/elasticsearch/elasticsearch.yml
cluster.name: app-logging
node.name: es-node-1

# Prevent swapping
bootstrap.memory_lock: true

# Heap — no more than 50% RAM, no more than 31g (JVM compressed oops limit)
# Set via ES_JAVA_OPTS or jvm.options

# Slow log for debugging slow queries
index.search.slowlog.threshold.query.warn: 2s
index.indexing.slowlog.threshold.index.warn: 1s

# Thread pool for bulk indexing
thread_pool.write.queue_size: 200

Optimal shard count: 1 shard ≈ 20-40 GB. Too many small shards (oversharding) is a frequent cause of cluster degradation.

Alerts via Kibana Alerting

// Watcher for alerting on 5xx errors
PUT _watcher/watch/high-error-rate
{
  "trigger": { "schedule": { "interval": "5m" } },
  "input": {
    "search": {
      "request": {
        "indices": ["nginx-access-*"],
        "body": {
          "query": {
            "bool": {
              "filter": [
                { "range": { "@timestamp": { "gte": "now-5m" } } },
                { "range": { "status_code": { "gte": 500 } } }
              ]
            }
          }
        }
      }
    }
  },
  "condition": {
    "compare": { "ctx.payload.hits.total.value": { "gt": 50 } }
  },
  "actions": {
    "notify_telegram": {
      "webhook": {
        "method": "POST",
        "url": "https://api.telegram.org/bot<TOKEN>/sendMessage",
        "body": "{\"chat_id\": \"<CHAT_ID>\", \"text\": \"5xx error spike: {{ctx.payload.hits.total.value}} errors in 5m\"}"
      }
    }
  }
}

Timeline

Deploying ELK via Docker Compose, configuring Filebeat on 3-5 servers, basic pipelines for Nginx and application, ILM policy, initial Kibana dashboards: 2-3 working days.

Full setup with parsing all log types, alerting, security configuration (TLS, RBAC), production cluster configuration with replication: 5-7 days.

Scaling an existing cluster or migrating from another logging system is estimated separately after audit.