Centralized Logging with ELK Stack for Web Applications

After yet another production incident with 5xx errors, we spent 3 hours digging through logs on ten servers. This repeated every month. When a web application serves thousands of users, logs are generated in huge volumes—up to 50 GB per day on 10 servers. Without centralization, finding an error is like looking for a needle in a haystack. The ELK Stack solves this: all logs flow into a central storage with second-fast search. In our practice, incident response time drops by 70% after ELK adoption. Telegram alerts notify about 5xx, slow requests, application errors. We handle the full cycle: from Elasticsearch deployment to dashboards and alerts. Duration: 2 to 7 days depending on complexity.

How ELK Stack Solves Log Collection and Analysis Problems

ELK is a combination of Elasticsearch (storage and search), Logstash (parsing and transformation), and Kibana (visualization). Filebeat delivers logs from servers. The result is a single entry point for all logs, significantly simplifying log monitoring and error investigation.

Choosing a Scheme: ELK vs EFK vs Without Logstash

Logstash excels in capabilities: it parses legacy log formats using grok, enriches data (geoip, useragent). We use it in 80% of projects. However, Elasticsearch Ingest Pipelines are faster: they process up to 15,000 events/s—3 times more than Logstash (5,000). But Logstash handles unstructured data where Ingest falls short.

How We Set Up ELK for Your Project

Docker Compose for a Test Environment

We use the following compose file:

version: '3.8' services: elasticsearch: image: docker.elastic.co/elasticsearch/elasticsearch:8.13.0 environment: - discovery.type=single-node - xpack.security.enabled=true - xpack.security.http.ssl.enabled=false - ELASTIC_PASSWORD=changeme - "ES_JAVA_OPTS=-Xms2g -Xmx2g" volumes: - esdata:/usr/share/elasticsearch/data ports: - "9200:9200" ulimits: memlock: soft: -1 hard: -1 kibana: image: docker.elastic.co/kibana/kibana:8.13.0 environment: - ELASTICSEARCH_HOSTS=http://elasticsearch:9200 - ELASTICSEARCH_USERNAME=kibana_system - ELASTICSEARCH_PASSWORD=changeme ports: - "5601:5601" depends_on: - elasticsearch logstash: image: docker.elastic.co/logstash/logstash:8.13.0 volumes: - ./logstash/pipeline:/usr/share/logstash/pipeline - ./logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml ports: - "5044:5044" - "5000:5000" depends_on: - elasticsearch volumes: esdata: 

Logstash Pipeline: Parsing Nginx Access and JSON Logs

input { beats { port => 5044 } tcp { port => 5000; codec => json_lines } } filter { if [fields][log_type] == "nginx_access" { grok { match => { "message" => '%{IPORHOST:client_ip} - %{DATA:user} \[%{HTTPDATE:timestamp}\] "%{WORD:method} %{DATA:request} HTTP/%{NUMBER:http_version}" %{NUMBER:status_code:int} %{NUMBER:bytes_sent:int} "%{DATA:referrer}" "%{DATA:user_agent}" %{NUMBER:request_time:float}' } } date { match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"]; target => "@timestamp" } geoip { source => "client_ip"; target => "geoip" } useragent { source => "user_agent"; target => "ua" } mutate { remove_field => ["message", "timestamp"] } } if [fields][log_type] == "app_json" { json { source => "message"; target => "app" } mutate { remove_field => ["message"] } } } output { if [fields][log_type] == "nginx_access" { elasticsearch { hosts => ["http://elasticsearch:9200"]; user => "elastic"; password => "changeme"; index => "nginx-access-%{+YYYY.MM.dd}" } } else { elasticsearch { hosts => ["http://elasticsearch:9200"]; user => "elastic"; password => "changeme"; index => "app-logs-%{+YYYY.MM.dd}" } } } 

Sending Logs from Laravel

Through a custom Monolog handler:

class LogstashLogger { public function __invoke(array $config): Logger { $handler = new SocketHandler("tcp://{$config['host']}:{$config['port']}"); $handler->setFormatter(new JsonFormatter()); return new Logger('app', [$handler]); } } 

Now Log::error(...) sends JSON directly to Logstash.

On one project with 10,000 RPS load, we configured a 3-node Elasticsearch cluster with ILM and Logstash with grok patterns for parsing specific application logs. As a result, error search time dropped from 40 minutes to 10 seconds. This allowed the team to respond faster to incidents and reduce MTTR by 65%.

Why ILM Is Mandatory

Without ILM, indices grow uncontrollably, filling the disk in a month. We configure a policy: hot (5 GB or 1 day) → warm (3 days) → cold (30 days) → delete (90 days). This is done via an index template. It's the foundation of cost-effective log storage. Additionally, you can set rollover by size or age to avoid node overload.

Elasticsearch Performance: Practical Tips

• Heap: no more than 50% of RAM and never exceed 31 GB (due to compressed oops)
• Number of shards: 1 shard ≈ 20–40 GB of data. Oversharding is a common mistake
• Slow log: index.search.slowlog.threshold.query.warn: 2s
• Disable swap: bootstrap.memory_lock: true

Comparison: Logstash vs Ingest Pipelines

For complex transformations, Logstash is irreplaceable. Built-in pipeline processors handle typical tasks but cannot work with arbitrary text patterns.

What's Included in ELK Setup

• Deploying an Elasticsearch cluster with optimal settings (shards, ILM, security)
• Configuring Logstash to parse Nginx, PHP, and application logs
• Connecting Filebeat on all servers
• Creating Kibana dashboards for log monitoring (5xx, latency, traffic)
• Setting up ILM to save disk space
• Integrating alerts to Telegram or Email
• Documentation for operation and team training

Workflow

1. Analysis — we study your current logs, sources, and storage requirements
2. Design — choose the scheme (ELK/EFK), outline ILM and dashboards
3. Implementation — deploy the cluster, configure pipelines
4. Testing — verify data ingestion, alerts, search speed
5. Deployment — roll out to production, hand over documentation

We guarantee 99.9% SLA for the cluster. Our engineers have over 5 years of experience and have completed 30+ projects. Contact us to discuss your project and schedule an ELK implementation to reduce error search time. Pricing is determined after an infrastructure analysis.

Elasticsearch Guide