Centralized Logging Setup with Loki and Grafana

With 7+ years of experience and over 40 production deployments, we deliver robust Loki stacks. Logs scattered across dozens of servers — finding the root cause of a crash turns into a week-long quest. Under peak loads, logs get lost, and searching for an error takes hours. We solve this in a day: set up centralized logging with Grafana Loki and Promtail. Unlike ELK, Loki does not index content, only labels. In practice, this yields up to 70% savings on storage and two to three times less operational overhead. On one project with 50 microservices (10 million requests per day), we reduced time-to-detect from 2 days to 15 minutes. Contact us — we'll complete the setup in 2-3 days. Basic setup cost starts at $500 for the stack (Loki, Promtail, Grafana on Docker; one log source).

Why Loki is Better than Elasticsearch

According to official Grafana Labs data, Loki provides up to 70% storage savings compared to ELK. With 500 GB of logs per day, monthly savings can reach $10,000. Loki stores logs in compressed chunks without an inverted index. Query speed by labels is milliseconds. If you do not need complex aggregation over arbitrary fields, Loki is the right choice. Infrastructure savings can range from $5,000 to $15,000 per month depending on log volume.

How We Deploy the Stack

We use Docker Compose for rapid deployment. The configuration includes three services:

---
version: '3.8'
services:
  loki:
    image: grafana/loki:3.0.0
    ports:
      - "3100:3100"
    volumes:
      - ./loki-config.yml:/etc/loki/local-config.yaml
      - loki_data:/loki
    command: -config.file=/etc/loki/local-config.yaml
  promtail:
    image: grafana/promtail:3.0.0
    volumes:
      - /var/log:/var/log:ro
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
      - ./promtail-config.yml:/etc/promtail/config.yml
    command: -config.file=/etc/promtail/config.yml
  grafana:
    image: grafana/grafana:11.0.0
    ports:
      - "3000:3000"
    environment:
      - GF_AUTH_ANONYMOUS_ENABLED=false
      - GF_SECURITY_ADMIN_PASSWORD=admin_password
    volumes:
      - grafana_data:/var/lib/grafana
      - ./grafana/provisioning:/etc/grafana/provisioning
volumes:
  loki_data:
  grafana_data:

Loki Configuration

auth_enabled: false
server:
  http_listen_port: 3100
  grpc_listen_port: 9096
common:
  path_prefix: /loki
  storage:
    filesystem:
      chunks_directory: /loki/chunks
      rules_directory: /loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory
schema_config:
  configs:
    - from: 2024-01-01
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h
limits_config:
  retention_period: 744h # 31 days
  ingestion_rate_mb: 16
  ingestion_burst_size_mb: 32
  max_query_length: 721h
compactor:
  working_directory: /loki/compactor
  retention_enabled: true
  delete_request_store: filesystem

Promtail Configuration

---
server:
  http_listen_port: 9080
  grpc_listen_port: 0
  positions:
    filename: /tmp/positions.yaml
clients:
  - url: http://loki:3100/loki/api/v1/push
scrape_configs:
  - job_name: nginx
    static_configs:
      - targets: [localhost]
        labels:
          job: nginx
          env: production
          __path__: /var/log/nginx/access.log
    pipeline_stages:
      - regex:
          expression: '^(?P<ip>\S+) - (?P<user>\S+) \[(?P<timestamp>[^\]]+)\] "(?P<method>\S+) (?P<path>\S+) \S+" (?P<status>\d+) (?P<bytes>\d+)'
      - labels:
          status:
          method:
      - timestamp:
          source: timestamp
          format: "02/Jan/2006:15:04:05 -0700"
  - job_name: laravel
    static_configs:
      - targets: [localhost]
        labels:
          job: laravel-app
          env: production
          __path__: /var/www/app/storage/logs/laravel.log
    pipeline_stages:
      - multiline:
          firstline: '^\[\d{4}-\d{2}-\d{2}'
          max_wait_time: 3s
      - regex:
          expression: '^\[(?P<timestamp>[^\]]+)\] (?P<env>\w+)\.(?P<level>\w+): (?P<message>.*)'
      - labels:
          level:
          env:
      - timestamp:
          source: timestamp
          format: "2006-01-02 15:04:05"
  - job_name: docker
    docker_sd_configs:
      - host: unix:///var/run/docker.sock
        refresh_interval: 5s
    relabel_configs:
      - source_labels: [__meta_docker_container_name]
        target_label: container
      - source_labels: [__meta_docker_container_log_stream]
        target_label: logstream

How to Integrate Laravel with Loki via Monolog

We write a custom Monolog handler that sends log entries to Loki via HTTP API. It works fire-and-forget — does not block the request.

// app/Logging/LokiHandler.php
namespace App\Logging;

use Monolog\Handler\AbstractProcessingHandler;
use Monolog\LogRecord;

class LokiHandler extends AbstractProcessingHandler
{
    public function __construct(
        private string $lokiUrl,
        private array $labels = []
    ) {
        parent::__construct();
    }

    protected function write(LogRecord $record): void
    {
        $timestamp = (string)($record->datetime->getTimestamp() * 1_000_000_000);
        $payload = [
            'streams' => [[
                'stream' => array_merge($this->labels, [
                    'level' => $record->level->getName(),
                    'channel' => $record->channel,
                ]),
                'values' => [[$timestamp, $record->formatted]],
            ]],
        ];

        $context = stream_context_create(['http' => [
            'method' => 'POST',
            'header' => 'Content-Type: application/json',
            'content' => json_encode($payload),
            'timeout' => 1,
        ]]);

        @file_get_contents("{$this->lokiUrl}/loki/api/v1/push", false, $context);
    }
}

// config/logging.php
'loki' => [
    'driver' => 'monolog',
    'handler' => App\Logging\LokiHandler::class,
    'with' => [
        'lokiUrl' => env('LOKI_URL', 'http://loki:3100'),
        'labels' => [
            'app' => 'web-app',
            'env' => env('APP_ENV', 'production'),
        ],
    ],
],

Step-by-Step Setup for Collecting Laravel Logs

1. Create the LokiHandler class as shown above.
2. Register the loki channel in config/logging.php.
3. Add the LOKI_URL environment variable in .env.
4. Verify log delivery via LogQL: {job="laravel-app"}.
5. Optionally configure pipeline_stages for parsing multi-line exceptions.

How to Write LogQL Queries

LogQL is similar to PromQL. Basic patterns:

# All Laravel errors in the last hour
{job="laravel-app", level="error"} |= "Exception"

# Nginx 5xx
{job="nginx"} | json | status >= 500

# Error rate per minute
rate({job="laravel-app", level="error"}[1m])

# Top slow requests (if request_time is in the log)
{job="nginx"} | regexp `request_time=(?P<rt>[0-9.]+)` | unwrap rt | quantile_over_time(0.95, [5m]) by (path)

# Error count by type
sum by (level) ( count_over_time({job="laravel-app"}[5m]) )

How to Set Up Alerts on Error Spikes

Create an alert rule via provisioning YAML. Example condition: sum(rate({job="laravel-app", level="error"}[5m])) > 0.1. The alert fires if threshold exceeded for over 2 minutes. Configured in grafana/provisioning/alerting/alert_rules.yml. Also add an alert on sudden spike of Nginx 5xx errors.

Process and Timelines

Total: 2-3 days for the full stack. Pricing starts at $500 for the basic stack (Loki, Promtail, Grafana on Docker; one log source). Get a consultation on your project — we'll assess and propose a solution.

What's Included

• Deployed Loki + Promtail + Grafana stack in Docker Compose or on bare metal.
• Configuration of retention, labels, and pipeline_stages for your sources.
• Custom Monolog handler for Laravel (or similar for other frameworks).
• Grafana dashboards visualizing errors, trends, top endpoints.
• Alert rules on error spikes, 5xx, slow queries.
• Operations and recovery documentation.
• Handover and team training (1 hour).
• One-month warranty of correct stack operation after deployment.

Common Issues and Their Solutions

• Incorrect retention setup — compactor forgotten, logs not deleted. Always check compactor.retention_enabled: true.
• Missing pipeline_stages — labels not parsed, filtering difficult. For Laravel, always use multiline stage.
• Too many labels — each new label creates an index, slowing writes. Limit labels to 5-7 per source.

Learn more about logging on Wikipedia. Order the setup and get a consultation on your project.