A Guide to Centralized Logging with Grafana Loki, Promtail, and Alloy

Guide to Centralized Log Collection with Grafana Loki, Promtail, and Alloy

We often hear developers complain: "There are no logs, and the ones we have are in Elasticsearch, which eats 200 GB of RAM and costs a fortune." Elasticsearch indexes everything, giving flexible search, but at the cost of hardware. Grafana Loki offers a different approach: index only metadata (labels), store compressed logs in S3. This cuts costs by 3–5x at volumes from 100 GB/day. In our estimates, switching to Loki reduces log storage costs by 60–70% compared to Elasticsearch. For a typical 100 GB/day workload, this translates to savings of $1,400 per month.

Why LogQL Is Key for Log Analysis

LogQL is Loki's query language, divided into stream selectors (filtering by labels) and pipeline (processing content). Pipeline can include json, logfmt, regexp, line_format, and aggregation functions like PromQL. For example, error metrics by service:

// Stream selector + filter
{namespace="production", app="payment-service"} |= "error"

// Pipeline with JSON parsing and formatting
{app="api-gateway"} | json | status_code >= 500 | line_format "{{.method}} {{.path}} → {{.status_code}} ({{.duration_ms}}ms)"

// Metric query — rate of errors by service
sum by (app) ( rate( {namespace="production"} | json | status_code >= 500 [5m] ) )

From Grafana LogQL documentation

This query can be used in Grafana as an alert rule — exactly like a Prometheus metric. LogQL metric queries enable SLOs and error budgets based on logs.

How to Properly Configure Labeling and Avoid Cardinality

Loki indexes only labels, not log content. High label cardinality kills performance — each unique label combination creates a separate stream. Forbidden patterns:

---
# BAD — trace_id in labels creates millions of streams
labels:
  trace_id: "abc123def456..."

# BAD — user_id in labels
labels:
  user_id: "12345"

# GOOD — only stable identifiers
labels:
  namespace: "production"
  app: "payment-service"
  pod: "payment-service-7d9f8b-xkqp2"
  level: "error"
---

trace_id should be in the log body, not in labels. To jump from Loki to Jaeger, use derivedFields in the Grafana datasource.

Steps to Deploy Loki with Helm

1. Install Helm 3 if not already installed.
2. Add the Grafana repository: helm repo add grafana https://grafana.github.io/helm-charts.
3. Create a loki-values.yaml file with S3 configuration, retention (e.g., 30 days), and scaling.
4. Run the install command: helm upgrade --install loki grafana/loki --namespace logging --create-namespace -f loki-values.yaml.

These steps are for SimpleScalable mode. More complex installations require additional Kafka and Consul setup.

Deployment Architecture

Loki operates in several modes. For most production cases, simple-scalable via Helm is suitable:

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
helm upgrade --install loki grafana/loki \
  --namespace logging \
  --create-namespace \
  -f loki-values.yaml

# loki-values.yaml
loki:
  auth_enabled: false
  commonConfig:
    replication_factor: 1
  storage:
    type: s3
    s3:
      endpoint: s3.eu-west-1.amazonaws.com
      region: eu-west-1
      bucketnames: my-loki-chunks
      access_key_id: ${AWS_ACCESS_KEY_ID}
      secret_access_key: ${AWS_SECRET_ACCESS_KEY}
      s3forcepathstyle: false
      insecure: false
  schemaConfig:
    configs:
      - from: "2020-01-01"
        store: tsdb
        object_store: s3
        schema: v13
        index:
          prefix: loki_index_
          period: 24h
  limits_config:
    ingestion_rate_mb: 64
    ingestion_burst_size_mb: 128
    max_query_parallelism: 32
    max_streams_per_user: 10000
    retention_period: 744h
    reject_old_samples: true
    reject_old_samples_max_age: 168h
  compactor:
    retention_enabled: true
deploymentMode: SimpleScalable
backend:
  replicas: 2
read:
  replicas: 2
write:
  replicas: 2
minio:
  enabled: false

Deployment Mode Comparison

SimpleScalable is the optimal compromise for loads up to 50 GB/day. In this mode, components (read, write, backend) are separated and can scale independently, without the extra complexity of microservices (Kafka, Consul). For small teams, this reduces time-to-market and simplifies operations.

Log Collection Agent Comparison

Typical Mistakes with Loki

• High label cardinality — using trace_id or user_id as labels kills performance. Always check the 'Series' tab in Grafana.
• Missing retention policy — without configuring compactor and retention, the log database grows endlessly, causing storage overflow.
• Single-binary in production — single-binary mode lacks fault tolerance and scalability. Use SimpleScalable or Microservices.
• Ignoring index schema — from version 13, store tsdb is used. Do not use old schemas (boltdb-shipper) unnecessarily.

What's Included in Our Service

We offer a turn-key service:

• Audit of current logging infrastructure (volumes, sources, retention).
• Deployment of Loki (SimpleScalable or Microservices mode) with Helm, S3 storage configuration.
• Installation and configuration of Promtail/Alloy on all hosts.
• Writing labeling rules and LogQL queries for key metrics.
• Creating Grafana dashboards and alerts (SLI, error budget).
• Documentation and team training.
• Access to Grafana dashboards and log storage.
• Post-deployment support for 30 days.

Our team has over 5 years of DevOps experience and dozens of Loki implementations on projects from startups to enterprises. We provide a stability guarantee and SLA for incidents.

Contact us for an audit of your logging infrastructure — we will assess volumes and choose the optimal configuration. Order Loki implementation and reduce logging costs without losing functionality.