Backend Logging Setup (ELK Stack) for Mobile App

Backend Logging Setup for Mobile Apps (ELK Stack)

Logging to a file is not logging. When a mobile app backend is running on multiple servers and generating tens of thousands of log lines per minute, finding the cause of an error via ssh + grep takes hours. ELK Stack (Elasticsearch + Logstash + Kibana) or its lighter variant EFK (with Fluent Bit instead of Logstash) transforms this into a query with filters in seconds.

What Exactly We Set Up

The log collection stack has three parts: structured logging at the application level, an agent for collection and sending, and a storage with search engine and UI.

Structured Logs

The application must write logs in JSON — not plain text. A line like [ERROR] 2024-01-15 14:23:11 UserService: null pointer is garbage for Elasticsearch. JSON logs are data:

# Python, structlog
import structlog

logger = structlog.get_logger()

def authenticate_user(user_id: str, device_id: str):
    logger.info(
        "auth_attempt",
        user_id=user_id,
        device_id=device_id,
        platform="ios",
        app_version="3.2.1",
    )
    try:
        token = auth_service.verify(user_id)
        logger.info("auth_success", user_id=user_id, token_expires_in=3600)
        return token
    except InvalidTokenError as e:
        logger.warning("auth_failed", user_id=user_id, reason=str(e))
        raise

// Go, zerolog
log.Error().
    Str("user_id", userID).
    Str("device_id", deviceID).
    Str("endpoint", "/api/v1/auth").
    Int("status_code", 401).
    Dur("duration_ms", elapsed).
    Msg("authentication failed")

Mandatory fields in every log: timestamp (ISO 8601), level, service, request_id (for tracing requests across services), user_id (if applicable). request_id is a UUID generated on incoming request in middleware and passed to all child calls via context.

Fluent Bit for Log Collection

Fluent Bit is preferable to Logstash for most setups: consumes ~1MB RAM vs 500MB for Logstash, configures in INI/YAML, runs as a DaemonSet in Kubernetes.

# fluent-bit.conf
[INPUT]
    Name              tail
    Path              /var/log/app/*.log
    Parser            json
    Tag               app.*
    Refresh_Interval  5

[FILTER]
    Name   grep
    Match  app.*
    Regex  level (warn|error|fatal)

[OUTPUT]
    Name  es
    Match app.*
    Host  elasticsearch
    Port  9200
    Index mobile-backend-logs
    Type  _doc
    Logstash_Format On
    Logstash_Prefix mobile-backend

The grep filter at Fluent Bit level — reduces data volume in Elasticsearch if debug logs are only needed locally.

Elasticsearch and Indexing

For a mobile backend with moderate load (up to 10M events per day), a single Elasticsearch node or managed cluster (Elastic Cloud, AWS OpenSearch) is sufficient. Index lifecycle management (ILM) is mandatory: logs older than 30 days move to cold tier or delete, otherwise disk runs out in a week.

// Mapping template to prevent dynamic mapping
{
  "mappings": {
    "properties": {
      "timestamp": { "type": "date" },
      "level": { "type": "keyword" },
      "service": { "type": "keyword" },
      "user_id": { "type": "keyword" },
      "request_id": { "type": "keyword" },
      "message": { "type": "text" },
      "duration_ms": { "type": "long" },
      "status_code": { "type": "integer" }
    }
  }
}

Dynamic mapping is a source of problems: Elasticsearch infers field type from its first value, and if status_code first arrives as a string — all subsequent numeric values cause mapping errors.

Kibana: Search and Dashboards

After setting up collection — create Index Pattern in Kibana, configure Discover for field search, build Lens dashboards:

• Error rate by endpoint over the last hour
• Top-10 slowest queries (duration_ms > 1000)
• Active users by user_id in real-time
• Heatmap of errors by hours and days

KQL (Kibana Query Language) for search is simpler than SQL: level: error AND service: auth-service AND duration_ms > 500 — and immediately see all slow auth errors.

Security

Logs contain user_id, device information, sometimes — fragments of request data. Never log: passwords, full tokens, card numbers, personal data. PII in logs is a GDPR violation. Mask at logger level:

def mask_sensitive(data: dict) -> dict:
    sensitive_keys = {'password', 'token', 'card_number', 'cvv'}
    return {k: '***' if k in sensitive_keys else v for k, v in data.items()}

Timeframe

Basic EFK stack with Docker Compose, structured logging for one service, basic Kibana dashboards: 2–3 days. Production-ready setup with ILM, Kibana Watcher alerts, multiple services and security: 5–8 days. Cost calculated individually.