Setting up Prometheus Long-term Storage with Cortex

Setting up Prometheus Long-term Storage with Cortex

We frequently get requests from teams whose Prometheus metrics no longer fit within the default 15-day retention. The usual options are to sacrifice history or spin up another Prometheus and configure federation. Instead, we offer Cortex — a horizontally scalable multi-tenant backend that ingests data via remote_write in real time. Over 5 years working with distributed systems, we have set up Cortex for dozens of projects: from SaaS platforms to internal monitoring.

How Cortex solves long-term storage?

Cortex stores all metrics in object storage (S3, GCS, MinIO) and allows retention to be set in years. Ingestion happens instantly: data is sent via remote_write, bypassing the ~2-hour delay of Thanos Sidecar. Unlike Thanos, Cortex does not require a Sidecar for each Prometheus and provides built-in multi-tenancy. For one of our platforms with 2 million active series, we cut the latency for fresh metrics from 2 hours to 30 seconds.

When Cortex, not Thanos

Cortex is preferable when:

• strict multi-tenancy isolation is needed (different teams/clients do not see each other's data)
• horizontal scaling of ingestion without the limits of a single Prometheus is required
• data must be available immediately without waiting for upload
• Prometheus may go down — Cortex provides HA on the storage side

Cortex is more complex operationally: it's a distributed system with 8+ separate components. But the result is a unified storage with retention up to 2 years, 99.9% SLO, and full data separation.

Why multi-tenancy matters for teams?

In large organizations, each team sees only its own metrics. Cortex implements isolation via the X-Scope-OrgID header. We configured this for a SaaS product where 5 teams simultaneously wrote data to one Cortex cluster without knowing each other's metrics. We guarantee correct RBAC setup and access auditing.

Component architecture

Prometheus --remote_write--> Distributor --> Ingester (ring) --> Object Storage
Querier <--> Store Gateway (chunks) <-- Object Storage
Compactor
Query Frontend (cache layer)
Ruler (recording/alerting rules)
Alertmanager (alerts)

Distributor accepts remote_write, hashes series by labels, distributes to Ingesters via consistent hashing ring. Ingester holds data in memory, periodically flushes to object storage. Runs in a ring with replication: each series is written to 3 Ingesters. Querier reads from Ingesters (fresh data) and Store Gateway (historical data). Compactor compacts and downsamples blocks.

Which Cortex components are most resource-intensive?

Ingester is the heaviest: holds the last ~2 hours of data in memory. At high volume (1M+ active series), each Ingester consumes 8–16 GB RAM. We recommend planning 2–3 Ingester replicas for fault tolerance. Other components (Distributor, Querier) are less demanding — 2 vCPU and 4 GB RAM suffice.

Prometheus remote_write configuration

remote_write:
  - url: http://cortex-distributor:9009/api/prom/push
    queue_config:
      max_samples_per_send: 10000
      max_shards: 30
      capacity: 20000
    write_relabel_configs:
      - source_labels: [__name__]
        regex: "go_.*"
        action: drop
    headers:
      X-Scope-OrgID: "team-platform"

X-Scope-OrgID is a mandatory header for multi-tenant Cortex.

Minimal Cortex configuration (single-process)

For testing, run as single-binary:

---
target: all
auth_enabled: true
server:
  http_listen_port: 9009
ingester:
  ring:
    kvstore:
      store: memberlist
    replication_factor: 3
blocks_storage:
  backend: s3
  s3:
    bucket_name: cortex-blocks
    region: eu-west-1
  tsdb:
    dir: /tmp/cortex/tsdb
  compactor:
    data_dir: /tmp/cortex/compactor
store_gateway:
  sharding_enabled: true
  sharding_ring:
    replication_factor: 3
limits:
  ingestion_rate: 100000
  max_label_names_per_series: 30
  retention_period: 90d

Production: microservices mode

In production, each component is deployed separately via Helm:

helm repo add cortex-helm https://cortexproject.github.io/cortex-helm-chart
helm install cortex cortex-helm/cortex --namespace monitoring -f values.yaml

Key sections of values.yaml:

---
config:
  auth_enabled: true
  blocks_storage:
    backend: s3
    s3:
      bucket_name: cortex-production
      endpoint: s3.amazonaws.com
  ingester:
    replicas: 6
    resources:
      requests:
        memory: 8Gi
        cpu: 2
  distributor:
    replicas: 3
  store_gateway:
    replicas: 3
    persistentVolume:
      size: 50Gi

Multi-tenancy in Grafana

To separate data, configure Grafana datasource with tenant header:

datasources:
  - name: Cortex-Team-Platform
    type: prometheus
    url: http://cortex-query-frontend:9009/api/prom
    jsonData:
      httpHeaderName1: "X-Scope-OrgID"
    secureJsonData:
      httpHeaderValue1: "team-platform"
  - name: Cortex-Team-Backend
    type: prometheus
    url: http://cortex-query-frontend:9009/api/prom
    jsonData:
      httpHeaderName1: "X-Scope-OrgID"
    secureJsonData:
      httpHeaderValue1: "team-backend"

Cortex vs Thanos: practical comparison

Cortex is better in scenarios requiring strict data isolation and real-time. For simple cases, Thanos may be cheaper.

What is included in our Cortex setup

• Audit of current Prometheus infrastructure
• Architecture design: choice of object storage, tenant schema, retention policies
• Deployment of Cortex in single-binary or microservices mode
• Integration with existing Prometheus (remote_write) and Grafana
• Configuration of Cortex monitoring (Grafana dashboards, alerts)
• Operations documentation and runbook
• Knowledge transfer to the client's team

Timeline: 3 to 10 days depending on complexity. Cost is calculated individually.

Why choose us?

5 years of experience with Prometheus, Cortex, and Thanos. Over 20 successful projects deploying observability stacks. We guarantee 99.9% SLA on the Cortex cluster. Get a consultation for your task — we will assess the project and offer the optimal solution. Contact us to discuss the details.