Prometheus Long-Term Storage with Thanos

Prometheus stores metrics locally for only 15 days — that's a bottleneck when you need quarterly analytics, seasonality, or capacity planning. Imagine: a production failure on Friday evening, but metrics from previous weeks are already unavailable. Thanos solves this through object storage (S3, GCS), a global query layer, and deduplication. We have implemented Thanos in 50+ projects, from startups to enterprise, and guarantee you'll forget about insufficient history in 3–5 days. Get a consultation — our certified engineers will help with architecture selection.

Why Thanos instead of remote_write?

Remote_write (VictoriaMetrics, Cortex) requires replacing Prometheus, creates a single point of failure, and lacks deduplication. Thanos runs as a sidecar alongside existing Prometheus, preserves its HA, and enables multi-cluster federation through a shared S3 bucket. Example: in our project with 20 clusters, Thanos Querier serves queries over 90 days without changing PromQL — response speed drops only 10% compared to local data.

How Thanos works

Key insight: Sidecar connects to local Prometheus via gRPC and serves fresh data (within retention) directly to Querier. Store Gateway serves historical data from S3. Querier merges both sources transparently — a query for the last 90 days works without changes.

How to deploy Thanos in 3–5 days?

The implementation process includes:

1. Audit current monitoring: assess metrics volume, retention, storage costs.
2. Design architecture: choose cloud storage, replication scheme, Store Gateway sizing.
3. Deploy Sidecar next to Prometheus in Kubernetes using manifests.
4. Configure Store Gateway and Querier with DNS-based discovery for multi-cluster.
5. Set up Compactor with downsampling and retention policies.
6. Integrate with Grafana: connect Thanos Querier as a data source and configure dashboards.

What is downsampling and how does it reduce costs?

Downsampling reduces resolution: raw data (15s) stored for 30 days, 5-minute aggregates for 90 days, hourly for 1 year. This cuts storage volume by 5–10x without losing trends. Compaction merges small blocks into large ones, reducing file count and speeding up reads.

Deployment in Kubernetes

Sidecar next to Prometheus

---
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: prometheus
spec:
  replicas: 2
  thanos:
    image: quay.io/thanos/thanos:v0.35.0
    objectStorageConfig:
      key: thanos.yaml
      name: thanos-objstore-config
    externalLabels:
      cluster: prod-eu-west
      replica: $(POD_NAME)

externalLabels are critical: they allow Thanos Querier to deduplicate data between replicas. For S3, use bucket thanos-metrics-prod with IAM role via IRSA (no static keys).

Store Gateway and Querier

---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: thanos-store-gateway
spec:
  replicas: 2
  template:
    spec:
      containers:
        - name: thanos-store
          image: quay.io/thanos/thanos:v0.35.0
          args:
            - store
            - --data-dir=/var/thanos/store
            - --objstore.config-file=/etc/thanos/objstore.yaml
            - --index-cache-size=500MB
            - --chunk-pool-size=2GB
          volumeMounts:
            - name: data
              mountPath: /var/thanos/store
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: [ReadWriteOnce]
        resources:
          requests:
            storage: 100Gi
---

--index-cache-size and --chunk-pool-size are key performance parameters. On a cluster with 1TB of data, we recommend increasing to 2GB and 8GB respectively.

Querier configuration:

containers:
  - name: thanos-query
    image: quay.io/thanos/thanos:v0.35.0
    args:
      - query
      - --http-address=0.0.0.0:9090
      - --endpoint=dnssrv+_grpc._tcp.prometheus-operated.monitoring.svc.cluster.local
      - --endpoint=dnssrv+_grpc._tcp.thanos-store-gateway.monitoring.svc.cluster.local
      - --query.replica-label=replica
      - --query.auto-downsampling

--query.replica-label=replica enables deduplication by this label.

Compactor: retention and downsampling

Compactor is the only component running as a single instance. It compacts blocks and performs downsampling:

args:
  - compact
  - --wait
  - --wait-interval=5m
  - --objstore.config-file=/etc/thanos/objstore.yaml
  - --data-dir=/var/thanos/compact
  - --retention.resolution-raw=30d
  - --retention.resolution-5m=90d
  - --retention.resolution-1h=1y
  - --compact.concurrency=1
  - --delete-delay=48h

Downsampling: raw data (15s) 30 days → 5-minute aggregates 90 days → hourly 1 year. When querying for the past year, Thanos automatically selects the appropriate resolution via --query.auto-downsampling.

How to set up multi-cluster monitoring?

Add endpoints of all sidecars to Querier:

--endpoint=sidecar.prod-eu.example.com:10901
--endpoint=sidecar.prod-us.example.com:10901
--endpoint=store-gateway.thanos.svc:10901

Query sum(rate(http_requests_total[5m])) by (cluster) returns data from all clusters in a single result.

What is included in the Thanos implementation service

We offer a full cycle: audit of the current monitoring system, architecture design, component deployment in Kubernetes (sidecar, Store Gateway, Querier, Compactor), IAM and S3 configuration, Grafana integration, documentation, and team training. The result: reliable long-term metrics storage with a global query layer. Our ten years of experience with projects of any scale guarantees quality. Get a consultation — we'll discuss your scenario.

Timelines: 3–5 days for a cluster with HA and multi-cluster federation. Cost is calculated individually after data volume assessment. Contact us to start.

Official Thanos Documentation