Infrastructure Monitoring with Thanos: Scalable Prometheus

Infrastructure Monitoring with Thanos: Scalable Prometheus

We often see this scenario: multiple Prometheus instances collecting metrics from different clusters, data centers, or cloud regions. There is no single point to query all data at once. And Prometheus retention is limited to local disk — storing metrics longer than a few months becomes expensive and inefficient. Meanwhile, load can reach 1 million metrics per second, and each Prometheus instance stores data locally, leading to duplication and increased costs.

Thanos solves both problems. It provides a single global view across many Prometheus instances and moves long-term metric storage to object storage (S3, GCS, Azure Blob). Thanos does not replace Prometheus; it complements it, preserving familiar PromQL and Grafana compatibility. In practice, this can reduce infrastructure costs by up to 40% through storage optimization and downsampling.

Our team has deployed Thanos on dozens of projects, from startups to large enterprise installations handling hundreds of thousands of metrics per second. Below is a practical breakdown of components, typical mistakes, and the deployment process.

Thanos Architecture: Components

Storage Comparison: Local Prometheus vs Thanos Object Storage

Why Thanos Over Standalone Prometheus?

Prometheus is efficient for a single cluster or group of servers. As soon as a second Prometheus appears — for another region, project, or tenant isolation — you face the problem of "fragmented glasses." You have to switch between datasources in Grafana or write complex proxies. Thanos unifies all sources under a single Query endpoint, and global retention is configured once for all data. Our clients save up to 40% on metric storage costs thanks to downsampling and tiered retention.

Avoiding Data Duplication in Thanos

If Prometheus runs in HA (two instances with the same load), Thanos Querier will see duplicate time series. Deduplication is enabled with the flag --query.replica-label=prometheus_replica. But there is a condition: both Prometheus instances must have identical all external labels except the replica label. A typical mistake is different values for cluster or env. As a result, Querier treats them as different sources and does not deduplicate — you will see doubled values on graphs.

Example of correct configuration:

global:
  external_labels:
    cluster: production-eu
    env: production
    prometheus_replica: replica-0 # for second instance use replica-1

We recommend agreeing on the external_labels scheme for each Prometheus instance in advance and testing deduplication on staging.

Why Store Gateway Can Be Slow

Store Gateway caches only indexes, not the actual metrics. Without caching, a query for a month of data can be slow (up to 10 seconds). Optimal setting: --index-cache-size=512MB and deploy Redis or Memcached as a shared cache. This reduces latency by two to three times.

Bucket config with retention for downsampled data:

---
type: S3
config:
  bucket: metrics-thanos-prod
  endpoint: s3.eu-west-1.amazonaws.com
  retention:
    resolution_raw: 30d
    resolution_5m: 90d
    resolution_1h: 365d
---

Raw data (15-second intervals) is stored for 30 days — that's expensive. Downsampled to 5-minute and hourly resolutions are stored longer and cheaper. Queries for the last quarter work with 5m, for the year — with 1h.

Reducing Storage Costs with Thanos

Thanos uses downsampling: rarely requested old data is stored in aggregated form (5-minute or hourly resolution). This reduces storage volume by 10–20 times compared to raw data. For example, storing raw metrics for a year would take ~10 TB, while with downsampling — 500 GB. Plus tiered retention: raw — 30 days, 5m — 90 days, 1h — 365 days. Thus, 70% of data is stored in cheap object storage, yielding up to 40% savings. For scaling Prometheus, Thanos is indispensable: it allows combining metrics from different Kubernetes clusters into a single storage, providing Kubernetes monitoring in tenths of a second.

Thanos Ruler: Cross-Cluster Alerting

Thanos Ruler executes Prometheus recording and alerting rules, querying data through Querier. This enables cross-cluster rules that are unavailable in plain Prometheus:

groups:
  - name: cross-cluster.rules
    rules:
      - alert: ClusterHighMemoryUsage
        expr: |
          avg by (cluster) ( node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes ) < 0.15
        for: 10m

Such an alert fires when the average memory utilization across the entire cluster drops below 15%. In Prometheus, you would have to run a separate query to each instance.