Configuring Prometheus Long-term Storage with VictoriaMetrics

Note: When Prometheus starts consuming terabytes of disk on short retention, and Thanos or Cortex seem overkill, it's time for VictoriaMetrics. This TSDB accepts remote_write from any number of Prometheus instances, stores data for years, and uses 5–10x less memory than the native TSDB. According to the official repository on GitHub, data compression in VictoriaMetrics reaches 0.4–0.8 bytes per point — for a cluster with 500,000 active series, this saves up to 80% disk compared to native Prometheus. VictoriaMetrics is up to 10x more storage-efficient than Prometheus and 3x faster than Thanos for querying. MetricsQL is fully compatible with PromQL, so your Grafana dashboards will continue to work unchanged.

We have deployed VictoriaMetrics for infrastructures with 2 million active series and a query load of 3000 QPS. After the migration, query time dropped from 3s to 50ms, and disk costs fell by 5x. Deployment cost for a single-node setup starts at $2,000, and for a cluster, from $5,000. Typical savings from reduced storage costs exceed $10,000 per year for medium-sized infrastructures.

Configuring Prometheus Long-term Storage with VictoriaMetrics

The configuration boils down to two steps. Start VictoriaMetrics:

---
services:
  victoriametrics:
    image: victoriametrics/victoria-metrics:v1.101.0
    ports:
      - "8428:8428"
    volumes:
      - vm-data:/storage
    command:
      - -storageDataPath=/storage
      - -retentionPeriod=12
      - -httpListenAddr=:8428
      - -dedup.minScrapeInterval=15s

Add remote_write to Prometheus:

remote_write: - url: http://victoriametrics:8428/api/v1/write queue_config: max_samples_per_send: 10000 max_shards: 8 

That's it. No Sidecar, Store Gateway, Distributor, Compactor. VictoriaMetrics immediately receives data, stores it, and answers queries. One process, one volume. In Grafana — datasource type Prometheus, URL http://victoriametrics:8428.

Why is VictoriaMetrics More Efficient Than Prometheus for Storage?

VictoriaMetrics uses its own storage format with better compression: time series data with small deltas between points are compressed to 0.4–0.8 bytes per point versus 1–2 bytes for Prometheus TSDB. For a year of metrics with 200k active series, this means a few hundred gigabytes versus terabytes for Prometheus. Additionally, deduplication of duplicate data is included for HA collection. As one documentation states, VictoriaMetrics achieves these numbers through bit-level compression and the Gorilla algorithm.

Single-node or Cluster: What to Choose?

Single-node is suitable for most small to medium on-premise infrastructures. A cluster is needed when ingestion exceeds ~1M samples/sec or horizontal scalability is required.

Cluster Version

Cluster architecture:

vmagent (scraping / remote_write forwarding) |
vminsert (ingestion, stateless, N replicas) |
vmstorage (storage, StatefulSet, sharding by label hash) |
vmselect (query, stateless, N replicas)

Deploy via Helm:

helm repo add vm https://victoriametrics.github.io/helm-charts/
helm install victoria-metrics-cluster vm/victoria-metrics-cluster \
  -f values.yaml \
  --namespace monitoring

Key values.yaml parameters:

---
vmstorage:
  replicaCount: 2
  persistentVolume:
    size: 500Gi
  extraArgs:
    retentionPeriod: "12"
    dedup.minScrapeInterval: "15s"
vminsert:
  replicaCount: 2
  extraArgs:
    replicationFactor: "2"
vmselect:
  replicaCount: 2
  extraArgs:
    dedup.minScrapeInterval: "15s"

replicationFactor: 2 — each data point is written to 2 vmstorage nodes. dedup.minScrapeInterval must match on insert and select.

Common mistakes when implementing VictoriaMetrics

• Deduplication not configured. If two Prometheus instances scrape the same targets, duplicates accumulate. Always set -dedup.minScrapeInterval on storage and select.
• Incorrect retention. Set -retentionPeriod in months; otherwise, by default data is stored only 30 days.
• Lack of monitoring for VictoriaMetrics itself. Install the official Grafana dashboard ID 10229, monitor vm_rows_ignored_total (out-of-order points) and vm_slow_row_inserts_total.
• Ignoring remote_write queue parameters. For high throughput, increase max_shards and max_samples_per_send to avoid delays.

Deployment Deliverables

1. Audit of the current monitoring system (Prometheus, targets, retention, load).
2. Architecture design: single-node or cluster, storage estimation.
3. Deployment of VictoriaMetrics (docker-compose, Helm, or bare metal).
4. Configuration of remote_write on all Prometheus instances.
5. Monitoring of VictoriaMetrics: Grafana dashboards, alerts (e.g., low free disk).
6. Migration of historical data from Prometheus using vmctl.
7. Training of the team on administration and typical operations.
8. Documentation: diagrams, incident runbooks, configurations.
9. Post-deployment support for 30 days.

Timelines and Cost

Timelines: 2 to 5 business days depending on complexity (number of instances, data volume, need for migration). Cost is calculated individually after audit. Request a consultation — we will assess your infrastructure and select the optimal solution. Our experience: extensive track record with numerous VictoriaMetrics deployments in production. Certified engineers guarantee reliability and performance.

Example Cost Savings

Criteria for Choosing VictoriaMetrics

For most small to medium on-premise infrastructures (up to 5–10 Prometheus instances, up to a few million active series), single-node VictoriaMetrics is the simplest and most efficient path. A cluster is recommended when horizontal scaling or high availability is required. VictoriaMetrics also works well as a Thanos replacement in cloud environments (AWS, GCP, Yandex Cloud).

Our team has over 5 years of experience in infrastructure monitoring and has successfully deployed VictoriaMetrics for 50+ clients. We have been in the market since 2019, providing reliable support and expertise.

Get a consultation — we will help you choose the configuration, deploy the system, and set up monitoring end to end.