Setting up infrastructure monitoring with VictoriaMetrics

VictoriaMetrics under peak load: where metrics are lost

Metrics loss almost never looks like a service outage — it looks like a flat line on a dashboard. Agents continue to send points, the graph is drawn, but only half the series reaches storage, and decisions are made based on an incomplete picture. Peak load only reveals what was built into the collection configuration.

The first thing to break is collection on the agent side. The local disk queue fills faster than it can be sent, and the agent starts dropping old points — as a rule, the ones most valuable for incident analysis.

Disparate agents make the picture worse: each has its own queue limits, intervals, and retry rules, so some nodes lose data while others write with delays of minutes.

The second bottleneck is cardinality spikes. New suffixes in container names and labels by user ID generate thousands of new series per hour: the index grows, writes slow down, and no one aggregates or drops the excess series.

This is fixed not by increasing disk size, but by rules on the collection side and aggregation by time windows.

The third is a single point of failure in storage. One node without replication means scheduled maintenance or disk failure wipes out the entire observation horizon at once.

Alerts at that moment are calculated from the same incomplete data: the rule stays silent on gaps or fires on stale values, and the on-call engineer receives false alerts instead of a signal of real degradation.

VictoriaMetrics closes these gaps architecturally, but not by default — retention parameters, deduplication, and alerting rules still have to be set manually. We break down exactly where telemetry collection breaks and how it is closed.

What a single metrics storage gives instead of disparate systems

Disparate systems give a different picture: each node has its own database, its own retention period, and its own downsampling rules.

As load grows, the number of labels multiplies, metric cardinality exceeds what was calculated at launch, and storage starts losing data first — exactly when it is needed for incident analysis.

Then investigation proceeds from fragments of graphs, and the team learns about the problem from users.

A single metrics storage removes this fork. Writes go to one point, re-sent series are recognized and not duplicated, and telemetry retention is set by a policy at the level of the entire storage: fresh data is kept at full resolution, old data is downsampled but not discarded.

We build this on VictoriaMetrics and set cardinality limits at the input — a series over the limit is dropped with a clear error instead of dragging down the node.

The effect is measurable. Incident response time drops from tens of minutes to two or three: the notification is based on the same series that the engineer sees in the graph, without discrepancies between systems.

Recovery time falls by roughly half — months of history are retrieved with a single query instead of assembly from multiple sources.

A whole class of failures due to storage overflow also goes away: telemetry volume grows predictably, by the number of active series, and it is visible in advance.

Data stops disappearing silently — and silent telemetry loss is more expensive than any loud failure, because it deprives the team of facts.

VictoriaMetrics deployment options: comparing modes

The VictoriaMetrics deployment mode is chosen not by metrics volume, but by the availability commitments you make: how quickly monitoring panels show data again after a failure, how long history is retained, and what happens to writes when part of the infrastructure is unavailable.

A single node is one assembly that receives metrics, stores them, and answers queries. Restart or upgrade causes a pause in ingestion, so a buffer agent is added on the source side.

A cluster separates writes, storage, and queries into separate roles and replicates data between storage nodes: failure of one node does not stop ingestion, and metrics fault tolerance no longer depends on a single process.

The practical rule is simple: start with a single node, and move to a cluster when there is a requirement for write continuity or a second independent environment.

Federation is chosen when existing storages need to be combined, rather than building one new one — otherwise you get a unified query without unified history. Below we break down how to move data between modes without losing metrics.

Monitoring implementation stages: from audit to handover

Monitoring rarely breaks because of tool choice — more often because it was implemented before understanding what exactly needs to be measured.

Therefore, monitoring implementation stages go from infrastructure audit to handover to operations: first inventory of sources and collection points, then a metrics collection scheme, and only then storage, alerts, and load testing.

Below is the order of work we maintain on projects so that the system delivers signal within the first day after handover.

1. Infrastructure audit and source inventory. We compile a list of nodes, services, databases, load balancers, and queues, marking who already exposes metrics, who writes only to logs, and who is not observable at all. The output is a coverage map: what is visible, what is partially visible, and where the blind spot is.
2. Metrics collection scheme. We choose collection points and agents, set polling frequency and unified label naming so that data can be correlated between services rather than parsed manually. Here we also fix which metrics are business-critical and which are needed only for incident analysis.
3. Storage and rule setup. We separate metric classes by retention periods, estimate disk volume for current growth, and configure compression and backup. In parallel, we describe trigger rules: thresholds by criticality, notification routing, so that the on-call engineer is not overwhelmed by noise.
4. Load testing. We feed a peak stream into part of the environment, disable a node, fill the disk — and watch what happens to points, latency, and time to notification. We separately verify that panels respond in seconds, not minutes.
5. Handover to operations. We provide access and role separation, documentation on metrics and rules, instructions for the on-call engineer, and train the team on real failure scenarios. After that, we provide support and expand coverage as the environment grows.

If the customer only needs the stack, it is enough to deploy the collector. The difference appears at the third stage: where rules and retention periods are tuned to real incidents, not to a standard set of panels.

Project artifacts: configurations, schemes, regulations

Everything that describes and reproduces the monitoring system on VictoriaMetrics is handed over to your repository — nothing settles in chats and engineers' heads. Here is exactly what you receive:

• Storage configurations — node parameters, metric lifetime by tag groups, downsampling of old data, and limits on the number of series: the registry does not grow uncontrollably, and the depth needed for investigations is preserved.
• Metric flow schemes — from agents on nodes to the moment of writing: intermediate links, buffering during connection loss, failure points, and how to quickly find where a series was lost.
• Alerting regulations — rules, thresholds, evaluation windows, notification grouping: the on-call engineer receives an event, not a storm, and immediately understands what exactly broke.
• Access policies — read and write roles, token issuance procedure, integration with corporate sign-in, visibility separation between teams.
• Dashboards and rules as code — manifests next to service configuration: a change goes through review and rolls back like a regular release.
• Documentation — architectural decisions, instructions for connecting a new source, a node onboarding checklist, and incident analysis procedure.
• On-call regulations — who responds and when, how escalation works, what counts as a false trigger, and how the rule is fixed afterward.

Materials remain in your environment and on your accounts, and the on-call shift learns from them as well — handover does not end with an acceptance certificate.

Case study: a single metrics storage for forty services

A platform of forty services had grown into disparate collection: some applications had their own agents, some had custom scripts, and metrics were stored in different databases with different retention periods.

There was no unified picture: during an incident, the on-call engineer manually compared graphs from several sources, and localizing the cause took tens of minutes. Some failures surfaced through customer reports rather than notifications.

We moved collection to VictoriaMetrics as a common storage and kept a single metrics write format for all services.

Agents on hosts write to a single ingestion point, and retention is set by policy by data class: operational metrics are kept for a long time, technical noise is evicted on schedule.

Monitoring panels and alerting rules are collected in one place — the on-call engineer sees the platform on one screen.

Metrics before and after

• failure detection: from 30–40 minutes to 2–3 — deviation from the baseline triggers, not the fact of an outage;
• recovery time: roughly halved, the cause is visible immediately on the summary graph;
• service downtime: the share of incidents with noticeable customer-facing downtime fell from a third to isolated cases;
• storage volume: the registry stopped growing — excess series are cut at the input, while months of retrospectives remain available;
• engineer workload: manual data collection from different panels is eliminated, and on-call duty no longer requires digging through logs.

How VictoriaMetrics differs from classic monitoring systems

A classic monitoring server stores each series as a separate set of files and keeps the label index in memory. Therefore, as the number of unique combinations grows, you hit memory and write merge operations, not disk.

VictoriaMetrics builds a single storage with columnar compression and a shared label index: a series does not create a separate file, and the number of active series per node is measured in millions.

High cardinality is the main source of failures. It is enough to add a pod ID, build hash, or client address to labels, and the classic stack starts losing data while month-long queries stretch to minutes. Here the constraint remains disk volume and response time, not a crash of the collection process.

Metrics migration does not require rewriting collection: agents continue to send data, only the ingestion address changes, and dashboards and alert rules transfer with almost no edits.

Running two storages in parallel on the same metrics stream allows you to compare values on identical queries and roll back if discrepancies turn out to be significant.

Common concerns: cardinality, storage, migration

Concerns about VictoriaMetrics usually come down to four topics: whether storage will withstand growth in the number of new metrics, how to migrate accumulated history, what happens to alerts during cutover, and how to recover data after a failure. We address each one — indicating the settings responsible for it.

How is history stored and what about backup?

Retention policy is two-tier: fresh points are kept at full resolution, then we lower detail by age.

Storage backup is done with snapshots to object storage, and recovery is verified on a test environment on schedule — otherwise a copy exists but there is nothing to deploy it with.

Can history be migrated without losing alerts?

Data migration goes through export to an open format and subsequent backfill, so past periods remain available in the same monitoring panels.

Alert rules during cutover are calculated on two environments at once: discrepancies are compared over control windows, and the old one is turned off only after they match.

Let's discuss the task: implementing VictoriaMetrics for your environment

To come back with a project estimate and a realistic work plan, full access to the working environment is not needed — a description is enough. The more accurate the inputs, the fewer clarification rounds and the fewer discrepancies between promised and actual timelines.

What to send or tell us so that we discuss the task concretely:

• how many nodes, services, and environments we cover with monitoring and where they are located — own racks, cloud, or hybrid;
• how services expose metrics now: exporters and agents, scheduled polling, or push from the application side;
• expected volume of metric series and polling frequency — this determines storage size and the number of VictoriaMetrics cluster nodes;
• where notifications should go: chats, email, on-call systems, ticket queue;
• how long to retain data, whether a long-term archive is needed, and what recovery times after failure you consider acceptable;
• access and security constraints: closed environment, own certificates, access rules for panels;
• which monitoring panels and alerting rules are working now and which of them must not be lost during migration.

After that, we assemble a deployment scheme, estimate storage volumes and fault tolerance, and come back with a monitoring implementation plan and handover options.

Configuration, on-call regulations, and access remain with your team — you choose further support or independent operation.