Grafana Mimir: why fragmented monitoring does not scale
While there are several teams, metrics sprawl across their own systems: each group has its own collection agent, its own storage, its own alerting rules.
Formally, everyone has observability, but fragmented monitoring does not provide a single picture of the product, and an attempt to assemble it in one place runs into incompatibility of sources.
The second problem is truncating metric history.
To fit on disk, retention is cut to weeks, and investigating an incident that happened a month and a half ago begins with "the data is already gone." Slow degradation, seasonal peak, gradual resource leak — everything visible only on a long window has nothing to compare against.
Then dashboards diverge. Different label names, different scrape interval, their own aggregation methods — two graphs for the same dependency show different numbers.
Finding the cause of failure turns into manual correlation of events by time across several panels and asking colleagues who have data retained longer.
A single metrics storage removes this by design: data from all teams flows into one layer, history length is set by retention policy rather than the disk size of a specific machine, and one query shows the whole picture. Grafana Mimir is one way to assemble such storage; let's break down how this is covered.
What changes after moving metrics to a single storage
Before a single storage, metrics live in several local databases: some nodes write to one, some to another, each with its own retention rules and history depth.
When such a database goes down, the on-call engineer sees a hole in graphs and cannot distinguish a real failure from a data gap — investigation stretches because first you need to understand what exactly is silent.
A single collection point covers this: all sites write to a shared cluster, and deduplication by the "source + series" pair collapses copies that appear when scraping from multiple replicas and when nodes move.
The panel keeps one truthful series instead of a set of diverging ones, and queries are served by the shared Grafana Mimir interface rather than a local database.
Long-term history retention is offloaded to object storage, so comparing "how it was a quarter ago" no longer requires exports and manual reconciliation.
Horizontal scaling is done by adding ingestion and read nodes — growth in the number of series does not hit the disk of a single server, and old blocks simply move to the cold layer.
The class of failures also changes. Losing a node does not bring down ingestion: replicas continue writing, reads switch to live instances, and recovery boils down to replacing a node in minutes rather than restoring a backup for hours.
Monitoring fault tolerance stops being a separate project — it is built into the replica placement scheme and verified by routine node shutdown.
Metric storage options in Grafana Mimir: comparing approaches
Grafana Mimir hides topology behind a single query API, so the storage mode can be changed without rewriting dashboards and alerting rules.
In practice, the choice is between a single collection node, a cluster mode with metric replication, and a hybrid scheme where part of the flow remains local and part goes to long-term storage.
The difference shows up in two places: how many nodes can fail without data loss and how scaling limits change as the number of sources grows.
| Option | When it fits | Limitations |
|---|---|---|
| Single collection node | Small environment, dozens of sources, a testbed for validating rules | No failure margin: a node stop interrupts metric ingestion, scaling is vertical only |
| Cluster mode with replication | Production with continuity requirements, hundreds of sources | More expensive to maintain: state coordination, replication control, and separate disk monitoring |
| Hybrid storage scheme | Fresh metrics are needed locally and quickly, old ones only for history | More complex query routing, and the junction of environments becomes a new failure point |
Switching between schemes does not require changing instrumentation: metric sending remains the same, history is read through the shared API, and replication is set by a configuration parameter rather than reworking collection.
Therefore, starting on a single node does not close the path to a cluster — expansion is done as the number of sources grows, without migrating already accumulated data.
Implementation stages: from metrics collection audit to handover
We implement Grafana Mimir as a single layer for metrics ingestion and queries, and the work proceeds in stages.
Each stage is closed with a verifiable artifact — a list of targets, configuration, load report — so it is clear what the system rests on and what will remain with the team after handover.
- Audit of metric sources. We build an inventory of targets: which components emit metrics, at what frequency, how many time series they generate. Separately, we calculate label cardinality — it is what determines storage volume and query speed, not the number of servers.
- Storage scheme design. We lay out data by retention periods: operational — with second-level resolution, long-term — with downsampling. Deletion rules are prescribed in advance so the registry does not grow uncontrollably.
- Metrics ingestion setup. Distributed ingestion with replication and a buffer in case a node is unavailable. Compatible ingestion from existing agents is enabled immediately so as not to rebuild collection on all hosts at once.
- Queries and alerting setup. We bring up the query layer, migrate dashboards and rules: the on-call engineer should receive a signal about degradation, not a stream of noise. We check that old dashboards read data without modifications.
- Load testing. We run a profile close to production: cardinality growth, peak writes, heavy queries over a long period. Separately, we look at behavior during node failure — ingestion must not stop.
- Handover to operations. We provide access, the scheme, the on-call regulations, and the expansion procedure. The shift reviews typical failures on a training testbed, not on the production environment.
- Disabling parallel collection. Once we are sure the data is complete, we shut down duplicate environments and record the final scheme in documentation.
Next comes maintenance: a cardinality review once a quarter and a revision of retention periods based on actual consumption.
What remains with your team after monitoring implementation
Implementation ends not with working dashboards, but with a set of artifacts that the team maintains itself. We hand them over to the repository next to the service code: changes go through review, change history is preserved, rollback takes minutes.
- Storage configurations — metric retention rules for each tenant, limits on the number of active series, settings for Grafana Mimir long-term storage and query cache. Everything is in files, so a change is rolled back with one commit.
- Metric flow scheme — from the collector on the host through the receiver to storage: where data is aggregated, where it is dropped, which node hits its limit first. It answers the question "why is there a hole in the panel."
- Access policies — which departments see which tenants, where permissions are read-only, how metric ingestion keys are issued and revoked, which accounts are service accounts.
- Dashboards and alerting rules as code — versioned descriptions of graphs and thresholds, divided into "production is on fire" and "can be checked in the morning."
- Monitoring documentation — a map of services with owners, a list of baseline indicators, the investigation procedure: what to open first when there is a complaint about degradation.
- On-call regulations — escalation levels, response times for each class of alerts, the procedure for temporarily disabling noisy rules, and post-incident review.
Typical operations — connecting a new service, creating a tenant, changing a threshold — are described step by step, so after a couple of months the team does them itself. We get involved when the storage architecture changes, not for daily operations.
Case: time to find the cause of failure dropped several times over
We dealt with a platform of several dozen services: metrics were collected by three separate systems, each with its own retention period and its own dashboard.
When a service went down, the on-call engineer first guessed which one to look at, then manually combined graphs from two sources — finding the cause took about 40 minutes, and another 20 for confirming the hypothesis.
What was changed
We consolidated collection into one storage on Grafana Mimir: local agents send metrics via a single protocol, long-term storage is offloaded to object storage, and recording rules aggregate high-frequency series into lower-frequency ones.
Dashboards and alerts were moved to a shared source, and history was preserved in full — instead of two weeks, it is available for the entire observation period.
- Time to find the cause of failure: was ~40 minutes, now 5-7 minutes for typical scenarios.
- Service recovery time: reduced from 45 to 12-15 minutes due to ready-made alerts and linking metrics with logs.
- Coverage: all critical services under monitoring instead of 60%, each with a single set of baseline dashboards.
A whole class of failures also disappeared — discrepancies between systems, when graphs showed different values for the same service.
Previously this cost hours of root-cause analysis and undermined trust in the data; now there is one source, and the question "which graph to believe" simply does not arise.
What to do if monitoring already runs on an old stack?
On-call is active, alerts fire, metric history has accumulated over years. Switching in one day means a blind spot exactly during the failure time and loss of retrospect — and that is needed precisely at such moments.
Therefore, we carry out migration by running the two systems in parallel, not by "turning off the old and turning on the new."
On metric sources — exporters and servers of the old environment — we add a second write address, remote_write to Grafana Mimir.
Both databases are filled in parallel for several weeks: queries and alerts continue to be served by the old stack, while the new one runs in the shadows. We compare discrepancies on control series and make sure the aggregates match.
We migrate history in two streams.
First, we load the hot window — the last weeks that are actually used for on-call and release rollbacks — then the archive in blocks, in the background and during off-peak hours, so the import does not compete for disk and network with production writes.
We switch on-call in parts: team by team, environment by environment, only when dashboards and rules are put in order and verified against data reconciliation. We store rules as code, so the way back remains a configuration change, not a rewrite.
Before decommissioning the old stack, we verify three things: history completeness, matching alert firings, and response time for typical queries. Describe your configuration — we will come back with a transition plan and assessment.
Technical doubts: storage, load, access
Before starting, the customer checks the same things: how much space the data will take, whether the system will withstand peak load, who will get access. Below are short answers to what is usually put on a separate meeting during Grafana Mimir implementation.
How much storage will the metrics take?
We calculate growth before the start — by the number of active series and scrape frequency. Old blocks are compacted according to retention rules, so the registry grows predictably, and headroom for seasonal peaks is allocated in advance.
Will the system withstand peak load?
Collectors buffer data and limit parallelism, and limits on the number of series are set per tenant — a heavy node does not crowd out the others. We check the boundary with a load test before going live.
Will a stream of notifications overwhelm the on-call engineer?
An alerting rule is an expression over a metric plus a hold period: a signal is sent only if the condition holds for the specified time. Related firings are grouped into a single summary, and thresholds are calibrated against the actual spread.
Can the system be deployed in an isolated environment?
Yes, without outbound access to the internet: components, dashboards, and rules remain inside the perimeter, and updates are brought through an internal image registry.
How is access rights separation handled?
By tenants: a team sees its own metrics and dashboards, while the on-call engineer sees everything but only read-only. Editing rules is a separate role, login is through the corporate directory, and actions are logged.
Let's discuss the task: we will assemble a monitoring scheme for your environment
An estimate based on the description "we need monitoring" is always approximate. To calculate the scope of work and timelines, we need input data about the environment — usually the on-call team or infrastructure documentation already has it. Send whatever is at hand, in any form: a list, a hand-drawn diagram, an export of configurations.
- list of environments and perimeters — where metric collection is needed and whether there are isolated segments without outbound access;
- sources: nodes, containers, databases, queues, load balancers, the applications themselves;
- what already works — agent collectors, a local metrics server, manual exports to spreadsheets;
- expected flow and retention depth: how many points per second and how far back history is needed;
- alerting rules — what we consider degradation, whom we notify, how on-call is organized;
- requirements for access, role separation, and action auditing;
- what hurts now: missed incidents, slow dashboards, an overgrown storage.
In response, we return a monitoring scheme: where we place collectors, how we separate short and long retention, how Grafana Mimir accepts the flow and serves queries over history, how rules and alerts are built.
To the scheme we attach a staged implementation plan, a scope of work with responsibility areas, and a timeline estimate — with the assumptions the estimate depends on.
Describe the task in free form — we will come back with an estimate and plan. If some data is missing, we will clarify the gaps on a call and record what we take on and what remains on your side.