Per-tenant monitoring: where the shared perimeter loses the signal
A unified observability perimeter shows service health on average. While a hundred tenants operate normally and one sees response time growing on a critical request, the aggregate graph stays in the green zone — averaging swallows the degradation of an individual customer.
The problem surfaces not from an alert but from a tenant's email: this is no longer monitoring, but post-mortem firefighting.
The reason lies in how data collection is designed. Metrics are labeled with service, instance, or node name, but not tenant, so requests cannot be split by customer at query time.
Logs go into a shared stream without a mandatory tenant attribute, and tracing loses attribution at the very first asynchronous hop. This is how metric blind spots appear: the numbers exist, but there is no way to answer whose performance specifically degraded.
The price of this scheme is time. Service degradation is discovered through complaints; then an engineer manually matches logs, response durations, and customer identifiers to assess the scope. Triage takes hours, and some tenants manage to leave before we even learn about the problem.
Per-tenant monitoring removes this limitation: the tenant label becomes mandatory in metrics, logs, and tracing, and separate telemetry streams make it possible to set sensitivity thresholds for each customer individually. Below we break down how to build such a perimeter and what it delivers in practice.
What metric and log isolation delivers: measurable effect
Metric isolation starts with every series and every log record receiving a mandatory tenant label.
Prometheus with such a label scheme makes it possible to build a query for a single customer rather than subtracting it from the shared stream — log and metric separation follows the same key, and that key also becomes the boundary for limits, retention, and access rights.
Until the label exists, any incident is investigated using the aggregate graph, where one tenant's spike looks like degradation of the whole service.
The effect is measurable in three quantities. Incident localization speed drops from hours to minutes: the on-call engineer immediately sees which tenant's error rate has grown and on which operation, instead of sequentially checking customers.
The share of missed failures falls because thresholds are calculated from each tenant's baseline rather than from an average across all.
Noise in alerts stops being background: a rule tied to a tenant fires on its anomaly and stays silent when a neighboring customer legitimately enters a peak.
Service availability is calculated separately for each tenant, so one tenant's degradation does not dissolve into the overall number or distort the picture for the others.
- incident localization — from hours to minutes through label-based querying;
- false positive rate — several times lower with per-tenant thresholds;
- availability is measured per customer, not on average across the service;
- retention and limits are configured individually, and the registry does not grow linearly with the number of tenants.
Collection modes: shared perimeter, full isolation, hybrid
The three schemes differ not by tool but by where the boundary runs between tenant infrastructure and the shared perimeter. This determines how much effort it takes to onboard a new customer and what exactly is visible during failure triage.
| Scheme | When it fits | Limitations |
|---|---|---|
| Shared perimeter | Dozens of tenants, even data separation requirements, one screen across all is needed | Tenant label in the query: an error in it mixes foreign data, and retention is adjusted to the most demanding tenant |
| Full isolation | Strict data separation requirements, different release cadences across tenants | Its own collection agent and its own storage for each: the number of failure points and support volume grow |
| Hybrid | Shared infrastructure metrics plus isolated logs and sensitive indicators | Two access models and two search locations — discipline in routing rules is required |
Hybrid scheme wins where infrastructure metrics are the same for everyone, while load data and log contents must reside separately.
The limitation is that routing rules have to be maintained manually: if a new metric is assigned to the wrong perimeter, it either leaks into shared storage or gets lost.
The scheme is fixed before the first customer. Moving from a shared perimeter to full isolation means rebuilding collection agents and reconstructing metric history, not changing a setting.
Implementation procedure: audit, design, labeling, verification
Implementing per-tenant monitoring is a sequence of steps, each ending with a verifiable artifact: a source registry, a label scheme, a routing rule, a test report.
The order is such that the next stage relies on the previous stage's result — so labeling does not diverge from reality a month after launch.
-
Source inventory. Collect all places where metrics and logs originate: services, queues, databases, background jobs, edge nodes. The output is a registry with source owner, frequency, and collection method; it also becomes the acceptance checklist.
-
Metric stream labeling. Introduce mandatory labels — tenant, environment, service — and validate them at the collection level, not in application code.
Forgetting a label becomes technically impossible; otherwise some graphs end up "shared" and useless during incident triage.
-
Log routing. Convert records into a structured form with a tenant attribute; then rules distribute them across storages and retention periods. A large customer does not see other tenants' records in search and does not incur storage volume for them.
-
Access and alerts. For each tenant, describe who sees what, separately — alert thresholds and on-call routing. Verify that the signal goes only through its own perimeter.
-
Isolation verification. Run attempts to retrieve other tenants' data through queries, dashboards, exports, and alerts. Record a report: how many paths were checked, which were closed, what remained in exceptions and for how long.
-
Perimeter handover. Provide documentation, dashboard and rule templates, the procedure for onboarding a new tenant, and common failure analysis. After that, expansion proceeds without us: a tenant is connected according to the instructions in a single pass.
Deliverables: configurations, schemes, runbooks
Everything we configure remains in your perimeter and your repository: configurations, dashboards, rules — not in closed consoles or in chat threads.
The deliverables are designed for an on-call engineer who did not participate in implementation: they should understand the system from documentation, not from someone else's memory.
Tenant labeling is end-to-end — from metric source to notification channel — so incident triage does not turn into a search for the data owner.
- Observability configurations as code: metric collectors, log agents, retention rules, cardinality limits, and tenant labeling — in the repository, with change history and mandatory review.
- Routing schemes: a map "source → tenant label → rule → notification channel", including summary signal routes and deduplication mode during a mass failure.
- Access control: roles and visibility scopes — the customer's team sees only its own tenant, the on-call engineer receives a summary; plus integration with corporate authentication and a data access log.
- On-call runbook: severity levels, escalation windows, notification templates, procedures for a false positive and for missing data from a tenant.
- Observability dashboards: "per-tenant" and summary templates with automatic tenant substitution, separately — a health dashboard for the observability system itself.
- Storage map: where metrics and logs reside, retention periods by data type, behavior on overflow, and the procedure for exporting a single tenant for incident triage.
- Handover log: architecture review with your team, on-call checklist for the first shifts, list of known limitations, and a development plan — availability targets and synthetic checks per tenant.
Case study: one service was bringing down the shared observability perimeter
The customer is a platform with several dozen tenants on shared infrastructure. Observability was monolithic: one metrics receiver, one log collector, and a shared set of alert rules for all tenants.
The first major failure came from a tenant with an atypical load profile — the application spawned dynamic labels, and the number of active series grew in jumps.
The metrics receiver went into restart, and along with it dashboards disappeared and alerts fired for all other tenants, including fully healthy ones.
The on-call engineer received a flood of alerts, seven out of ten turned out to be false, and incident triage stretched to 4-6 hours: first, someone else's noise had to be separated from one's own outage.
What changed: metric and log streams were separated by tenant, separate limits on series count and collection frequency were introduced, alert rules and routing were separated by owner, and retention periods were set separately for heavy and light sources.
Visualization remained shared — Grafana with source switching by tenant, without dozens of duplicate dashboards.
Metrics before and after
| Indicator | Before | After |
|---|---|---|
| False alerts | 7 out of 10 | 1 out of 10 |
| Incident triage time | 4-6 hours | 30-40 minutes |
| Dashboards during a spike at one tenant | failure for all | degradation only for that tenant |
| Volume of active series | uncontrolled growth | decreased by four times |
The eliminated failure class is cascading: one tenant's failure no longer takes down observability for the entire perimeter, and its load can be limited selectively without touching the others.
Per-tenant monitoring vs. shared dashboard: what's the difference?
Access control and observability separation solve different tasks, and on a shared dashboard they are easy to confuse.
In the first case, all metrics reside in one storage, and the user role only narrows the query — the tenant sees its own series, but alert rules, aggregates, and retention depth remain shared. In the second case, each tenant has its own full observability perimeter.
A shared observability dashboard is usually built on one visualization tool — Grafana — and differs only in access permission sets.
That is not enough when one customer produces a spike: in the shared series it is smoothed by the average, and the on-call engineer learns about the problem at a specific tenant last.
An alert threshold set from the averaged picture either stays silent or fires on someone else's noise. At the same time, metric cardinality grows, and a query for one tenant starts to require as many resources as a query across all.
The isolation boundary is not about access rights to metrics but about the ability to affect someone else's observability.
Access rights to metrics are a read filter: they limit what a tenant sees, but do not prevent it from flooding storage with its series or drowning the shared notification channel.
True separation means separate views and alert rules for each tenant, quotas on series count and retention, separate alert routing, and its own trigger thresholds.
In practice, this gives predictable detection time: an incident at a specific customer is visible in minutes, not after someone among the neighbors on the shared dashboard notices it.
A tenant availability report is assembled without manual filtering, and overload of one perimeter does not distort the picture for the others.
What to clarify before labeling the metric stream
Before labeling the metric stream, we clarify with the customer who reads the data and how: the on-call engineer, support, or the customer itself. This determines the label scheme, retention, and which alerts go to the on-call shift.
Will storage volume grow because of tenant breakdown?
The breakdown adds one dimension to a metric, and by itself it has almost no effect on volume. Growth comes from label cardinality inside a tenant — request identifiers, user addresses, arbitrary strings.
Therefore, we send to labeling only those attributes that are actually used for filtering and building alerts, while auxiliary values go into an aggregate without a label.
We set retention separately for detailed and aggregated series — detailed ones live short, while summaries live long and take up little space.
Is it necessary to separate the entire stream completely?
No. Partial separation works better than total: two or three dozen metrics receive the tenant label, the ones used to calculate availability and errors, while the rest remains in the shared aggregate.
This way the customer sees the picture for each client where it is needed for on-call, and does not incur storage volume for splitting everything indiscriminately.
What will happen to already collected data?
We perform migration without stopping collection: new labels are written in parallel with the old ones, dashboards are switched to the new scheme, and the old one lives out its retention and expires on its own.
Rollback is returning dashboards and alert rules to the previous queries, while history is preserved.
How will this affect the on-call shift?
The on-call engineer receives an alert with the tenant specified and a link to a dashboard with the filter already applied.
Instead of manually searching the shared stream — one transition to the problematic customer, so the time from alert to triage start is reduced, and the number of false escalations falls.
Let's discuss the observability separation design
An analysis of per-tenant monitoring starts not with tool selection but with your current observability picture: how metrics and logs are collected, where data from different tenants is mixed, and which questions remain unanswered.
Below is what to prepare for a task review with engineers: these items also form the basis for the effort estimate.
- Problem statement — the list of tenants, how their perimeters differ, what requirements exist for observability data isolation, and how quickly an incident should become visible.
- Current observability design — which metric and log collectors are used, how records are labeled, where tenant attribution is lost, and which dashboards and alerts are already in place.
- Architecture review — how services, metric storage, and logs are connected, where the access boundary runs, and how rights to tenant data are separated.
- Volume and retention — the growth rate of the number of time series, retention depth, and requirements for selective sampling, so separation does not double collection load.
- Effort estimate — what we change in labeling and collection rules, what checks we add, and which bottlenecks we address first.
- Implementation plan — the sequence of stages, what we verify at each, and how we roll back if the new perimeter diverges from the current one.
Send the current design: collector settings, metric list, examples of log queries. Describe the task in your own words — we will return with an estimate and a plan, and we will conduct the review together with the engineers who will implement it.