Azure Monitor: where observability breaks in production
Most Azure Monitor implementations break not because the tool is missing, but because of how it is assembled.
Agents are installed on some machines, diagnostic settings are enabled selectively by service, some metrics come through platform counters, some through a custom collector.
There is no single answer to the question "what happened on Wednesday night": data is spread across several Log Analytics workspaces with different retention depth and different access rights.
The second bottleneck is time. Virtual machines, clusters, and databases emit records in UTC, the application writes in local time, and some services write in their own time zone entirely.
When investigating an incident, metric collection and log collection stop aligning: correlation drifts by hours, and the engineer spends time on arithmetic instead of finding the cause.
The third is alerts. The threshold is set on the average over a long window, the aggregation rule swallows short spikes, notification goes to a channel where no one is on duty.
The trigger arrives after customers have already noticed the failure: observability formally exists, but incident detection does not work.
This is fixed not by a new dashboard, but by structure: a single workspace, data collection rules with an explicit set of sources, aligned time zones, thresholds based on the tail of the distribution, and notification routing by on-call schedules. We break down how this is addressed.
What changes after setting up end-to-end metric collection
Previously, the picture was assembled from disconnected signals: machine metrics in one place, application logs in another, ingress errors only in the load balancer log.
The problem surfaced when a user noticed it, and investigation began with the question "what even broke." This is the most expensive scenario: while you collect data piece by piece, the outage window grows.
A unified observability perimeter removes this fork. Metrics, logs, and traces land in a common store and are linked by the same dimensions — request ID, service name, build version, node instance.
Azure Monitor works here as the assembly point: sources are connected consistently, not as a set of separate agents.
Time to detection changes first. Thresholds are set not on CPU load, but on service degradation signals — error rate, response time on significant routes, queue length.
The signal arrives before the user notices the failure, and the "we heard it from the user" class stops being the main source of tickets.
Incident investigation accelerates for the same reason. End-to-end tracing shows the request path through services: instead of sequentially viewing five dashboards, the engineer sees the segment where time or errors increased.
Root causes are confirmed by data — comparing metrics before and after rollout separates a release regression from a network failure or dependency outage.
There is also a cumulative effect: unified retention of metrics and logs keeps past incident investigations reproducible. Recurring outages are closed by monitoring rules, not by re-discovered causes.
Data collection modes and agents: comparing options
Azure Monitor pulls telemetry through several independent paths, and the chosen path determines not only the set of fields, but also the table schema in the workspace.
A mistake here is expensive: some data cannot be collected retroactively later, and changing the collection method on a running fleet of machines requires redeployment on every node.
The decision comes down to three forks: whether access from inside the guest system is needed, whether direct outbound traffic to the cloud is allowed, and whether there are custom event sources. Below are the options actually chosen between.
| Option | When it fits | Limitations |
|---|---|---|
| Agent on node | System and application logs, counters, custom sources | Extension is maintained on each machine, outbound access required, updated only through policies |
| Agentless collection | Platform resource logs, service metrics, change audit | No data from inside the system, field set is defined by the platform, custom logs unavailable |
| Intermediate gateway | Closed segments without direct outbound access, one stream for many machines | Gateway is a single point of failure, limited in throughput, some data types do not go through it |
| External connector | Network devices and third-party systems streaming events | Format normalization required, higher latency, everything depends on channel stability |
Options are not mutually exclusive: usually two or three run on the same perimeter at once. Before rollout, we check on a pilot node which fields arrive and with what delay.
Implementation stages: from source audit to handover
Work proceeds strictly in order: first we describe what already exists, then design data routes, and only after that enable alerts. The reverse order produces log noise, false triggers, and dashboards no one uses.
- Infrastructure and source audit. We go through subscriptions, virtual machines, clusters, databases, and application services; note who already emits metrics and logs, who is silent, where data is duplicated.
- Data flow schema. For each source we define the workspace, retention period, separation by environments and teams, and estimate daily ingestion volume — the registry must not grow uncontrolled.
- Collection setup. We connect agents and diagnostic settings, set unified naming rules for tables and tags so queries work the same in test and production perimeters.
- Alert rules. We calculate thresholds from the actual baseline, not by eye: configure severity, routing to on-call, and duplicate suppression during planned maintenance.
- Test incident validation. In the test perimeter we deliberately take a service down and watch the entire path: did the condition trigger, how many minutes did the signal take, did it reach the right on-call engineer.
- Handover to operations. Dashboards, saved queries, threshold review procedures, and instructions for connecting new sources; we train the shift and stay in touch during the support period.
Deliverables: configurations, schemas, procedures
The deliverable set is not a folder of files, but a set of artifacts by which an Azure Monitor perimeter can be handed to another team or reproduced in a new environment. Each document is tied to a specific action: deploy, verify, roll back, investigate an incident.
Below is what exactly remains in your hands after go-live.
- Configuration files — declarative descriptions of workspaces, collection rules, and telemetry retention periods; redeployment in a new environment happens without manual setup.
- Telemetry flow schema — which sources emit what, along which routes data travels, where it is filtered, where it is stored, and how long each class of records lives.
- Access policies — who can see logs and at what level, who is allowed to change rules and delete data; fixed by roles, not verbal agreements.
- Engineer roles — a matrix of "role / responsibility area / required permissions" by which a new person is onboarded into the perimeter in a day, not a month.
- Alert rule description — conditions, thresholds, recipients, and severity levels; separately lists which notifications are intentionally not sent so noise does not wake the on-call engineer.
- On-call procedure — schedule, escalation order, response time by criticality level, and mandatory actions before shift handover.
- Documentation — instructions for standard operations: connect a new service, change a rule, investigate an incident, roll back configuration to the previous one.
Case study: how failure detection time was reduced
The client came with a fragmented observability perimeter: virtual machine metrics in one dashboard, application logs in a separate store, queue and load balancer events in the on-call engineer's email.
During a failure, the engineer manually cross-checked three sources, and it took 40-60 minutes to understand what exactly had degraded.
What was changed
We consolidated everything into a single perimeter on Azure Monitor and Log Analytics: one workspace for logs, collection of metrics from nodes, applications, and network services, alert rules tied to symptoms rather than individual counters.
Thresholds were set by deviation from the baseline — night alerts stopped waking the team because of normal load peaks.
We added workbooks with ready-made slices by failure type and on-call routing: the message reaches the person who owns the service, together with a link to the needed slice.
Incident investigation, instead of assembling data from scratch, became a check of a ready timeline — from the first deviation to recovery.
Result: average detection time decreased from 45 minutes to 5-7, manual investigation takes about 20 minutes instead of two hours, and the share of recurring failures fell by about threefold — each investigation ended with a rule that catches the same class of failure in advance.
How is Azure Monitor different from Log Analytics?
Azure Monitor is a telemetry platform: it collects data from all subscription resources, stores, processes, and sends signals. Log Analytics is its part: the workspace where event logs are stored and log queries are executed.
The boundary is not by tools, but by data type: numeric metrics and event text are stored separately, with different lifetimes and different response speed.
Metrics are numeric series with a fixed step: node load, number of connections, queue depth. They are written by the platform itself, are delivered within seconds, and serve as the basis for alert rules and charts.
Event logs are records of what happened, with an arbitrary schema; they must be explicitly sent to the workspace through diagnostic categories.
| Layer | What it is responsible for | When to use |
|---|---|---|
| Metrics | Numeric series by resource, fast signals | Reaction to threshold: node failure, latency growth |
| Event logs | Arbitrary records in the workspace | Incident investigation, audit, service correlation |
| Log queries | Sampling and joining events | Summaries, root cause search, long-term storage |
A common mistake is exporting metrics to the workspace "just in case" and in parallel writing to logs what is already there. Ingestion volume doubles, while metric alerts still remain faster and more reliable. The rule is simple: counters go to metrics, context and text go to logs.
Areas of responsibility at the junction are divided as follows: the resource itself is responsible for the metric, diagnostic settings are responsible for its collection, and the workspace with its retention, access, and category composition is responsible for logs.
When this separation is fixed in documentation, the on-call engineer knows where to look for the number and where to look for the cause.
Technical concerns and limitations
Objections to implementing monitoring in Azure are usually the same: will already working log stores have to be rebuilt, how to contain volume growth, and does the team have enough permissions. Below are direct answers with which we start scoping the task.
We already collect logs another way — will everything have to be rewritten?
Collection is configured alongside the current source: data goes to the workspace, while old collectors continue to work until new dashboards start providing the same picture. We migrate in parts — first one perimeter, then the rest.
How do we limit log storage volume?
Retention is set separately for each table: diagnostic logs are kept briefly, security logs longer. Sampling rules on the source side cut noise before writing, so what grows is not volume but the share of useful events.
Do we need to change application code?
For basic metrics — no, they are taken from the platform. Custom counters are added with a standard library, without rewriting logic or stopping services — we roll them out with the nearest release.
How do we separate access by roles?
Permissions are granted at the level of subscription, resource group, and specific workspace: a developer sees their service's logs, the on-call engineer sees only alerts, and audit does not read application data. Accounts are created through the existing directory.
Let's discuss the task and come back with a plan
A conversation about monitoring is substantive when the current state of the infrastructure is visible before it. You do not need to prepare a report — what you already have at hand is enough: schemas, lists, failure history.
Gather this before discussing the task, and we will come back not with clarifying questions, but with a work plan and timeline estimate.
- List of services: what runs, who owns each resource, which components already emit metrics to Azure Monitor, and which remain a "black box."
- Critical scenarios: which chains (database, application, external interface) break first, what counts as unacceptable downtime, and what can wait until morning.
- Current notification methods: where notifications arrive, who reads them, at what thresholds they trigger, and how often they turn out to be false.
- Dependency map between services and known failure points — even a hand-drawn list of connections will do.
- History of past failures: how the problem was noticed, how long recovery took, what was missing for fast diagnostics.
- Constraints: data storage requirements, access roles, what cannot be changed without approval, and in which windows work is allowed.
With this set, we will determine which metrics and logs to collect, how to separate thresholds by criticality, where to route notifications, and how long to store history.
Describe the task — we will come back with a work plan, timeline estimate, and a clear area of responsibility at each stage.