Turnkey setup of Yandex Monitoring and Cloud Logging

Setting up Yandex Monitoring: why alerts arrive late

Metrics are collected, dashboards open, alerting rules are configured — yet the user is the first to report the failure.

This is the typical picture when Yandex Monitoring is set up formally: metric collection works, but the path from the start of degradation to the signal is measured in tens of minutes, and all this time the failure accumulates in queues, incomplete operations, and errors that customers have already received.

Late failure alerts are a consequence not of data collection but of the rule.

An averaging window of several minutes smooths out the spike, the evaluation period adds as much again, and the threshold is set to an absolute value, without a baseline and daily seasonality of the specific service.

This produces two effects at once: the signal arrives when the counter has already passed the critical mark, and some fluctuations cause false positives — and the notification channel gradually starts to be ignored.

Next are classes of failures that are not visible at all.

Missed service failures usually look like a "green" metric while there is real degradation: the background task queue grows, the share of successful responses on a specific route falls, a handler hangs, one instance out of several responds.

The second reason is that metrics and logs live disconnected: an increase in the number of errors is visible in the records but is not exposed as a counter, and from the log alone it is impossible to distinguish a system failure from isolated messages.

Below we look at how this is addressed: by linking metrics, log records, and rules with a correct window, tagging by services and environments, and escalation that does not leave a nighttime failure without an owner.

What changes after implementation: detection, investigation, on-call

Before implementation, the user notices the failure: they write to support, support looks for the on-call engineer, the on-call engineer opens the console — time to detection is measured in tens of minutes and depends on who noticed first.

After implementation, metrics are collected from applications, nodes, and the load balancer, rules compare them with a baseline, and the degradation signal reaches the on-call engineer within minutes — before requests start failing en masse.

The second effect is noise. Thresholds are calculated by percentiles rather than averages, the rule requires a sustained deviation across several consecutive windows, and duplicates of the same failure are merged into a single incident.

The false positive rate falls from half to a few percent, and the on-call channel stops being ignored.

Service observability is consolidated into one loop: metrics, logs, and traces are linked by a common request ID, and each service has a dashboard with dependencies and latencies.

In Yandex Monitoring lie the metrics and rules, with application and node logs next to them, so the path from a triggered rule to a specific request takes minutes.

Incident investigation from logs stops being archaeology: records are structured, search is done by fields — response code, route, request ID, duration. The share of incidents where the context is ready at the moment of triggering grows from a third to the majority.

On-call becomes predictable: instead of a stream of noise, the engineer receives a description of the problem, a chart, and a link to logs, and after the incident is closed the rule is refined — the next similar failure is caught earlier.

Telemetry collection options: what suits your environment

The telemetry collection method is chosen not by convenience but by where the data can physically be retrieved from.

They get into Yandex Monitoring and Cloud Logging in four ways: an agent from the node, the provider's API, publication from the application itself, and a combination of these methods.

Within one environment, methods are usually combined, so there is no need to choose "one for everything" — it is more important to understand the limitations of each. Below is a comparison by applicability conditions.

Implementation stages: from source audit to handover to operations

Monitoring does not begin with installing an agent — it begins with inventorying what can be collected at all and answering the question of which signals actually mean service degradation and which are just noise. We keep the order of work the same; only the set of sources and collection modes change.

1. Telemetry source audit. We go through services, databases, queues, load balancers, containers, and virtual machines; we record where metrics and logs are already emitted natively, where an exporter is needed, and where collection points will have to be added to the code.
2. Metric and log collection scheme. We separate what is collected by polling and what is sent from the application, which labels are needed for breakdown by service, environment, and node role, and set retention periods and counter aggregation.
3. Deployment of collection and routing. We bring up agents, log receivers, and routing rules: hot search separate from long-term storage, debug streams are cut off at the input so that volume does not grow indefinitely.
4. Alert threshold configuration. We calculate thresholds from the normal load profile, not by eye: errors, latency, disk and memory saturation, unavailability. Next are on-call routes, grouping, and duplicate suppression so that the on-call engineer receives one call per incident.
5. Linking metrics with logs and dashboards. From a chart we move to a specific log window by time and labels — investigation proceeds without manually comparing exports from different systems.
6. Testing with failure scenarios. We shut down a node, fill up a disk, bring down a dependency, inject latency. We check whether the signal fired, reached the right on-call engineer, and whether there was enough data to understand the cause.
7. Handover to operations. Instructions for typical triggers, on-call regulations, rules for reviewing thresholds, and team training.

After handover, the environment runs without us: the customer's team adds new services to the collection scheme independently, following the described labeling and routing rules.

What remains in your environment: artifacts and regulations

Deployment does not end with green checkmarks in the interface but with the customer's team being able to bring up the environment themselves, explain to the on-call engineer why an alert fired, and add a new service to monitoring without our involvement.

Therefore, everything we configure is handed over as files and text in your repository and knowledge base — with a change history and a clear owner for each artifact.

• Collection configurations. Description of metric and log sources, line parsing rules, retention and rotation, mapping of services to projects and environments — as files that live next to the code and go through review like a regular change.
• Alert configurations. Thresholds, averaging windows, grouping of related signals, routing by channels, and duplicate suppression rules: it is clear which condition we consider an incident and which is background.
• Log access policies. Roles by projects and teams, separation of sensitive fields, masking of personal data at ingestion and its retention period — with a rationale for why exactly this visibility boundary was chosen.
• Telemetry flow diagram. From collection point to dashboard and alert recipient, with marked bottlenecks and behavior when an intermediate link is unavailable.
• Dashboards by service. Sets of charts with thresholds, tied to owners: the on-call engineer sees the service state on one screen instead of assembling the picture from ten queries.
• On-call regulations. Schedule, areas of responsibility, escalation procedure, response time by event classes, and the rule by which a signal goes beyond the first line.
• Incident investigation guide. Step-by-step procedure, typical scenarios with ready-made queries to metrics and logs, places to record the finding, and a post-incident review.
• Handover to operations. Access with role separation, acceptance checks for node failure, short training for the on-call shift, and a list of tasks that remain on your side.

All of the above is not an appendix to a report but a working environment: it can be handed over to a new shift, tested in drills, and expanded as services grow.

Describe the task — we will come back with an estimate and a plan for the order in which this is deployed on your infrastructure.

Case study: how we reduced time to failure detection

Before: failures were detected from complaints

The client had a service with a constant stream of requests, but spikes in errors were noticed not by on-call engineers but by users.

The engineer opened logs by the time of the complaint and manually compared records from several services: time to detection reached 40–50 minutes, and the cause often remained unclear because no metrics were collected at all. Long recoveries and repeated failures for the same reason were the norm.

What we built

We set up collection of application and node metrics in Yandex Monitoring, and consolidated logs in Cloud Logging with unified request and trace fields.

The result was a single observability dashboard: share of error responses, latency on slow calls, queue depth, external dependency failures — next to the release chart and load.

We tied alerts to symptoms rather than every unhealthy node, so that the on-call engineer saw one signal instead of an avalanche.

Metrics before and after

Detection — 3–5 minutes instead of 40: the signal comes from the system, not from support. The share of incidents found before user reports grew to about 9 out of 10.

Root cause investigation was reduced from hours to minutes — the trace shows which call and which release caused the drop. Repeated failures almost disappeared: after the first signal, the threshold and alert rule are refined, and the same situation is caught earlier.

How alerting differs from an observability dashboard and what to implement first

An observability dashboard and alerting are different tools, and confusing them is costly. The dashboard answers the question "what is happening and why", while alerting answers "who and when should intervene".

The dashboard is passive: it waits for someone to look at it, and at night there is no one to look at it.

Therefore, the order of implementing observability and telemetry collection does not begin with charts. First metrics and logs — without them there is nothing to compare.

Then 3–5 rules for the most user-visible symptom: growth in error rate, drop in availability, response time above normal. And only after that — dashboards that make incident investigation fast.

Alerting is built on the symptom, not the cause. The user notices an increase in response time, not a full disk — so the rule is set on latency and error rate, with a confirmation window so that a single spike does not wake up the on-call engineer for nothing.

We keep logs next to metrics, in the same Yandex Monitoring environment: from the signal it is immediately clear which node or release became the source. Causes are then investigated using the dashboard.

Both extremes are dangerous. Dashboards without alerts create a feeling of control: the charts are beautiful, but you learn about the failure from the client.

Alerts without dashboards have the opposite effect — the on-call engineer receives the signal but searches for the source blindly, and recovery is delayed.

The readiness criterion for the environment is simple: for every critical scenario there is both a rule and a dashboard, and the check is a drill failure that results in a notification to the right person.

It is worth starting with one service and expanding the environment as the previous one stops making noise.

Technical doubts before starting: we answer directly

Before starting, the same questions come up — the answers determine the scope of work and how the system will behave in six months. Below are short, direct answers.

We already have third-party collection deployed. Do we need to rewrite everything?

No. The working environment remains a source, we connect it to Yandex Monitoring and gradually migrate the check rules. We do not touch the previous collection until it is decommissioned — rollback to it takes minutes.

Where and how long should logs be stored?

The period is set by the retention policy by event type: operational logs are stored for a shorter time, audit and access events — longer. Rotation, compression, and export to cold storage are configured immediately so that volume does not grow uncontrollably.

How to remove noise in alerts?

We calculate the threshold by deviation from the baseline rather than a static one, group related triggers into one notification, and separate them by criticality. The on-call engineer receives a few messages per shift instead of hundreds.

Who supports the system after handover?

We hand over configurations, diagrams, and on-call regulations; rules are described as code and versioned. After that, your team maintains them — or we remain on support under an agreed mode.

Let's discuss the task — we will come back with an estimate and work plan

So that the conversation takes one pass and immediately yields an estimate rather than a series of clarifying questions, it is useful to prepare inputs about the infrastructure.

Below is a pre-start checklist: it covers what actually affects the scope of work for setting up Yandex Monitoring and Cloud Logging.

• List of services and environments that need monitoring: production, pre-production environment, isolated segments.
• What is already collected: metric and log sources, where they are written, what collects them — custom agents or native platform tools.
• Critical scenarios: which failures must wake the on-call engineer before the user writes about them, and by what signs to catch them.
• Access and roles: service accounts, who on the team sees what, whether isolation between departments is needed.
• Storage requirements: lifetime of logs and metrics, whether data must be kept inside the environment.
• Current on-call practice: how alerts arrive and where — email, messenger, on-call phone.

After that, the order is simple. We discuss the observability task: which signals are important, what is already configured, where the bottlenecks are.

We come back with an estimate and work plan — by stages, with the composition of monitoring dashboards, alert rules, and clear timelines. After agreeing on the scope, we start: first basic metric and log collection, then alerts and false positive review.