Why Cloud Monitoring in GCP doesn't show the cause of failure
The typical picture: charts in Cloud Monitoring are full, alerts fire, but there is no answer to the question “why are requests failing for some users.” Metrics are collected by instances and resource types, logs are stored separately, and correlating a latency spike with a specific request, build version, or node has to be done manually, switching between windows.
The cause is not a lack of tools, but the fragmentation of sources. Data comes from multiple projects and environments, metrics have no common label schema — service, environment, version — so aggregates mix prod and staging.
Logs are written in free format and without a request ID, which is why infrastructure observability gets stuck in manual text search.
A separate monitoring bottleneck is formal thresholds. An alert on CPU load or free memory fires when the problem is already visible to users, and says nothing about which processing path degraded.
Failure diagnosis requires request-level metrics: error rates, percentile latencies, connection pool saturation — otherwise there is a signal but no cause behind it.
We break down how this is closed: a unified label schema, linking metrics, logs, and traces by request ID, symptom-based alerts with a clear owner, and a dashboard that answers in a minute what exactly broke.
What end-to-end observability provides: metrics, logs, alerts
In Google Cloud, metrics, logs, and alerts live in one loop: for a number of metrics, log entries, and alerting policies share a common set of labels — service, environment, version.
Therefore, the state of an individual service is assembled with one filter, rather than by switching between disparate dashboards. Labels are inherited from the resource, so a new instance falls under monitoring automatically, without manually editing dashboards.
The key link is correlating logs with metrics. The chart answers “what broke”: latency increased, success rate dropped.
Then, in one transition, logs open for the same interval and the same filter — and “why” becomes visible: timeouts to an external dependency, parsing errors, write failures.
Alerts are built by service, not by an individual machine. A rule is formulated at the level of a quality indicator — availability, latency, error rate — and fires when the service goal as a whole is burning.
This removes most false calls: a planned restart, autoscaling, and a brief spike no longer page the on-call engineer.
End-to-end observability is measured by failure detection time. When a metric, log, and alert are tied to one service, diagnosis starts from ready context, not from searching across hosts.
In practice, this shifts detection from hours to minutes and shortens triage: you immediately see which service is affected and at which version.
Metric collection modes: agent, built-in tools, custom collector
The three modes differ not in convenience, but in who is responsible for delivering telemetry.
Built-in Cloud Monitoring tools take indicators from managed services without installing anything on nodes, an agent adds depth for the operating system and processes, and a custom collector gives full control over the composition and routing of data.
The choice is determined not by “which tool is better,” but by which limitations you are ready to consciously accept.
| Option | When it fits | Limitations |
|---|---|---|
| Built-in tools | Managed services, load balancers, databases, container platforms | The metric set is fixed, scrape frequency cannot be set, no process depth |
| Node agent | Virtual machines, need a slice by disk, memory, process queues | Requires uniform installation and updates, its own labels, delays when delivery fails |
| Custom collector | Non-standard sources, multiple environments, metric transformation on input | Operation and updates on your side, higher entry threshold, failure control needed |
What limits collection
The main limitation for all three options is the same — label cardinality. An extra dimension like a request ID turns ten series into millions and makes historical queries slow.
Therefore, a custom collector is installed more often not for collection itself, but for filtering and renaming labels on input, and log collection is run in a separate stream with its own retention rules.
Implementation order: from audit to handover of procedures
The implementation stages proceed sequentially: each next step relies on the artifacts of the previous one. Skip inventory — and the data collection schema turns into a set of guesses, while alert thresholds turn into noise that the on-call engineer stops reading. Below is the working order and the result of each step.
- Observability audit. Collect a list of services, environments, and owners: where metrics are enabled by default, where logs are written nowhere, which alerts are duplicated, and which are silent. The result is a gap map with priorities.
- Data collection schema. Decide what we collect with standard metrics, what we supplement from logs, and which records we exclude so storage does not grow on garbage. Fix a unified label dictionary and access boundaries by project.
- Setting up collection and dashboards. Deploy agents in Google Cloud, configure retention and query policies, and assemble dashboards: an overview by services and a detailed one by nodes.
- Alert thresholds. Build policies from symptoms to causes: first what the user feels, then the resources that led to it. Set severity levels, suppression windows, and routing by on-call schedules.
- Alert testing. Run a training failure: take down a node or inject errors into the log and see whether the policy fired, reached the on-call engineer, and in what time. Fix false positives before handover.
- Handover of procedures. Provide documentation: what is configured, why the thresholds are exactly these, how to connect a new service, and where to go when an alert fires.
Handover artifacts: configurations, schemas, on-call procedures
After the work is handed over, the customer’s cloud platform retains not a set of screenshots from the console, but a working observability configuration: it continues collecting metrics and logs without our access. Below is what exactly is handed over by act and how it is checked at acceptance.
- Monitoring configurations — availability check rules, latency and error-rate thresholds, resource grouping, and aggregation conditions. Stored as code in your repository, not only in the web console.
- Data flow schemas — where metrics and logs come from, through which links they pass, where they are stored and after how long they are evicted, with retention periods by each type.
- Access policies for metrics — roles for on-call engineers, developers, and contractors, separation of rights to read and change rules, service accounts instead of personal ones.
- On-call procedures — criticality levels, to whom and in what time a notification goes, what we consider a false positive, and when a rule is rewritten.
- Maintenance documentation — how to add a new check, how to roll back a change, and what to do if the system goes silent and stops sending data.
- Instructions for frequent incidents — step-by-step scenarios for 4-6 typical failures: what to look at and how to confirm recovery.
- Ownership map — who is responsible for each rule and notification channel, when thresholds are reviewed, and how they are transferred when the responsible person changes.
Acceptance goes by checklist: each rule is checked by forced firing in a test environment, and you make the change to the configuration yourself — if a rule cannot be changed without us, it is a dependency, not a handover.
Case study: reducing failure detection time from hours to minutes
At the customer, metrics were collected but lived separately: some services exposed their monitoring dashboards, some only text logs. Alerts were configured on technical signs — CPU load, disk filling, a single node going down.
The team learned about degradation from support tickets: 3-4 hours from the first user errors to the incident being noticed.
Triage got stuck in the data: the picture was assembled manually from log fragments in different places, and this took up to two days. Seven out of ten firings turned out to be noise, so alerts stopped being trusted — and that is more dangerous than their absence.
What changed
Metrics, logs, and traces were brought into a unified observability loop on Cloud Monitoring and Cloud Logging, with common labels for service, environment, and release version.
Rules were rewritten on the signs the user feels: error rate, response time, availability of critical scenarios. Thresholds were set by rate of change, not by absolute value, related alerts are grouped, and repeats are suppressed by a silence window.
Metrics before and after
Incident detection time — from 3-4 hours to 5-7 minutes, false positive rate — from 70% to 12-15%. The share of incidents the team learned about before users grew from 20% to 85%.
Incident triage was reduced from two days to 30-50 minutes: the alert links to the needed log slice and metric charts for the degradation period, and the engineer immediately sees which release and which node became the source.
This is the practical value of the observability case: not dashboards for dashboards’ sake, but predictable response time.
How agent-based metric collection differs from built-in tools
The question is usually posed as a fork: install a metrics collection agent or get by with built-in collection tools. In practice, it is not choosing one of the two — the telemetry layers are simply different.
Built-in tools see the platform level: instance state, disks, network, load balancers, cluster events. The infrastructure itself collects them, and we have nothing to maintain — metrics appear with the resource and are immediately tied to it.
A metrics collection agent is needed where the application begins. It is a process on a host or next to a container: it takes indicators from the service itself and its collectors and sends them to the same storage as platform ones.
Only this way do queue length, external dependency response time, error rate on a specific handler, and labels with build version and environment appear.
The price for this is collection overhead: memory and CPU time on each node, agent updates, and configurations that must be versioned with the application.
Hybrid collection is the working solution: platform-level is taken built-in, application-level — by an agent, and at the junction we agree on a single source of truth for each metric.
Without this, alert rules duplicate and labeling diverges, and metrics cannot be matched in one query.
Service coverage is conveniently laid out by layers. Managed services are covered by built-in tools, own applications — by an agent, external dependencies — by separate availability checks.
If metrics are needed only for infrastructure, an agent adds maintenance without benefit: costs exist, diagnosis does not change.
Hence the criterion: an agent is justified when without it the cause of degradation is not visible. Platform metrics will show that the service responds slower, but not at what stage it does so. We analyze where this boundary lies for you and build a collection schema for it.
Limitations and technical concerns before starting
Concerns before starting are usually engineering: will metric collection fit the current landscape, will the log stream grow too much, will service account permissions suffice, and when will the efforts pay off. Below are four questions that most often slow the transition to observability in Google Cloud.
Infrastructure is heterogeneous — will the collector install on everything?
The collection agent installs both on virtual machines and in a cluster, and application metrics are accepted in an open exposition format. Existing exporters and dashboards are reused, and off-cloud environments are connected via relay.
Will logs take up all storage?
The stream is filtered on the agent side: debug levels and availability checks are cut before writing.
The rest is laid out in buckets with different retention — short hot and long archive, and registry volume drops several times without loss of incident triage for the past period.
What permissions to grant and won't we expose too much?
For the service account — only write metrics and logs, read by separate roles for on-call engineers and analysts, and data access is included in the audit log. We do not store long-lived keys, and we grant access with temporary accounts.
When will efforts pay off?
The first set of dashboards and alerts for key services works in the first weeks after connection. The result measure is the share of failures that the on-call engineer sees and closes before the user contacts support.
Let's discuss the task: what data is needed for a work plan
To make the estimate concrete, not “plus-minus,” we need source data on the infrastructure and observability requirements. Send what already exists — even a partial set shortens the first conversation and removes extra iterations with clarifications. Below is a list of what really affects the work plan.
- List of services and environments: what runs in development, test, and production environments, which processes are critical, and which can be left uncovered.
- Current monitoring state: existing dashboards, firing rules, thresholds, and how much they can be trusted in incident triage.
- Expectations for signals: which failures need to be noticed before the user, which values are considered normal, and which are deviation.
- Log composition and storage requirements: retention, approximate volumes, whether export is needed for incident triage and checks.
- Access: accounts with read-only rights, roles, and project restrictions — this is enough for triage, and nothing will need changing.
- Notification channels: where notifications should arrive and who is on call, so the signal reaches the responsible person, not a general chat.
It is more convenient to discuss the task with the engineer who will run the project: the conversation takes 40–60 minutes, we analyze the schema, clarify boundaries, and answer technical questions without general words.
After it, we will return with a work plan: which metrics and logs we collect, how dashboards and alert rules are structured, what stages implementation consists of, and by what criteria we accept the result. The next step — describe the task in free form, and we will return with an estimate and plan.