Blind database monitoring: degradation is noticed from complaints
A typical picture: database monitoring is configured partially — CPU load and free space are collected, while history on query execution time, locks, and replica lag is not collected at all.
While traffic is steady, this is enough, but as the load on the DBMS grows, database degradation does not come as an outage but as a creeping increase in response time — first by percentages, then severalfold.
Such a bottleneck in the database is not visible until connections start queuing.
Users react first: “the form takes long to save,” “the report won’t open.” The investigation starts from scratch — there is no baseline, and it is unknown what exactly changed: an application release, a new index, a grown table size, or a degraded query execution plan.
The cost of blindness is not only downtime. A database incident noticed from complaints always takes longer to investigate: while the engineer collects state snapshots, the problem either goes away on its own or worsens, and the cause remains unknown until next time.
Below we look at how this is closed: continuous collection of Percona PMM metrics with change history and thresholds that reveal a deviation before a user notices it.
What changes after connecting databases to monitoring
Database monitoring changes not reporting but the moment at which the team learns about a problem. A query starts slowing down, a replica falls behind, disk space runs out — previously this was discovered from complaints and post-mortem analysis. With a unified metrics dashboard, the signal arrives before degradation becomes an outage.
Early detection of degradation instead of analyzing consequences
Database monitoring is built on thresholds tied to the behavior of a specific workload: query execution time, wait queue, log growth, replica lag.
Early alerting triggers on sustained metric growth rather than on the fact of failure — problem detection time drops from hours to minutes.
The second effect is metrics history. Instead of snapshots of “how it was at the moment of the incident,” a continuous series remains: you can see when growth began, what preceded it, and which parameter brought the system back to normal. Incident investigation relies on data, not on witness accounts.
A unified picture across heterogeneous databases
Different DBMSs and different hardware are brought into a single loop: agents collect metrics, the server stores them and exposes them through a common interface.
Our engineers work on an open platform on the level of Percona PMM — it covers common DBMSs, including clusters and replicas, and does not lock you into a single vendor.
The practical result is one screen instead of a dozen consoles.
The on-call engineer sees which database degraded, how much its behavior differs from neighboring ones, and when it started, while alert thresholds remain in your infrastructure and are tuned to your workload pattern.
Methods for collecting metrics: local agent, remote polling, hybrid
The collection scheme determines whether you will see disk degradation or only an increase in query latency. PMM retrieves metrics in two ways — by polling the node over the network or with a collector next to the database — and the set of counters differs in each case.
You should decide before connecting the first instance: changing the scheme after the fact means reworking dashboards and alerting rules.
When remote polling is enough
The server itself contacts the node over the network and retrieves metrics from an open port. It is suitable when the database exposes its state externally, there are few nodes, and access between segments is already open: nothing is installed or updated on the hosts.
You pay with depth — buffer pool state, per-table I/O, and internal queues are only partially visible from outside, and if connectivity is lost, the data simply disappears.
When a local agent on the node is needed
The agent runs next to the database process and reads what is not exposed externally: the planner, table-level I/O, replica lag. It requires maintenance on each node and permissions to the service socket.
In return, an incident is investigated by cause rather than symptom, and metrics are not lost during network failures.
| Option | When it fits | Limitations |
|---|---|---|
| External polling | Fleet of up to ten nodes, stable network, quick start | Less depth, no data when a node is unavailable |
| Local agent | Loaded databases, planner and I/O metrics needed | Agent on every node, socket access |
| Hybrid | Heterogeneous fleet: critical — with agent, peripheral — external | More complex setup, two schemes in one dashboard |
Depth and polling frequency are chosen separately from the scheme: a rare interval saves resources but smooths out spikes, while a frequent one gives an accurate picture and noticeably loads the node.
Implementation procedure: from inventory to handover
Monitoring implementation proceeds sequentially: first we record what exactly exists in the infrastructure, then we connect metrics collection, and only after that do we configure thresholds. The reverse order produces false positives and noise in which real failures are lost.
Connection scheme and permission separation
- Database inventory: we collect a list of instances, roles, replicas, and clusters, mark business-critical and decommissioned ones. A missed node later surfaces during an incident.
- Scheme design: we determine where the monitoring server is located, how it collects metrics from each node, and what happens if the channel breaks.
- Permission separation: we create a separate account with the minimum sufficient permissions — to read metrics and state, without access to the data itself.
- Collection setup: we enable metrics, the slow query log, and execution statistics, and set retention periods so that volume does not grow uncontrollably.
Verification of thresholds and notification channels
- Threshold configuration: we set limits for load, replica lag, locks, and free space, separating critical values and warnings.
- Alert verification: we run scenarios under real load — deliberately take a node down and make sure the signal reached the on-call engineer rather than staying in the dashboard.
- Solution handover: we provide the on-call procedure, instructions for standard responses, and access, and review recent alerts with the customer’s team.
What your team retains after implementation
Implementation is complete not when dashboards with graphs open, but when the team has a set of artifacts by which the system can be brought up, expanded, and handed over to a new engineer without its author.
Everything listed below we deliver to version control and the internal knowledge base — not as “settings on the administrator’s server” that live until the first vacation.
Package of configurations and diagrams
- Monitoring configuration: agent connection files for Percona PMM, environment variables, and the list of collected metrics — a new database is added by copying and editing two parameters.
- Connection diagram: the topology “database — metrics collector — storage” with addresses, ports, and network boundaries; it immediately shows what is accessible externally and what is closed.
- Access policies: service accounts with minimal permissions and role separation — the on-call engineer sees graphs, while changing thresholds is available only to the service owner.
- A set of dashboards for your roles: separate views for the on-call engineer, developer, and manager, so that each opens their own picture rather than a common list of a hundred metrics.
On-call and escalation procedures
- Alerting procedure: which threshold raises what, who receives the signal, what is considered a false positive, and in what order sensitivity is adjusted.
- Database documentation: inventory of instances, versions, location, owners, replication scheme, and failure points.
- Escalation procedure: who receives the signal first, when a senior engineer joins, and what is recorded in the log after the investigation.
Case study: reduced time to find the cause of degradation
A case of monitoring a fleet of several dozen databases: a transactional core, replicas, analytical exports with different workload profiles.
Before implementing Percona PMM, problems were learned about from users, and investigation began only after degradation became noticeable externally.
What it was like before connection
Diagnostics relied on indirect signs: the slow log, operating system counters, manual queries to system tables.
Finding the cause of degradation took from 40 minutes to two hours, and in some cases the cause remained a hypothesis — metrics history was not stored, and by the time of investigation the values had already been overwritten by current ones.
The same class of failures also recurred regularly: unnoticed resource exhaustion — growth of long transactions and connection pool overflow, followed by switching the node to read-only mode.
What changed after
The collector takes metrics from all nodes on a unified schedule, and history is stored with a depth reserve, so there is something to compare current behavior against.
The cause is identified from the graph: you can see which metric started growing first and what grew next — instead of sifting through guesses.
Recovery time dropped from hours to minutes: the engineer opens the dashboard for the required interval and immediately sees the bottleneck.
Before-and-after metrics: the share of incidents detected by alert rather than by complaint grew from isolated cases to the majority; average cause finding — from 40–60 minutes to 5–10 with the same fleet size.
The class of failures with unnoticed resource exhaustion disappeared: thresholds for free space, connections, and transaction duration trigger before the failure becomes visible externally.
How does database monitoring in a cluster differ from node monitoring?
A node and a cluster are different objects of monitoring: the metrics of one do not answer the questions of the other. An agent on the host shows CPU load, free space, and slow queries of a specific machine.
But in a cluster, fault tolerance is determined not by the state of the server but by the relationships between nodes: who is primary, whether the secondaries have caught up with the change stream, and whether role switching occurred without losing transactions.
Without the cluster level, replication lag, desynchronization, and broken links between nodes fall out of view. A node can be completely healthy while the cluster has already degraded: reading from a lagging secondary returns stale data, and role switching rolls back some writes. That is why database monitoring is built on two levels at once.
| Option | When it fits | Limitations |
|---|---|---|
| Node monitoring | Single server, test environment, investigation of an incident on a specific machine | Does not see roles and topology; switching looks like a metrics failure |
| Cluster monitoring | Database cluster with replication and automatic role switching | Requires access to service views and a single collection point |
| Both levels together | Production cluster: both resources and replication logic matter | More complex to configure, metric normalization required |
In practice, the levels are brought into one collector: Percona PMM polls agents on hosts and service views of each node, so the state of the server and the cluster is visible on one graph — replica lag, primary change, diverged counters.
This makes it possible to distinguish cause and effect: delays on a secondary are a lagging replica or a lack of space.
Common concerns before connecting databases to monitoring
Concerns before connecting databases to monitoring usually relate not to the metrics dashboard itself but to consequences: how many resources the collector will take, where the history will reside, what to do with isolated environments, and who is responsible for alerts after handover. Below are answers to each point.
Will metrics collection load production databases?
The collector reads DBMS internal counters and service views; the polling interval is configurable — from frequent to gentle. Before enabling, we take a baseline at peak load and after; if there is a deviation, we increase the interval or selectively disable heavy collectors.
The noticeable contribution comes not from collection itself but from query tracing with execution plans — we introduce that in a dosed manner.
Where is history stored and for how long?
Metrics are written to time-series storage on a separate monitoring server, not on production nodes: detailed data is kept for weeks, rolled-up aggregates — for months.
Volume grows predictably and depends on the number of databases and polling interval; the retention policy is defined by rules.
Databases in an isolated environment, without direct access from the common network
We deploy the monitoring server inside the environment; a single outbound channel with aggregates and notifications goes outside. Direct access to the DBMS is not needed — an account with read-only access to service statistics is sufficient.
We already have third-party monitoring tools deployed
They do not conflict: the collector reads the same counters. Some checks can be left in the current system, while what is missing there can be moved to the new one — so duplication does not arise.
Who is responsible for alerts after handover?
We record thresholds, notification channels, and on-call rules in a document; we hand over configurations and access to your team along with training.
After that, monitoring is performed by you or by us according to an agreed procedure — the responsibility boundary is defined in advance.
Let’s discuss the task and come back with an estimate and plan
To come back with an estimate and work plan rather than general wording, we need a minimum of input about your environment. Below is a list of what is worth attaching to your request.
- List of databases and their roles — which DBMSs are in the environment, where the master is, where the replicas are, and which databases serve only reports and analytics.
- Deployment diagram — hosts, network segments, presence of proxies and load balancers, and how access to servers is organized.
- Current monitoring tools — what already collects metrics, where they are stored, what alerts are configured, and who receives them.
- History retention requirements — retention depth for metrics and query texts, and whether separate detail per database is needed.
- On-call mode — who receives alerts and during what hours, through which channels, and what is considered an incident in your practice.
- Maintenance windows and acceptable interruptions — when metric collectors can be installed and services restarted without consequences.
- Those responsible for access — who issues accounts and approves changes in the configuration of monitored systems.
We will compare the list of databases with the deployment diagram, check how Percona PMM fits your DBMS versions, and come back with a project estimate and work plan. Describe the task — we will discuss the details and propose an implementation procedure.