Percona PMM: why MySQL bottlenecks remain unnoticed
The standard set of metrics — CPU load, free memory, availability checks — almost always looks calm, even when the application is already slowing down.
Database degradation does not start with a server crash, but with individual slow queries: the execution plan changes after a table grows, an index stops being used, and response time grows gradually, without a single obvious failure.
The second blind spot is table locks. A transaction that holds a row longer than usual builds a queue behind it, and at peak this results in second-long delays for everyone accessing the same table.
In terms of server load, such an event is almost invisible: the CPU is free, while users are already writing to support.
Replication lag is the third type of problem. The replica's lag behind the master is initially measured in fractions of a second and does not fall under any alerting rule, but a report collected from the replica starts serving stale data.
This is usually discovered from a complaint: "the numbers are not what they were yesterday."
Percona PMM covers all three scenarios at once. Query Analytics breaks down the load by specific queries and shows which one became heavier over the last day, while alerting rules catch locks and replication lag before the business notices. Below is how such MySQL monitoring works in practice.
What continuous database monitoring provides
Continuous monitoring makes the database observable: Percona PMM collects server and query metrics, stores history, and compares current behavior with the baseline.
The engineer sees not "the site is slow," but a specific query, its execution plan, and the moment of degradation. Below are the effects that the customer notices in the first weeks.
Incident detection time
Alerts on the number of connections, response time, and replica lag arrive before user complaints. The on-call engineer receives a signal within minutes, and the window in which a failure is still preventable grows from hours to minutes.
Root-cause analysis of slow queries
Query history allows comparing behavior before and after a schema change and finding the source of load growth without trial and error. The share of requests resolved by eliminating the root cause, rather than by restarting, grows — repeat incidents for the same query disappear.
Service availability and failure classes
Checks of replication, free space, and table growth catch disk overflow and desynchronization before an accident. Entire classes of failures — connection exhaustion, replica lag, unindexed selections — stop reaching users.
Database load
It becomes clear which reports and background jobs cause the peak, and they are moved to a low-activity window. The load becomes predictable, and node scaling is planned based on facts, not guesswork.
Metric collection modes and deployment approaches
In Percona PMM, the deployment approach determines where the metrics come from and where they are stored.
On a test bench of three databases the difference is unnoticeable, but when there are dozens of databases and some are in closed environments, the choice becomes architectural.
A local monitoring agent is installed on the same host as the database, takes metrics from the process, and sends them to a shared metric storage.
It sees internal counters unavailable from outside, but shares resources with the database, requires rights on the host, and is updated together with the server fleet.
| Option | When it fits | Limitations |
|---|---|---|
| Local agent | Databases in one environment, process-level counters are needed | Shares resources with the database, is updated together with the fleet, requires host access |
| Remote metric collection | Databases in isolated environments, host access is closed | Only external metrics are visible, some causes of degradation remain out of sight |
| Hybrid scheme | Mixed fleet: some databases are open, some are behind a closed perimeter | Two sets of rules, unified normalization of names and labels is needed |
The polling frequency is a trade-off between reaction speed and metric storage volume: a frequent step catches spikes but quickly bloats the history. Usually a less frequent collection is used for queries, and a more frequent one for availability and replication.
Server isolation determines where to place an intermediate node: if the environments cannot see each other, metrics are transferred through one node in one direction, without inbound access to the databases.
Monitoring implementation stages: from audit to handover
MySQL monitoring is implemented not by installing an agent, but by building an observability scheme: first we agree on which deviations count as an incident, and only then do we deploy data collection.
The order of steps matters — if the simulated degradation test is skipped, alerts will fire for the first time during a real accident.
- Survey. We collect the list of instances, replication topology, service owners, and critical queries — we record what we consider normal.
- Observability scheme. We determine which metrics and query texts we collect, where and how long we store history, and what rights we grant the agent.
- Deployment. We install the Percona PMM server as a separate node, and agents on each instance and replica; we verify that data comes from all nodes, not only the primary.
- Query Analytics setup. We group queries by pattern, set the duration threshold, and collect execution plans so that heavy queries, not service noise, end up in the report.
- Alerting setup. Thresholds for replication lag, locks, number of connections, and free space; we route each rule to the on-call channel with clear text.
- Testing alerts on simulated degradation. We artificially create replica lag and a long lock — we check whether the rule fired, how quickly the notification arrived, and whether anything extra fired.
- Handover to operations. We provide dashboards, the on-call regulation, and instructions: what to check when an alert fires and when to escalate.
After handover, the scheme works without us: the team has the rules and thresholds in the repository, query history accumulates, and regression analysis takes minutes instead of hours.
Configurations, monitoring dashboards, and regulations at output
We close out the Percona PMM work not at the moment the dashboards open, but when the team has a set of artifacts according to which the on-call engineer can act without the implementation author.
Everything listed is stored in a version control system alongside the rest of the infrastructure, not in chat and not in one engineer's head.
- Monitoring configuration: agent and server files with comments — what is responsible for metric collection, retention depth, retention periods, and access.
- Alerting rules: thresholds for replication lag, long locks, number of connections, and free space, with notes on what a trigger means and where to look first.
- Dashboard templates: separately by instance, by cluster, and by query stream, with a labeled metric source on each graph.
- Response regulations: who receives the signal at night, what actions are allowed without approval, and at what point the database owner joins.
- Database documentation: inventory of instances, roles and access, list of critical queries, known bottlenecks with change history.
- Instructions for maintaining the monitoring itself: updates, backup of metric storage, and recovery procedure after a failure.
A month after handover, we review together with your team which alerts turned out to be noisy and adjust the thresholds. This way the system remains workable instead of turning into a set of graphs that people stopped looking at.
Case study: eliminated a failure class in replication
What the situation was
The replica under reporting load fell behind by tens of seconds, and at peaks by minutes. Analytics and some user scenarios were read from it, so people saw stale data and reported it before the on-call engineer learned about the problem.
Formally, there was an alert on lag, but the threshold was set on an absolute value, and the rule stayed silent precisely in the area where lag began to grow.
What we changed
We deployed Percona Monitoring and Management, enabled query analysis, and rewrote the rules. The threshold was tied to the replica log parsing rate and to the time of heavy queries — triggering is based on the trend, not on the fact after data has already diverged.
The on-call engineer immediately sees which query is holding the replica and how far behind it is, and goes into analysis with a ready answer instead of a hypothesis.
What the result was
Peak lag decreased from minutes to seconds, and in normal mode the replica stays within a few seconds. The time from the start of lag growth to the on-call engineer's response dropped from hours to a couple of minutes — thanks to this, cases reaching users stopped.
False positives went from a third of alerts to a few per month: the rules became precise, and the on-call engineer trusts them again.
Then we transferred the same approach to the remaining services: the same set of metrics, the same thresholds, and each false positive analyzed as a small rule fix.
Replication lag stopped being a class of failures — now it is a manageable parameter with a clear owner and response regulations.
Percona PMM or custom monitoring: what to choose
Short answer: for typical operation of MySQL and MariaDB, a ready-made solution closes the task in hours.
The agent is installed on the host, starts collecting metrics, the slow query log, and execution plans, and query analysis immediately shows which requests consume server time.
Custom monitoring is justified when metrics are needed from non-standard sources or your own alerting logic is required on top of an existing stack.
The difference shows not at the start, but in maintenance. With a ready-made solution, updates, compatibility with new branches, and standard dashboards come from the community — the team analyzes anomalies instead of maintaining the tool itself.
A custom setup gives full control, but you handle updates, reliability of metric storage, rotation, and access rights yourself.
| Option | When it fits | What limits it |
|---|---|---|
| Ready-made solution (Percona PMM) | Need quick coverage: metrics, slow query log, and query analysis across all nodes | Non-standard sources and custom alerting logic require refinement |
| Custom monitoring on exporters and your own dashboards | There is an engineering team and requirements not present in ready-made templates | Maintenance is entirely on you: updates, metric storage, alerting rules |
| Hybrid | Base from a ready-made solution plus your own application and business-logic metrics | Two systems need to be synchronized by metric names and thresholds |
Practical criterion: if you need observability of the whole database and quickly — take the ready-made one; if the metrics are unique and there is someone to maintain them — build your own.
Calculate the cost of support over a one-year horizon: with a ready-made solution it is almost linear in the number of nodes, while with a custom one it grows with every infrastructure change.
Technical doubts before implementation
Before starting implementation, the same questions come up: will the production database sag, what to do with servers outside the perimeter, and will the on-call engineer drown in notifications. We work with Percona PMM, so we answer according to this scheme.
Will the agent slow down the production database?
Basic metric collection runs with minimal load. Full query analysis is enabled selectively — only on databases that already raise questions, and with a configurable interval.
What rights does the agent need?
A separate account with read-only access to metrics and performance statistics. Rights to modify data and schema are not needed; secrets are stored in a secure vault, not in open configuration files.
What to do with databases outside the perimeter?
The agent itself opens an outbound connection to the monitoring server; inbound ports on production hosts remain closed. Hosts without an external channel are connected through an intermediate relay node.
Will we drown in false positives?
We derive thresholds from the accumulated norm of a specific database, not from general templates. Rules have a wait window and repeat suppression — the on-call engineer sees only events that require action.
What about the metrics we already collect?
We do not break existing dashboards and alerts: data is sent to the current visualization system, and duplicate collection is disabled. On-call engineers stay in their familiar dashboards.
Who maintains the system after handover?
Configurations, alerting rules, the flow diagram, and response regulations remain with you. We train on-call engineers to read dashboards and analyze triggers — then by agreement.
Let's discuss your monitoring task
To come back with a work estimate and an implementation plan, rather than with general questions, we need a minimal snapshot of your infrastructure.
Usually a description of the server fleet composition and the current monitoring scheme is enough — after that we ask specific clarifications on controversial points ourselves.
- Server fleet composition: how many MySQL instances and compatible builds, their roles (source, replicas, intermediate nodes), and where they are located — cloud, own racks, containers, provider-managed services.
- Current scheme: what is already collected, by which agents, where metrics are written and how long they are stored, and whether there are ready-made dashboards for the on-call shift.
- What hurts now: where you learn about a problem — from complaints, the slow query log, or an alert — and how much time passes from degradation to response.
- List of queries and metrics that need alerts: critical business operations, thresholds for replication delays, locks, connection exhaustion, and table growth.
- Access: service accounts with read-only rights for diagnostics, network rules between the metric collection node and databases, and security service restrictions.
- Where notifications should arrive and who is on call: chat, email, ticketing system, and separation of night and day shifts.
- Environment limitations: ban on outbound connections, requirements for local storage of telemetry data, and change regulations in the production environment.
Send this in free form — we do not need a perfect document. We will come back with a work estimate, a phased implementation plan with checkpoints, and a list of risks that we see in your scheme.
If something is missing, we will ask clarifying questions before the start so as not to redo the configuration on a live environment.