Setting Up MTTR: Why Report Numbers Diverge from Reality
Incidents seem to be closed, reports are submitted, but the recovery time in them doesn't add up. The on-call engineer counts from the first monitoring signal, the manager counts from the user's message to support, and the tracker shows the task closure date. Three numbers about the same failure—and none of them is suitable for deciding priorities.
The root cause is that the incident policy does not fix the start and stop points of the count.
Does recovery time include the on-call engineer's confirmation of the signal, diagnostics, approval of access to the production environment, rollout of the fix—or are only the minutes of service unavailability counted?
As long as the answer stays in the on-call engineer's head, recovery time tracking remains an agreement between people, not a metric.
Hence the metric's unreliability: month to month, the numbers are incomparable, no trend can be built, and no one evaluates losses from downtime. Improvements that would eliminate an entire class of failures lose out to tasks that were louder at the planning meeting.
A post-incident review without a single timeline also degenerates: people discuss who was on call instead of understanding why the failure reached the user.
We examine how this is closed in practice: from choosing a service level indicator and counting rule to incident labeling and a dashboard that both on-call engineers and the department head trust.
What Changes When Recovery Is Measured Accurately
While MTTR is counted from "noticed" to "closed ticket," the metric mixes four different processes: detection, confirmation, diagnostics, and recovery.
Broken down into separate SLIs—time to alert trigger, time to on-call engineer start, time to service restoration—they give each stage its own owner and its own point of improvement.
Without such decomposition, "speed up recovery" remains a slogan, and the team optimizes not the place that actually slows things down.
When SLIs are captured automatically from alert statuses and request traces, it usually turns out that most of the downtime goes to waiting for confirmation, not to fixing.
Then work proceeds through specific levers: routing the alert directly to the on-call engineer, runbooks for typical failure classes, automatic release rollback.
Recovery time decreases, but more importantly, its spread narrows, and service restoration timelines become a predictable quantity rather than a hope.
Linking an incident to a specific change—by commit, rollout time, configuration version—directly reduces the share of repeat failures.
The review stops ending with a restart: it includes the fix, a test that reproduces the failure, and verification that this class of failure no longer occurs.
On the Grafana dashboard, next to MTTR sit the share of changes that led to an incident and the error budget for each service. From this follow directly the priorities for improvements: we fix what contributes most to total unavailability time, not what was discussed loudest.
At the same time, the real cost of each rollout becomes visible—planning change timelines stops being guesswork and relies on history, not feelings.
Methods for Tracking Downtime: From Manual Logs to End-to-End Signals
The tracking method determines what actually makes it into the metric. If incident boundaries are reconstructed from memory—in the incident log, during a review, retroactively—downtime is understated, and some failures do not make it into tracking at all.
The earlier the start is recorded by a system rather than a person, the higher the measurement accuracy and the fewer disputes about the number.
Options in Order of Increasing Accuracy
| Option | When it fits | Limitations |
|---|---|---|
| Manual incident log | Several services, on-call duty without automated checks | Start and end are written from memory; missed failures are invisible; MTTR depends on the on-call engineer's discipline |
| Service labeling in the tracker | There is a single point for registering tickets | The fact of a failure is recorded, but not its duration; without a service label, downtime cannot be summarized by owner |
| Collecting signals from monitoring | Availability checks and notifications are configured | The start is captured automatically; the end remains manual without recovery confirmation |
| End-to-end tracking: signals, tracker, dashboard | Several teams with shared availability commitments | Requires service labeling and rule maintenance; an error in labels distorts the summary more than manual entry |
The mode is chosen based on infrastructure maturity, not on its name. As long as there is no service labeling, an end-to-end dashboard will show a beautiful but incorrect picture, and incident review will turn into a dispute about the data source.
A practical transition order: first unified labeling and automatic start capture, then recovery confirmation from the service itself, and only after that a summary where signals converge on a single panel.
Setup Stages: From Incident Audit to Handover of the Policy
Setting up MTTR is not installing a monitoring panel but assembling a closed loop: failure signal, interval registration, root cause analysis. Each project stage ends with an artifact that can be verified.
-
Incident audit. We pull up on-call logs and alert history: where start and recovery times were recorded, and where they were reconstructed from memory. The output is a list of data sources and gaps.
-
Tracking scheme. We define what counts as the start of a failure and what counts as recovery, and on whose command the timer stops. Without a shared scheme, the metric is incomparable across services.
-
Service labeling. For each service, we assign an owner, criticality, and threshold for inclusion in tracking. Critical paths are labeled separately; otherwise the average MTTR is diluted by minor failures.
-
Data collection and panel. We link events from the alerting system and the ticketing system by incident ID, and output breakdowns: by service, failure class, and quarterly dynamics.
-
Test failure validation. We trigger a controlled failure in the test environment, walk through the path from alert trigger to closure, and compare the time on the panel. Discrepancies are corrected before acceptance.
-
Work acceptance. We compare the automated calculation with a manual review of several real incidents, record the acceptable deviation, and hand over metric descriptions and access.
-
On-call policy. We describe who records events, how an incident is escalated, and how the metric is used in reviews—otherwise the loop will fall apart in the first weeks.
Set of Documents and Settings That Remain in Your Environment
The set is assembled so it works without us: artifacts are stored in your repository, versioned alongside service code, and changed via the usual edit process—through review, not from the on-call engineer's memory.
- Flow diagram. Where metrics, logs, and events come from, which nodes emit what, and where data is lost during a failure—the topology is readable without follow-up questions.
- Labeling rules. A unified set of labels: service, environment, criticality, owning team. Without it, a breakdown for a specific service in the monitoring panel cannot be assembled, and recovery time is calculated across the entire environment.
- Collection configurations. Description of collection targets, aggregation rules, and retention periods—as files, not settings on a single server that only one person knows about.
- Monitoring panel in Grafana. Recovery time dynamics, share of failures closed automatically, breakdown by service and cause class—with variable templates so everyone can view their own slice.
- Response policy. Severity levels, target response time for each, the decision point for rolling back a version, and the procedure for notifying adjacent teams.
- On-call engineer instructions. Step-by-step review of typical failures: what to check first, where to look, to whom to hand off, and how to record an incident so it enters the statistics.
- Responsible roles. Who owns the service, who edits the labeling rules, who approves changes to the collection scheme, and who reviews recurring failures.
- Knowledge transfer. Procedure for onboarding a new on-call engineer: walkthrough of the scheme using a live example, a control incident under a mentor's supervision, and a readiness checklist.
After handover, we conduct a review together with your team: we adjust wording in the instructions based on the on-call engineers' actual questions—this way the documents remain working, not formal.
Case Study: How Recovery Time in a Payment Service Was Reduced Threefold
How It Was Counted Before Changes
In the payment service, an incident was closed when "payments go through again," and recovery time was estimated from the chat in the on-call channel.
The average consistently exceeded an hour, and more than half went not to fixing but to finding the culprit—the relevant release and dependency were determined manually.
Repeat failures occurred almost every month: the cause was found, but verification of the fix was not completed, so conclusions did not accumulate.
What Was Changed
First, we fixed the indicators: share of successful payments, response time on the critical path, and errors by type.
They were tied to service and release version labels, and incident start and completion began to be marked automatically—from alert trigger to closure, without manual entry.
On the Grafana dashboard, recovery time is broken down by phase: detection, diagnostics, rollback or fix, and result verification. This is exactly what showed where minutes are lost.
What Resulted
Average recovery was reduced by approximately a factor of three, and repeat failures in the quarter were isolated. Incident reviews follow a single template: timeline, root cause, change with owner and deadline, verification after two weeks.
The dynamics of the indicators are visible on the same dashboard, so the argument "did it get better or not" no longer arises. This implementation case rests on a simple principle: as long as recovery time is counted manually, it cannot be managed.
How Does MTTR Differ from Detection Time?
Detection time and MTTR are two consecutive segments of one incident; their sum gives the total service unavailability time.
The first is counted from the system's deviation from normal to the first notification of the on-call engineer; the second is from notification to the metric returning to the target range. The sum can be managed only separately: these segments have different natures and different levers.
Detection time is affected by the alert threshold and the data sampling frequency.
A wide aggregation window smooths noise but delays the signal: until the value crosses the threshold and remains there for the required number of checks, the on-call engineer does not know about the problem.
Symptom-based signals—user response delay rather than node load—together with a compressed window reduce this segment severalfold. The flip side: too sensitive a threshold turns a rare failure into a stream of false alarms.
MTTR is reduced by other means: prepared rollback, canary rollout, automatic failover, and well-rehearsed first-minute actions. What matters here is not monitoring sensitivity but the speed of diagnostics and the reversibility of changes. Mixing these efforts is pointless—halving MTTR with long detection barely affects availability.
Therefore, separate tracking is needed: record three marks—start of deviation, moment of notification, and moment of recovery. The difference between the first and second gives detection time; between the second and third, MTTR.
Without this, optimization strikes blindly: it is unclear whether to fix monitoring or the recovery procedure.
The boundaries of an incident are agreed upon separately. Recovery is considered the return of metrics to normal, not the on-call engineer closing the ticket; otherwise, MTTR begins to measure form-filling discipline rather than service operation.
Technical Concerns: What Is Most Often Clarified Before Starting
Before work starts, engineers and managers ask a similar set of questions: where the metric boundaries are, what will be required from the team, and how long it will take. Below are short answers so the first conversation does not drift into basic clarifications.
Downtime or response time?
We count from the first signal of the monitoring system to confirmed recovery. Time to detection is tracked as a separate metric—otherwise the number "improves" simply because the failure is learned about later.
Will the team start gaming the metric?
The risk is real, so we look not at a single number but at a set: share of failures found by automation, share of recoveries without manual intervention, and breakdown by failure class. The metric is tied to the service, not to people.
What data and access are needed?
Exports from the incident tracking system, monitoring logs, and deployment logs. A service account with read permission is sufficient: we do not touch records or production.
How long does implementation take?
The first version of end-to-end collection and the dashboard takes two to four weeks if exports are available. Refinement of failure classes and thresholds comes as the next step, as statistics accumulate.
Some failures are recorded manually—what then?
We start with what exists: manual records are entered into a common registry using a template, then we automate further. Next to it, we show the coverage share so incomplete data does not look complete.
Let's Discuss the Task: What Data We'll Start With
The accuracy of MTTR measurement is determined not by the dashboard but by where events come from. Before building graphs, three sources need to be brought together: availability metrics, the incident log, and on-call shift marks.
If at least one is maintained manually or diverges from the others, the calculation has to be calibrated—that is the first step of the work.
To get back with an assessment and plan, not general words, we need access to your environment and a small set of data:
- read access to the monitoring system and metric storage;
- incident log for several months—with recorded start and closure times;
- list of services and their boundaries by which availability is calculated;
- on-call policy: who opens and closes an incident and how;
- examples of 5-10 real failures where recovery time is questionable;
- contact of an engineer who will confirm that the incident definition is correct;
- list of manual operations during recovery that are currently not recorded anywhere.
With this set, we will describe the scope of work: where data is already suitable, where normalization is needed, which calculation rules and panels will appear at the first stage, and what is logical to move to the second.
Describe the task in free form—we will return with an assessment and plan, timelines, and responsibility boundaries.