Setting up Change Failure Rate measurement (share of failed changes)

Change Failure Rate: why release failures are invisible to the team

The team deploys several times a day: the pipeline is green, checks have passed, the artifact is delivered.

Delivery has become routine, and that is precisely why the question of how many releases had to be rolled back or urgently fixed within a day catches everyone off guard — neither the on-call engineer nor the department head has a clear number.

The reason lies in how accounting is set up. A rollback lives in the on-call engineer's terminal, an incident lives in the alerting system, and a manual fix on a server is not recorded anywhere at all.

The connection between a release and a subsequent failure is reconstructed from memory during an incident postmortem, when finding someone to blame is more important than counting.

The share of failed changes exists only as a feeling that "there seems to be more of them."

Change Failure Rate turns this feeling into a number: the ratio of rollbacks and emergency fixes to the total number of releases over a period.

The failure criterion is defined formally — reverting to the previous build, releasing a fix outside the plan, changing configuration bypassing the process. Release quality becomes visible on a monitoring dashboard every day, not only after a major failure.

The main difficulty is tagging. The release history must retain a marker: which artifact went out, who confirmed success, what happened in the observation window after deployment.

Then release rollback accounting stops depending on the on-call engineer's diligence, and the causes of rollbacks group into categories — configuration, migrations, external dependencies, lack of checks on specific paths.

We will discuss how to set up this measurement on an existing release process without adding manual work for engineers.

What regular measurement of failure rate in development gives

A metric by itself does not improve anything — the effect comes from a closed loop: a change goes into a release, a failure is recorded, and the postmortem ends with a fix to the process itself.

The approach traces back to DORA research: teams that measure the share of rollbacks regularly connect cause and effect faster and stop discussing reliability at the level of gut feelings.

The checklist converges into a single picture: if the share of rollbacks stays flat while resolution time grows, the bottleneck is in diagnostics, not in code quality.

If everything grows together, the problem is in the changes themselves and how they are checked before deployment.

Quality feedback compresses from weeks to hours: an engineer sees the consequences of their change in the same work cycle.

At the same time, an entire class of rollbacks goes away — those caused by manual steps during deployment and the absence of checks before release: each such case now gets tracked and receives a specific fix.

Formats for tracking failed changes: comparison of approaches

The share of failed changes can be calculated in different ways, and the first thing that determines the approach is classification.

Until it is fixed what exactly counts as a failed change (a full release rollback, an emergency fix, disabling functionality with a flag, a configuration rollback), each team counts in its own way, and the numbers are incomparable between reports.

The second factor is whether the delivery loop has a single change identifier by which a deployment is linked to a subsequent incident.

There are usually three sources of incident data in automatic collection: deployment events from the delivery loop, monitoring alerts, and tickets from the issue tracking system.

The "deployment → alert → ticket" link provides a connection that can be verified by eye: on-call systems like PagerDuty provide the trigger time, and the tracker provides the fact of the incident and its cause.

In practice, a sustainable metric comes from the hybrid option: a fully manual review diverges by multiples between reviewers, while a fully automatic one catches extra triggers.

How we implement failed change tracking: stages of work

Tracking the share of failed changes (Change Failure Rate) rests on one thing: the metric is only as honest as the concept of "failed change" is unambiguous.

Therefore, the order of work is built from agreeing on the definition to validating against historical data — otherwise, a month later the numbers diverge from reality, and the metric stops being trusted.

1. Agreeing on the definition of a rollback. We fix what we consider a failed change: release reversion, an emergency corrective release, disabling functionality with a flag. We discuss controversial cases with the team and record them in the process.
2. Auditing release history. We compare release logs, tasks in the tracking system, and incident records for the past period. This shows where data already exists, where there are gaps, and what the baseline share of failed changes is.
3. Data collection scheme. We define sources and the linking rule: how a change event is matched to an incident — by tag, by time, or manually.
4. Setting up collection. We introduce a single change identifier that passes through all systems, and configure the calculation of the share and breakdowns by service and team.
5. Validation on past releases. We run the mechanics on historical data and compare it with a manual review of several releases. We explain discrepancies and correct the matching rules until the numbers converge.
6. Process and thresholds. We fix who looks at the metric, what level we consider normal, and what we do when it is exceeded — a review of causes without looking for someone to blame.
7. Handover. We provide the scheme, documentation, and access, and train the on-call engineers; a month later we reconcile the data and adjust when sources change.

Process, dashboard, and release rollback review schemes

The metric lives not in a report but in artifacts that remain in your repository and under your responsibility.

We assemble them so that six months later a new engineer can open the repository and understand where the numbers come from, why a particular release was classified as a failure, and how to change the rule if the definition changes.

• Collection configurations — the rules by which release and incident history is consolidated into a single Change Failure Rate metric (share of failed changes), indicating sources and recalculation frequency.
• Incident classification scheme — what we classify as a release failure, what we consider a rollback, what is planned degradation, and what is an external failure and excluded from calculation.
• Release metrics dashboard — dynamics by week, breakdown by service and team, the plotted threshold at which a release is recognized as failed.
• Rollback review process — who conducts the review, participant composition, timelines for recording the cause, and what counts as a closed review.
• Metric documentation — definition, formula, list of sources, known limitations, and history of changes to the calculation rules.
• Data access rights — roles: who sees incident details and exports, who only sees aggregates, who can change the classification scheme.
• Operating instructions — how to connect a new service, how to correct the calculation rule, and what to do when data diverges from a manual export.

We conduct the first two or three reviews together with your team, then it runs them on its own — that is what the process is written for. Describe the task, and we will come back with an estimate and a plan.

Case study: failure rate dropped threefold after setting up tracking

Before implementing the metric, the picture of failures was assembled from disparate sources: engineers wrote in notes, managers conducted manual surveys once a month, and everyone interpreted the fact of a failure in their own way.

Three teams — three definitions: for one, a rollback within an hour counted as an incident, for another it did not. Arguing about rollback dynamics under such conditions is pointless: the numbers did not even match between adjacent releases.

How tracking was built

First, a single definition of Change Failure Rate was fixed: a failed change is one that, within an agreed window after deployment, required a rollback or emergency fix.

Incident markers were tied to specific releases via GitLab — tags, build numbers, and rollback records ended up in one place. Each release received a short review: what broke, at which check stage it was caught, and what we add to regression.

Implementation results

The share of failed changes over the first quarter of observation fell from 24% to 8% — not because there were fewer releases, but because release reviews began returning defects into the automated test suite.

The time from detection to rollback decreased from several hours to tens of minutes, and the share of releases that went through review grew from a third to all of them. A single quality indicator stopped being a subject of dispute: one metric, one window, one table by releases.

What to do if releases are frequent but there is no rollback tracking

The first thing to fix is the very definition of a failed release. Whether we consider a failed release to be a change that had to be rolled back, or any change after which an incident with a hotfix was opened, these are fundamentally different numbers.

Until the definition is agreed between teams, the metric is incomparable even within a single product: one group records only crashes as rollbacks, another records any defect found in the first day.

Incident history can often be reconstructed retrospectively. The delivery pipeline already retains deployment records with version tags and timestamps, while the tracker has closed incidents with dates.

A retrospective review of releases over several months is built on comparing them: a change is marked as failed if an incident requiring an unscheduled fix appeared within the correlation window after it.

The window is chosen based on team practice — from several hours to a day — and it is better to record the chosen value immediately so the number does not "drift" on the next recalculation.

The minimum data set for such a reconstruction is small: change identifier, time of reaching production, environment, rollback or hotfix marker, link to the associated incident, and recovery duration.

This is enough to obtain both the share of failed changes and the time to recovery — without new systems or storage. Extra fields at the start only slow down collection: no one fills them in manually anyway.

The first step of implementation is not buying a tool, but fixing the definition and a single source of truth for the release marker. Then collection grows around the place where the marker already appears automatically — there is no need to set up anything extra there.

If you start with a manual journal for several teams, within a month you can see which releases pull the metric up and which changes cause them.

Technical doubts when setting up a rollback metric

Questions about the metric arise not around the formula but around boundaries: what counts as a failure, whom to trust in assessment, and whether the team will start hiding failures. Below are the things we most often discuss during setup.

Will teams start hiding rollbacks if the metric goes into reporting?

They will if the indicator is tied to punishment. That is why we calculate from delivery system data rather than self-reporting, and we discuss classes of causes, not specific people.

We separately measure time to detection: a team that quickly catches and rolls back looks better than one that does not acknowledge the problem.

Is there any point in the metric with manual releases?

There is, but accuracy is lower. We record releases in a single journal: composition of the change, who released, outcome, whether there were urgent fixes.

Such a journal provides a stable share of failed changes after just a couple dozen releases and also becomes a list of what to automate first.

What to do with failures for external reasons, and when can the number be trusted?

External causes — provider outage, dependency failure — we mark with a separate attribute and do not include in the main share, but we do not discard them from statistics: they show where redundancy is needed.

We trust the indicator when it reproduces from cycle to cycle and matches a manual sample; a single release proves nothing, a stable trend does.

Let's discuss the rollback measurement task and the implementation plan

A rollback metric cannot be calculated out of the box: first you need to agree on what counts as a failed change specifically for you, where to get incident facts, and how to link them to a specific release.

That is why we start not with a monitoring dashboard but with a conversation about your infrastructure and processes.

On a call, we clarify the initial inputs — that is enough to come back with a work estimate and a metric implementation plan. What is useful to prepare in advance:

• A map of environments and build pipelines: where artifacts appear, how and where deployment goes, which steps are performed manually.
• Definition of failure: what you already consider an incident — release rollback, hotfix, production degradation.
• Sources of facts: issue tracker, incident log, monitoring and alerting systems, deployment history.
• Release process: frequency, windows, who and on what basis makes the rollback decision.
• Responsible parties: who owns the service and who confirms the classification of changes.
• Boundaries of the first loop: which teams and services to include immediately, and which — in the next stage.
• Constraints: access, security policies, data storage and retention requirements.

Then we fix the order of further steps: data collection scheme, classification rules, areas of responsibility, and schedule. Describe the task in a convenient form — we will come back with a work estimate and a metric implementation plan.