Why the code coverage metric lies in CI/CD and breaks releases
The code coverage metric looks objective exactly until it is compared with the history of failures. The team keeps the coverage threshold above 80%, the build is green, yet regressions in release arrive consistently.
This is not a paradox: the coverage report shows which lines were executed at least once, but is silent about whether they were tested on different data and with different system states.
The first source of falsehood is uncovered branches despite formal line execution. A line with a condition is counted as covered if execution reached it, while error handling, an empty response from an external service, or a connection failure remain untouched.
The release moves on with a confident “coverage increased,” and then fails on the first abnormal scenario—the very one for which that branch was written.
The second is tests for the number. When the coverage threshold turns into a reporting metric, checks appear that call a function and verify that the call happened, not the behavior.
Formally, test quality grows; in reality, the volume of code that no one has meaningfully verified grows, and the maintenance cost of such a suite exceeds its benefit.
The third is outdated exclusions. A file once landed on the ignore list: generated code, a one-off adapter, a section under migration.
The exclusion lives in the configuration indefinitely, the section changes, and the metric for it is not calculated—there is a hole in the report that no one remembers.
The fourth is the lack of run history. The threshold is checked on every commit, but there is no trend: coverage quietly slides from 84% to 76% over a dozen changes, each of which formally passes the gate.
Below we break down how this is closed: what to measure, where to set thresholds, and how to catch regressions rather than a number.
What coverage management provides: measurable effects
Changed-lines threshold instead of an overall percentage. The check looks only at lines that came in the current changes: if coverage of new sections is below the agreed boundary, the merge stops before review, not after.
Such a threshold does not block work on existing code and does not require heroic efforts to raise the overall percentage—the reduction in regressions is achieved in a targeted way.
Coverage trend as a managed metric. Values are stored by branch and release, so you see not “how much now,” but where coverage growth is moving: which module sagged after a rewrite, where new tests cannot keep up with the code. The blocking decision is made based on dynamics over several weeks, not a one-time snapshot.
Time to investigate a failed run. The report comes with a breakdown by file and a link to uncovered lines right in the build job—there is no need to download an artifact and match it with logs manually.
The alert in the team channel contains the branch, the specific section, and the author of the changes, so investigation takes minutes.
Predictability of merges. The same rules apply to all branches and to a local run, so the result of the check is known even before changes are submitted.
The reviewer discusses logic rather than arguing about coverage, and the merge queue stops depending on who exactly launched the check and with what settings.
Approaches to coverage control: comparison and limitations
Coverage checking rarely rests on a single number. An overall threshold evades itself: a new module without tests dilutes the average percentage, while a drop by tenths of a percent blocks a release for no good reason—the team starts muting the check or lowering the threshold.
Therefore several rules live side by side: a strict threshold on changed lines, a soft overall one, and a trend.
SonarQube can calculate all metrics from a single run, so the choice is not what to measure, but which rule makes the check blocking.
| Approach | When it fits | Weak points |
|---|---|---|
| Overall coverage threshold | Stable baseline, metric changes slowly | A new file without tests dilutes the percentage; the threshold has to be lowered |
| Changed-lines threshold | Active development, frequent edits | Does not see degradation in old code that is not touched |
| Trend control | Large repositories where the threshold has hit the ceiling | Requires accumulated measurement history and reacts with a delay |
| Observation mode | Rolling out a check on existing code | Blocks nothing: regression passes until the rule is switched |
The working combination is a changed-lines threshold as a blocking check, a trend as a signal of slow sliding, and an overall threshold as background. Exclusion rules are formalized explicitly and narrowly: generated code, migrations, configuration. Otherwise exclusions grow and the check loses meaning.
Implementation: from pipeline audit to coverage alerts
Implementing coverage is not installing a tool, but restructuring the pipeline: build, tests, reports, and merge checks begin to work as one system.
The order of work below is arranged so that each stage relies on the data from the previous one—otherwise thresholds turn into a formality that the team disables at the first inconvenient trigger.
- Pipeline audit. We analyze the existing build and test steps, look at which reports are generated and where coverage data is lost between stages. The result is a map: mandatory checks, duplicate runs, sections without report publication.
- Coverage metric collection. We bring all runs to a single scheme: a report in a common format, binding to a specific change and branch, loading into a history store. Coverage stops being output in a log and becomes data for each module.
- Run history. The accumulated base enables comparison: you can see where coverage is truly falling and where it fluctuates because of an unstable test. Based on the trend, we select modules with the greatest gain from targeted test improvement.
- Thresholds and notifications. Two levels—overall across the codebase and for new code in a change, stricter for critical modules. A notification about a drop goes to the team’s working channel with a list of files and the author of the change, so investigation proceeds without searching for the cause.
- Checking on real merges. We run it on actual changes, catch false positives, and set exclusions for generated code, migrations, and the tests themselves. Only after that does the check become blocking.
- Check regulations. We hand over a document: how to change the threshold, who approves an exclusion, what the on-call engineer does when notified. Plus a review with the team—so the rule lives rather than gets disabled.
After the regulations are handed over, the system maintains itself: thresholds change according to procedure, history accumulates, and a coverage drop is visible on the day of the change, not before release.
What we hand over: rules, configurations, regulations
Handover is the moment when the customer’s team changes thresholds and exclusions themselves, without re-learning how the build is structured.
Therefore, in addition to a working pipeline, we provide a set of documents by which a coverage decision can be made without our engineers.
- Coverage rules file—where scopes, report format, and merge points for data from unit and integration runs are declared.
- Exclusion policy—a list of paths and code types removed from calculation, with mandatory justification and a review date for each entry.
- Report template—slices by module, comparison with the previous release, and a trend over recent runs, rather than a single number.
- Notification scheme—to whom and where the signal goes when coverage drops, with separation between warning and merge blocking.
- On-call instructions—the procedure for investigating a drop: check new exclusions, failed tests, and the branch with uncovered sections.
- Threshold review regulations—who raises the bar and on what event, so it does not remain at the level of the first launch.
- Ownership map—which team is responsible for coverage of which module, so an exclusion question goes to the right place, not to us.
We verify the handover simply: on a test branch, we change the threshold and add an exclusion according to the instructions, without involving our engineers. If it worked the first time, the documents are working, not decorative.
Case: thresholds and trends removed regressions from releases
A product team came to us: a dozen services in one repository, releases twice a week, overall coverage around 70%, and complete trust in that number.
The problem was that the average metric did not reflect risk—edits in the payment and billing modules went to production without tests on the lines that changed.
Over a quarter, the team caught three to four regressions per release, and investigation with manual assembly of the previous version took from six to ten hours and required two engineers.
We shifted the emphasis from the overall percentage to changed-lines coverage: in each merge request, the share of added and changed lines covered by tests was calculated separately.
The threshold rule in SonarQube was tied to a mandatory check status—for the payment and billing modules, the build fails if a changed line is not covered; for other branches, the threshold is softer.
Alongside this, we built a weekly trend and alerts for drops, so quality was visible before release, not after an incident.
Metrics before and after
Over three months, changed-lines coverage grew from 46% to 84%, while the overall percentage barely moved—it was not the goal anyway.
Regressions per release became one every five to six releases instead of three to four, and recovery time dropped from six to ten hours to an hour and a half: rollback is launched with one command using the tag of the previous release. The build became two to three minutes more expensive—a tradeoff the team consciously accepted.
How does line and branch code coverage differ?
Line coverage answers the question “will execution reach this line,” while branch coverage answers “are all outcomes of each decision tested.” Line coverage covers a line if at least one of its instructions was executed.
Branch coverage requires passing both sides of a condition—true and false—including ternary operators, switch branches, and short-circuiting in logical expressions.
The gap between the metrics is the source of false coverage. A line with if (a && b) becomes “green” as soon as a turns out to be false: the right side will not execute even once, while the report shows 100%.
Early returns and exception handlers behave the same way—the lines are covered, but the error scenario is not tested.
| Metric | What it shows | Where it lies | Role in the gate |
|---|---|---|---|
| Lines | share of executed lines | short-circuiting, early returns | project trend |
| Branches | all decision outcomes | complex boolean expressions | threshold on changed code |
| Conditions | each operand separately | rarely fully supported | targeted, critical modules |
| Functions | the mere fact of a call | test quality is not visible | signal “module forgotten” |
We take branch coverage as the main metric: tools such as JaCoCo calculate it from the same run as lines, without additional tests. We leave line coverage as an indicator of trend and unused code.
We set the branch threshold only on changed code—otherwise legacy modules block any release—and keep conditions and functions as auxiliary metrics.
Common engineer doubts before setting up coverage
Doubts before introducing thresholds are usually technical, not organizational: engineers have already seen a metric that hindered more than helped. Below are situations that come up on almost every implementation, and how they are solved by configuration, not persuasion.
Will the run take twice as long—does coverage collection slow down tests?
Instrumenting every test is indeed expensive. Metric collection is moved to a dedicated configuration: one run with instrumentation produces the report, while the remaining jobs in the matrix run without it. Dependency caching and parallel executors return build time to its previous level.
Coverage is padded with empty tests—why have a threshold then?
By lines, it is padded; that is why the threshold is calculated on changed lines in a merge request, not across the entire repository. Additionally, the share of tests with assertion checks is recorded: a test without an assertion does not cover a line. Such a pair of metrics is much harder to game.
We have a monolith and old code—the repository will not reach the overall threshold.
An overall threshold is not needed anyway. The baseline records the current state, and the rule “no worse than before” applies to changed files.
Reports are consolidated into one store with binding to a commit, the trend is built by branch, and alerts trigger on a drop relative to the base branch. When the monolith is split, the bar is raised by module, and exclusions for inherited code are set by path with a review date.
Let's discuss the task: how to set up thresholds and notifications
It is impossible to set up thresholds and notifications “by default”: the same coverage percentage means different things for a calculation module and for code that simply proxies an external call.
Therefore we start work by analyzing what is already being collected: which reports are generated, for which branches, where the check is already blocking, and where it is simply ignored.
Then we fix which modules have the most expensive failure—for them the threshold will be stricter than for the rest of the code.
To come back with an estimate and implementation plan rather than general considerations, we usually need:
- access to the pipeline and rights to change the build configuration—we look at real steps, not a retelling;
- coverage reports for recent builds and for the main branch—from them we take a baseline for a threshold that does not fall below the achieved level;
- a list of critical modules and services—for them we set a strict threshold for all code, not only changed code;
- a list of branches and merge conditions—we determine where the check becomes mandatory;
- expectations for the check mode: block the merge or first only warn and collect statistics;
- the contact of the build on-call engineer—to investigate false positives immediately rather than accumulate them in ignore;
- points where the decision is yours: what level we start from and how quickly we raise the bar across repositories.
Describe the task—we will come back with an estimate and work plan.