AI unit test generation: where coverage breaks
Line coverage lies: the report shows 80%, but at release scenarios fail that are not visible in it. The metric counts executed lines, not traversed logic branches — early returns, ternaries, and exception handling blocks remain unchecked. This is the coverage bottleneck: the number grows, but confidence before deployment does not.
A separate weak spot is edits in old code. A change adds one condition, the overall percentage does not move, and the new branch is not checked by anything.
The quality gate on new code in SonarQube calculates coverage on changed lines — there it is immediately visible that automated code coverage lags behind the pace of edits, and regressions at release come from the delta, not from legacy.
The third area is error handling: timeouts, retries, partial failures of external services, empty and invalid responses. Reproducing them manually is slow and inconvenient, so they are most often bypassed — and they are exactly what triggers in production.
The fourth is integration seams: serialization, contracts between services, mapping between layers and storage. Tests inside the module are green, but at the seam the field format diverges or an empty value is lost. Below we break down how AI generation of unit tests addresses these four areas step by step.
What changes after implementation: coverage, review, failures
The first effect is visible on the quality gate for new code.
Generated tests appear in the same change as the logic, so coverage is not “pulled up” by a separate task a month later: the share of covered lines in the change itself grows from a third to 70–80% instead of a blurred repository-wide average.
The metric is visible in SonarQube alongside other build indicators, so code coverage growth is not a report but a measurable condition for the build to pass.
The second is time spent on change review. The reviewer stops manually running obvious scenarios: empty input, range boundaries, external call error — they are already captured by a test and fail on their own.
Discussion shifts to contracts and architecture instead of “why everything collapses on an empty value,” and a typical change fits into tens of minutes instead of hours.
The third is the class of failures that stops reaching staging and production. Regressions after pinpoint edits are caught at the build stage: a check at the module boundary turns red before deployment, not during an on-call shift.
The rollback rate in the release cycle falls, and release stability stops depending on who is on call today.
A caveat without which the effect does not hold: tests must be maintainable.
We generate them for every change, not “for the entire project at once,” do not duplicate overlapping scenarios, and do not tie checks to implementation details — otherwise, after a couple of months they are deleted, and coverage returns to its original level.
Comparison of modes: local run, pipeline, dedicated environment
The three run modes differ not in generation quality but in where the model runs and where the source code goes.
Local run keeps code on the developer’s machine, the build pipeline generates tests for every change, and the dedicated environment is deployed inside your network and serves several repositories at once.
| Option | When suitable | Limitations |
|---|---|---|
| Local test generation | Individual modules, early prototypes, code that cannot be taken outside the perimeter | Speed is limited by workstation resources; no unified ruleset — everyone writes tests differently; the result is poorly reproducible in another environment |
| Generation in build pipeline | Regular regression, large repositories, task of covering already written code | Code goes to an external service or requires network access; on large repositories the run is split into parts, otherwise build time grows; false positives need handling |
| Dedicated environment | Closed-source code, data protection requirements, multiple teams and repositories | Requires deployment and maintenance inside your infrastructure; takes longer to get running; access policy and model update procedure are needed |
The modes do not exclude each other: the working scenario is local generation on the developer’s branch, a run in the build pipeline for regression, and a dedicated environment for restricted-access modules. Limitations are removed not by choosing one option but by separating tasks between them.
Implementation stages: from codebase audit to handover
Implementation of test generation begins not with a tool but with marking up the codebase: where automated checks make sense and where they will hit hidden dependencies and side effects.
The order of work is the same for a monolith and for a set of services — only the audit scope changes.
-
Codebase audit. We mark modules by connectivity, branching density, and presence of explicit contracts. Separately mark code with external calls and global state — there generation requires manual stub setup.
-
Coverage scheme. We set priorities: critical paths, error handlers, boundary values. For each block, fix a target coverage figure so the result is measured, not debated by taste.
-
Configuring generation rules. We define templates, naming style, file structure, and the testing framework used. We specify which dependencies are replaced and which are checked against real data, and what is fundamentally not covered automatically.
-
Validation on a module sample. We run generation on 10–20 representative modules and compare with hand-written tests. We look at the share passing on first run and the time a reviewer spends per generated test.
-
Embedding into the development loop. Generation runs on changed files, tests go through the same review as manual ones, and a failed check blocks branch merge.
-
Documentation handover and training. We hand over rule descriptions, configuration, and analysis of disputed cases. We conduct walkthroughs on real modules so engineers can change generation rules themselves, without our support.
The stage completion criterion is that the team adds a new module to the coverage scheme and gets meaningful tests without contractor involvement.
What remains with the team: configurations, schemes, policies
Unit test generation lives in the repository, not in our account.
After handover, the team retains settings, a coverage map, module selection rules, documentation, and a maintenance policy — without this set, the solution lasts until the first major architecture change; after that, tests become outdated, exceptions accumulate, and the run stops being read.
- Test generator configurations — coverage thresholds by module, exception rules (generated code, migrations, wrappers around external services), run modes. The threshold for an individual module changes without our involvement.
- Module coverage map — for each module: which branches are covered, where only the success scenario is checked, where the gap is. Rebuilt on every run, not manually once a quarter.
- Module selection rules — criteria for inclusion in generation: branching complexity, edit frequency, place in the call chain. Priorities are revised as the code grows.
- Test update policy — what to do when a generated test fails: update the expectation, record a defect, or add it to exceptions. With an owner and procedure.
- Test documentation — running locally and in the build, composition of suites, reading the coverage report. A new engineer enters the loop without messaging us.
- Solution handover — access to the test repository and run settings, walkthrough on a live module, checklist before the first independent change. The team performs the run and edit in our presence.
Case: legacy service — coverage tripled
A legacy calculation service had been in production for about nine years: tests were written sporadically, some diverged from the code and were disabled. Any edit in the old module meant manually running scenarios in staging, and regression surfaced after deployment.
Baseline
Branch coverage stayed at 18%, and JaCoCo calculated it only for modules where tests were preserved at all — the rest remained opaque.
In staging, 18–22 failures per week were consistently recorded, and recovery after an edit in the old module took 4–5 hours, more than half of which went to localizing the cause.
What changed and what we got
First, static analysis ranked modules by edit frequency and connectivity — generation went where the cost of regression is highest.
The model proposed tests based on signatures, return types, and branches; mutation testing filtered out suites that pass on any implementation, and the engineer approved only those that verify the module contract, not its current behavior.
External calls were replaced with recorded responses, so the suite became deterministic and stopped depending on staging.
Branch coverage grew to 56%, the number of failures in staging dropped to 5–7 per week, and recovery time after edits in the old module decreased to 40–50 minutes.
What to do if the codebase is already covered by outdated tests?
Legacy coverage is almost always formal: either the run is red and has been learned to ignore, or it is green but checks implementation details rather than behavior.
Trust in runs falls to zero — regressions surface in production, and changes are rolled out “by eye.” We break down the order for sorting through such a suite.
First, inventory and three buckets: tests that fail under intentional logic breakage; tests tied to implementation — private methods, call order, exact snapshots; and flaky tests depending on environment or run order. The benchmark is not coverage percentage but the share of tests that react to an introduced error.
Next we separate value from scaffolding. Scenarios, test data, and boundary values collected from past defects are carried over — this is the most expensive asset of the suite.
We rewrite the scaffolding: run, data isolation, environment setup — it is because of it that the old suite turns red on unchanged code.
Rebuild order: first stabilize the run — flaky tests go to quarantine with an owner and deadline, but are not silently deleted. Then cover critical paths: access rights, calculations, data migrations. And only then expand coverage by generating tests from current code behavior.
Trust in runs rests on the rule: a red result blocks merge, green means “behavior has not changed.” Therefore tests are deterministic — isolated data, fixed time, no network calls. Then the suite grows by removing duplicates, not by run time.
Common concerns: quality, false positives, maintenance
Concerns before implementation usually come down to four questions: are generated checks formal, will the build start failing out of nowhere, will source code leak, and who is responsible for the suite after the author leaves. Each is checked measurably, not by assurances.
We consider the main metric not the share of covered lines — it is easy to pad with checks that verify nothing — but mutation robustness and the share of flaky runs.
How to make sure checks are reliable and not just “green”?
We run mutation testing: typical defects are introduced into the code — condition inversion, boundary shift, return value substitution — and the share detected is calculated.
Branches with zero sensitivity go to scenario refinement, and a suite that passes intermittently is not shipped.
Where do false positives come from and how do you remove them?
The source of most false failures is nondeterminism: time, random values, collection iteration order, external services. Generators are seeded, time and network are replaced with controlled doubles, data is prepared in an isolated database and rolled back after the run. The check reproduces identically for the developer and in the build environment.
Does code go to external servers?
We deploy inside your environment: the model runs on your capacity, source code and change history do not leave the perimeter, and outbound calls are closed at the network level.
Access is by your roles, artifacts and logs remain with you, and directories that cannot be shown even to a local model are excluded by policy.
What about maintenance when the authors leave the project?
The suite of checks becomes outdated during refactoring and requires an owner. We hand over the policy for updating and deleting checks, the composition of metrics, and control points in the build pipeline — the team maintains the suite itself. Optionally, we remain on maintenance with a fixed response time for failures.
Let’s discuss the task: what we need from you to start
To come back with an estimate and work plan, not general words, we need context about the code and development process. Below are the inputs for evaluation: the more items covered, the more accurate the timeline and the fewer clarifications at the project start.
- Repository or the part planned for coverage: language, framework, structure of modules and entry points.
- What already exists: existing tests, how they are run, current coverage — which logic branches are checked and which are not.
- How the project is built: build command, full test run time, how coverage is measured.
- Access restrictions: whether code fragments can be uploaded to external services or generation happens only inside your environment.
- Requirements for checks: which scenarios are mandatory, where stand-in objects are acceptable instead of real dependencies.
- Acceptance criteria: what coverage share and which modules are considered closed, who confirms this.
- Procedure for working with code: where we make changes, who reviews, whether an automatic check is needed on every merge.
This is enough for us to come back with an estimate and work plan: scope, stages, area of responsibility on your side, and the procedure for launching test generation. Describe the task in free form — we will discuss the details and propose a starting format.
