AI unit test generation: automating coverage and regression

Why unit test generation breaks on real code

Unit test generation looks like a simple counter: the model went through modules, added files, test coverage increased. A month later, the suite stops catching errors — edits in one place bring down dozens of checks, and no one notices real regressions in production.

The reason is not generation itself, but that tests were written for a metric, not for system behavior.

Coverage for the sake of a number produces checks like “call a method and make sure it did not fail.” Such a test is green under any logic because the expected behavior is not captured in it.

Metrics dashboards such as SonarQube do not show this: the share of covered lines stays high, while the branches with errors and boundary conditions remain uncovered.

The second source of failures is coupling to implementation. Tests generated from the current code repeat its structure: private methods, call order, stubs for every internal component.

Any harmless refactoring brings them down in batches — these are brittle tests that are fixed rather than read.

The third is a mismatch of lifecycles. The suite lives for years, code changes every two weeks, and no one has defined rules for updating checks. Unit test generation without control after it turns into a dump that is more expensive to delete than to maintain. Let's break down how this is closed.

Measurable effect: coverage, regressions, amount of manual work

Measuring the effect without a baseline is meaningless: before enabling generation, we take a snapshot of metrics on the current branch and repeat the snapshot at every release.

We compare not the number of tests, but branch coverage, mutation coverage, and the share of caught regressions that fail before reaching the main branch — these axes are visible in the JaCoCo report and in the mutation operator run.

What shifts in the metrics

The gap between line coverage and branch coverage shows where tests exist formally but decisions are not verified.

Therefore we focus on branches and surviving mutants: generation covers typical paths, and the engineer adds what the model cannot guess — contracts with external systems and failure handling.

How this is visible in the verification pipeline

Generated checks go into the same pipeline as hand-written ones.

The coverage change is compared with the base branch, a drop in mutation coverage blocks the merge, and a test that does not fail when the code under test is reverted is marked as empty and deleted — the suite does not grow with garbage and remains maintainable.

Modes of operation: comparison of approaches and limitations

The generation mode determines the source from which checks are taken: module structure, the diff in a merge request, or the text of requirements.

This affects both coverage completeness and maintenance cost — tests derived from code lock in existing behavior together with its bugs, while those derived from requirements verify scenarios that do not yet exist in the implementation.

Mode selection and environment isolation

In practice, modes are combined: changes-based mode covers everyday edits, specification-based mode covers new functionality, and hybrid mode is reserved for critical paths where the cost of a missed defect is higher than the cost of manual verification.

Each run goes in an isolated environment — its own database, stubs for external services, fixed test data — so checks do not depend on run order or the state of neighboring modules.

We confirm the quality of the tests themselves with a mutation run: if replacing a condition in the code does not bring down a single check, the coverage is formal.

How we build unit test generation into the pipeline

Test generation works only when it is built into the pipeline and relies on a risk map, not on a formal coverage percentage.

The order of stages is fixed: each subsequent stage relies on the result of the previous one, and each has an acceptance criterion — without it, the stage is not closed.

1. Coverage audit. We run existing checks with line and branch coverage, mark uncovered areas, and separate dead code from working scenarios.
2. Risk map. We rank modules by edit frequency, number of accesses, and cost of an error in the production environment — generation goes top-down through this list, not alphabetically by file.
3. Data and environment preparation. We set up an isolated environment: test data, substitutes for external services, fixed time. Without this, checks fail randomly, and the team stops trusting them.
4. Generation rules. We fix conventions: one scenario — one check, explicit fixtures, prohibition of network calls inside a test; the rules live in the repository next to the code.
5. Test drafts. The model suggests drafts, the engineer corrects expectations and removes duplicates — only checks that fail on broken code make it into the suite.
6. Mutation check. We introduce controlled defects and see whether the suite catches them: a test that passes on broken code is deleted as useless.
7. Pipeline integration and handover to the team. Checks run on every change and block the merge if they fail; the team receives the scheme, rules, and walkthrough, then maintains the suite itself.

Stage acceptance criteria

A stage is closed when the suite reproduces in a clean environment, marks real defects, and has passed an engineer's review. We record the share of covered branches and the number of caught mutations before and after — that is the work report.

What remains in the repository after implementation

After implementation, what remains in the repository is not a set of prompts to the model, but a reproducible setup: an engineer runs unit test generation locally and gets the same result as in the shared branch. Below is what exactly we hand over to the team so it can work with this without our involvement.

• Test suite — generated and manually refined cases, organized by module, with source labeling: you can see where the model was used and where a human edited.
• Generation configurations — prompt templates, module selection rules, coverage thresholds, and an exclusion list for legacy code.
• Test data rules — data factories, state isolation between cases, deterministic values, prohibition of real personal data in examples.
• Review regulations — acceptance criteria for a generated test and grounds for rejection: a check without meaning, duplication of an existing case, coupling to implementation details.
• Coverage report — a summary by module with dynamics: where the share of paths covered by checks grew, where it dropped after edits.
• Support documentation — extension points, analysis of typical generation failures, procedure for updating templates when code changes.

What you can edit yourselves

Templates, thresholds, exclusions, and test data are your team's code: change them without approvals. It is enough to run the full suite on the branch — the regression will show whether the new rules broke existing checks.

What we support

Updating configurations for changes in code and library versions, analyzing cases where generation produces a flaky test, and supporting the run infrastructure — so that coverage does not roll back over time.

Implementation practice: coverage growth and regression decline

The client came with a service where business logic had spread across dozens of modules, and tests were written on a leftover basis. Branch coverage stayed around 41%, while some checks contained no assertions and always passed — the number grew, but there was no trust in it.

Regressions surfaced only after rollout, and a noticeable part of the sprint went into analyzing each one.

What we did

We built the generator on source code analysis: it extracts branches and condition boundaries, rather than simply going through methods one by one.

For every coverage gap, a parameterized set is built — working values, boundary values, and empty values — after which the tests go through a mutation check: a surviving mutant means the check proves nothing, and it goes back for revision.

We separately configured run prioritization by changed files so that not everything enters the environment at once.

Project metrics

Branch coverage was measured before and after; regressions were counted by their share in the total number of defects found after release.

The key indicator here is the last row. Coverage by itself is easy to inflate with calls without assertions, so we count only tests that kill mutants as valid. This is what confirmed that the growth in numbers reflects verified logic, not formally executed lines.

As a result, release stopped being a lottery: regressions are caught before rollout, feedback on changes is within ten minutes, and developers no longer write typical scenarios by hand.

How generated tests differ from manual templates

Generated tests are built not on the author's memory, but on code analysis: the generator goes through branches and argument types and creates a check for every fork. Template blanks take a different source — a typical scenario that the team has already written somewhere. Hence the first difference: generation provides coverage, a blank provides repeatability.

The gain is noticeable where a human gets tired: empty collections, boundary and negative values, conflicting states, exceptions in the middle of a chain.

An engineer writes three to five tests per method and stops; the generator brings the set to dozens, and the mutation check shows what share of introduced errors it actually catches. The share of covered branches stops depending on the author's workload.

But there is a class of checks that cannot be derived from code. A semantic scenario — the order of steps in a process, an agreement between services, reproduction of a specific failure — lives in the task description, not in the function syntax.

Manual scenarios remain on critical paths: they answer not “what the method does,” but “what should happen to the order.”

Both types are united by resistance to refactoring. A test coupled to internal calls breaks from renaming and starts getting in the way — it is fixed more often than the code itself.

Therefore checks are built around the observable result and the public contract, and implementation details are removed from them. Then rebuilding a module does not demolish the suite, and regression remains a safety net, not a brake.

Technical doubts: false positives and brittle tests

False positives are the main reason teams disable automated tests. A generator without restrictions produces checks that depend on run order, time, and shared state. Therefore selection is built into the process: a flaky check does not reach the main branch.

How can you tell that a test checks something rather than simply calling code?

Calling a module and comparing nothing is easy — such a test creates the appearance of coverage. For each case, checking the result is mandatory: return value, state change, exception.

We count branch coverage, not line coverage, and run a mutation check: if replacing an operator in the code does not break a single test, there are not enough assertions.

Some of our code is closed — is generation even possible?

Some modules come compiled, without source code — for generation this is not a blocker. Tests are built from public interfaces, data schemas, and exchange contracts; checks capture behavior at system boundaries. Developers of closed modules receive a ready-made suite and run it locally.

What will happen to the tests we have already written?

We do not rewrite the existing suite: it remains the comparison point. First we run generation on unchanged code and see which areas are uncovered — new tests are added alongside, with the same helper functions.

Then the suite is maintained: checks that become outdated when code changes are marked explicitly, not deleted silently.

Let's discuss your coverage and regression task

To come back with an estimate of the scope of work, rather than general wording, we need a short project snapshot. Usually one conversation with the tech lead and access to the code is enough — we clarify the details ourselves afterward.

• Map of critical modules. Where an error in a release costs the most and which scenarios must be covered by tests. We use it to prioritize: payments, access rights, and calculations first, then the rest.
• Current state of the test suite. What is already written, how it is run, how much passes, what is disabled or constantly failing. This is the basis for coverage metrics — we count by modules and execution branches, not by an average number for the project.
• Accesses. Repository, build environment, test data. If data cannot be taken outside the perimeter, we work out an option with depersonalized snapshots on your side.
• Regression regulations. How often you run the suite, who analyzes failures, what you consider blocking for release. This determines whether we embed generation into your environment or keep it separate.
• Acceptance criteria. Coverage threshold, review rules for generated tests, what we consider a false positive. Without them, the metric grows while trust in the suite falls.
• Upcoming releases. What changes are coming and how much code they affect — this directly determines the scope estimate and implementation schedule.

Once we receive this, we come back with an estimate of the scope of work, a stage-by-stage implementation plan, and target coverage metrics — with owners and timelines. Describe the task in two paragraphs; we will clarify the rest with questions.