Why AI test generation does not save coverage by itself
AI test generation confidently produces dozens of checks per module — and that is where the illusion ends.
Unit test coverage grows in the report, but not in protection: new checks duplicate those already written, bypass exception branches, and cling to getters instead of calculation logic.
The number of lines in the suite grows, but the class of failures caught by the run does not change.
Duplicates and false positives are the first sign that no one maintains the generation.
Two tests for the same behavior, but with different names and stubs; a check that passes on any data because the assertion is worded as “the response is not empty”; a test that fails when the order of fields in the response changes.
After a month, the team stops reading the report and runs the suite “just in case” — trust in the result falls faster than the number of checks grows.
Next, the bottlenecks of the test suite become apparent: the generator does not know which branches are critical and covers what is easier, not what breaks for users.
An unsupported suite begins to fail on environment changes — floating dependencies, live network calls, shared setup data. Such a run adds minutes to the build and zero signal to the release.
Generation works as an amplifier, not as a replacement for engineering scaffolding: rules for selecting modules, isolation of external dependencies, a branch coverage threshold, and regular cleanup of the suite from duplicates. We'll break down how this is structured and what gaps it closes.
What changes after implementation: coverage, speed, defects
Implementation should be evaluated not by the number of generated files, but by the shifts taken directly from the continuous integration pipeline.
We record these indicators before the start and compare them after a month of work — if a metric does not move, the set of checks is rewritten. Below is what actually changes on projects where generation is built into GitLab CI and runs on every change.
- Coverage of new modules. Fresh code no longer arrives without checks: generation runs together with branch creation, and the share of covered areas in new modules reaches 70–80% immediately, not after several iterations.
- Early regression detection. A failure is caught at the merge stage, when the cost of a fix is minimal, rather than in a test environment where it is searched for manually. The class of missed defects shifts toward boundary scenarios that cannot be derived from the code.
- Check run time. The suite is split by levels: fast unit checks run on every change, heavy scenarios — on a schedule. Feedback on a fix arrives in minutes instead of dozens of minutes.
- Maintenance of automated tests. The share of checks requiring manual edits after code changes falls: the generator relies on current signatures and contracts, while outdated checks are marked and removed automatically.
- Suite stability. The share of flaky failing checks is calculated separately — they are isolated or deleted; otherwise engineers stop trusting the run result and miss real failures.
Comparison of modes: by changes, by snapshot, by schedule
The generator mode determines which tests will appear in the output and how long after a code change.
Generation by changes gives fast feedback directly in the branch, a full project snapshot — baseline coverage where there were no tests, a schedule — regular review of what has already been written. Below is how these modes differ in scope of work and where they run into project constraints.
| Mode | When it fits | Limitations |
|---|---|---|
| Generation by changes | Active development, short branches: only affected modules and their dependencies are processed | Does not grow coverage in code that has not been touched for a long time |
| Full project snapshot | First connection, when there are no or few tests; a baseline is needed before refactoring | Long first pass, some tests land on trivial areas — manual filtering is needed |
| Incremental run | The project is already covered: the generator picks up files without tests and changed ones | Requires coverage accounting and matching a test to the source, otherwise gaps accumulate |
| By schedule | Regular review of old code, nightly runs without haste | Regressions are found with a delay, not at the moment of change |
In practice, modes are combined: by changes works on every commit to the working branch, a full snapshot is run once at connection, a schedule closes accumulated debt. Incremental run reuses already generated tests and does not run generation over the same files twice.
Next, we'll break down which project constraints — closed dependencies, a monorepo, long builds — shift the choice one way or another.
Implementation stages: audit, scheme, setup, verification
We start with an audit: we look at which modules are covered by checks, where they duplicate each other, and where critical logic branches are not tested at all.
We calculate the share of covered branches and the suite run time — these two metrics show where generation will have an effect immediately, and where unstable tests failing for random reasons will first need to be fixed.
- Test suite audit. We analyze coverage by module, mark duplicates and dead checks, and record the run time and share of flaky tests as a baseline.
- Integration scheme into the build. We determine where generation is embedded: as a separate pipeline step, by file changes, or by schedule, and how the result gets into the shared branch.
- Setup of run rules. We define visible directories and exclusions, requirements for a test: boundary values, expected exceptions, isolation of calls to the network and external services.
- Verification on one module. We run the loop on a limited area, compare the result with manually written tests, and filter out variants that pass under any logic.
- Acceptance of the result. We calculate the share of passed checks, the increase in covered branches, and the run time; tests that do not catch logic substitution are removed from the suite.
- Handover of the loop to the team. We hand over configurations, run rules, and instructions on how to change requirements and connect new modules without us.
The order is exactly this: run rules without an audit produce thousands of useless tests, and acceptance without measurable metrics does not show that the loop works at all. As a result, the team gets a working scheme and clear control points, not a set of scripts.
Deliverables: configurations, rules, regulations
The generation loop is handed over together with the project: the team changes thresholds, adds exclusions, and adjusts rules on its own without contacting us.
Below are the artifacts that remain in the repository and documentation and allow the loop to be developed by the team's own engineers.
- Build configuration: test generation stages, their run order relative to linters and static analysis, dependency cache, and parameters affecting run time.
- Check run rules: which events trigger generation and the run, whether checks are mandatory for merging, coverage threshold for new and changed files separately.
- Exclusion policy: a list of paths, patterns, and external modules where tests are not generated, with the reason and review deadline — so the list does not turn into a dump.
- Test documentation: how to read the report, how generated tests are marked, by what signs a test is accepted into the main branch and by what signs it is returned for revision.
- Instruction sets for the model: templates for describing modules, examples of successful and rejected tests — the basis for further training without us.
- Review and rollback regulations: who confirms a disputed test, how quickly to silence a false positive, and how to roll back an unsuccessful rule change.
- Loop metrics: share of accepted tests, number of failures per run, build time before and after — they show whether the system is degrading or improving.
- Maintenance regulations: the person responsible for the loop, frequency of exclusion reviews, and procedure for updating the rules themselves.
Case study: how coverage grew and what happened with regression
A product service with a dense flow of changes: several releases per week, tests were written manually and only for critical scenarios. Coverage held at 41%, and checking changes took about 40 minutes on each merge request.
Over the quarter, nine defects reached production — six of them in modules where tests had not reached.
What was changed in the pipeline
Generation was built into GitLab as a mandatory step on every change: the tool parses edits by lines and function signatures and proposes tests, the engineer confirms or edits them in review — regular files go into the repository, not a report from a third-party service.
Mutation testing was enabled: a test that does not fail on an artificially introduced defect is automatically filtered out and does not count toward coverage.
The coverage threshold was set on changed lines, not the whole project: in a large monolith, the overall figure grows slowly and does not reflect the risk of a specific release.
What happened with the metrics
The coverage share grew from 41% to 76%, and on changed lines it stays above 90%. Checking changes dropped from 40 to 12 minutes, and reviewing tests takes 3-4 minutes instead of writing from scratch.
Missed defects over the quarter — two instead of nine, and both at integration seams where generation was deliberately not applied.
The metric is sustained not by a one-time campaign but by a pipeline threshold and automatic filtering of weak tests: without them, coverage rolls back within a couple of months. The same template was deployed on a neighboring service in a week.
Technical doubts before starting work
Engineering objections differ from general ones: they are about access to code, duplicates in the set of checks, and what will happen to automated tests after refactoring. Below are answers to those most often heard before the start.
Will the source code go outside?
Generation is deployed inside the customer's perimeter: the model runs locally, and only a function fragment after masking goes to external services — without entity names, keys, and connection strings, without test data.
If the policy is stricter, we assemble a fully closed loop without outbound traffic. Isolation rules are fixed in the configuration rather than remaining an agreement.
Will we not get a hundred duplicate tests?
Before adding, the suite is checked against the existing one: identical scenarios and checks already covered at another level are filtered out. Generation proceeds through uncovered branches and boundary values, so coverage grows, not the file volume and run time.
What will happen when code changes and refactoring?
Tests are tied to observable behavior, not to internal implementation details, so moving logic does not turn the suite into garbage.
The pipeline retains a step that stops merging if coverage of changed files falls below a set threshold — automated test maintenance becomes a regulation, not manual work.
How is it verified that a test is not empty?
Selective generation goes through risk areas, and each test is verified against a broken version of the implementation: if it does not fail, it is useless and is discarded.
Checks without assertions and tautological comparisons are filtered out automatically, then an engineer reviews the suite — quality control remains with a human, not the model.
Let's discuss the task and come back with an estimate and plan
For the estimate of the scope of work to rely on facts rather than assumptions, it is enough to describe the task in free form: what the service is, what it is written in, how the build is currently organized, and which modules hurt to cover first. This is usually enough to come back with a plan and precise clarifying questions.
During the conversation, we clarify six things — they show where AI test generation will have an effect immediately and where manual setup is needed:
- language, test framework, and current module coverage — we choose what to generate first;
- how the build pipeline is structured and at which step the test run connects to it;
- review rules: who accepts a generated test and by what criteria;
- restrictions on access, environment isolation, and work with closed code;
- priority scenarios where the cost of an error is highest;
- acceptance metrics: share of covered branches, run time, share of tests accepted without edits.
In response, an implementation plan comes back: stages, what we do and what is required on your side, connection points to the build, readiness criteria, and rollback procedure if the quality of tests is not satisfactory. Describe the task — we will come back with an estimate and plan.
What to do if the test suite has already grown too large and is not maintained?
An overgrown test suite grows faster than the product: checks accumulated under old interfaces, and some of them do not fail on any real defect. The run takes dozens of minutes, several checks fail intermittently — and the red report stops being read. This is the point where the team starts fearing refactoring more than the errors themselves.
The analysis begins with measurements, not deletion. We run the suite with branch coverage from JaCoCo reports, bring up the failure history for recent iterations, and add mutation assessment: if a logic change does not fail any check, then it verifies nothing. Then each check falls into one of the groups:
| What we do with the check | When it fits | Limitations |
|---|---|---|
| Leave as is | Fails only on a real defect, run time is normal | Requires edits with every contract change |
| Regenerate | The contract has changed, the check duplicates a neighboring one | Current requirements or a data schema are needed |
| Quarantine | Fails arbitrarily, cause not established | No longer than one or two iterations |
| Delete | Does not fail on any logic change | The wording is preserved in version history |
Quarantine is time-limited: an unstable check is either fixed or replaced. Test migration proceeds through risk areas — first the critical flow, then the rest.
New checks are generated according to current contracts, and old wordings remain in the version control system: they can be returned to at any time.
Test suite cleanup is complete when the run fits within the target time, branch coverage does not fall, and there are almost no flaky failures left.