Why AI test scenarios hit a wall with manual markup
Requirements for system behavior live in user stories and API contract descriptions, while checks live in a separate automated test repository.
The link between them rests on the author's memory: the story was rewritten during review, a field was renamed, a required parameter was added — the check remained the same and continues to "go green" on an outdated expectation. The divergence between requirements and code accumulates quietly until the first incident review.
Then the arithmetic kicks in. Regression lags behind the release: while the team runs manual runs on critical branches, two more changes manage to enter main.
One changed contract — a different error format or a new status when permissions are absent — brings down dozens of checks at once, and instead of analyzing causes, mass editing or disabling begins. Maintaining automated tests in such a scheme takes more engineering hours than it saves.
A separate loss area is negative branches. Empty and boundary values, permission escalation, conflicting states, unhandled types in the request body almost always remain out of coverage: they are written manually last, and they break first.
AI test scenarios remove this dependency: checks are derived from story text and contract descriptions, so a change in requirements immediately changes the set of checks instead of leaving outdated ones behind. We explain how this works and what is needed from the team on the input side.
What changes after implementation: coverage and run speed
The effect of AI test scenarios is measured not by the number of generated files, but by the share of requirements covered by checks, full run time, and which failures stop reaching release.
We record these metrics before the start — otherwise there will be nothing to discuss the result with.
- Requirements coverage. Every story with acceptance criteria is expanded into a set of checks: 80–90% of declared behavior is under automated control instead of selective 40–50% during a manual pass.
- Full run time. The critical suite fits within minutes and runs on every change; the full suite fits within tens of minutes by being split into independent parts; waiting for the nightly window goes away.
- Eliminated class of failures. Discrepancies between the service contract and the expectations of adjacent modules are caught before rollout: the source of truth is held by OpenAPI, not a "human-readable" description that diverges from the code.
- Reuse of checks. Steps are moved into a shared library — a new scenario is assembled from ready-made blocks, and the share of reused steps grows from release to release.
- Maintenance cost. Editing requirements updates related checks, unstable scenarios are marked automatically, and triage of false failures is reduced severalfold.
The main shift is predictability: release stops depending on whether someone had time to go through the list manually. We compare each item with measurements before and after, so the conversation about the result is based on numbers, not impressions.
Three approaches to AI test scenarios and their limitations
The three approaches differ not by tool, but by the source of truth for the check.
For template checks, it is a pre-described framework; for requirements-based generation, it is the text of the user story and API description; for the hybrid approach, it is a real call recorded in the environment.
This leads to different classes of failures: a template breaks when the data format changes, a generator breaks with an incomplete description, and a hybrid breaks with unstable test data.
The choice is made along two axes: how stable the requirements are and how many paths need to be covered. Below is what each option yields in practice.
| Option | When it fits | Where it breaks |
|---|---|---|
| Template checks | Stable contract, recurring scenarios, need for a predictable run | A new field or changed format — the framework is edited manually; non-standard branches remain uncovered |
| Requirements-based generation | Stories are described in detail, need a fast draft for dozens of scenarios | Under-specified validation logic — the model invents a check that does not exist in the system and produces false failures |
| Hybrid approach | Live environment, long call chains, high cost of regression | Requires environment isolation and stable data, otherwise the recorded step is not always reproducible |
The limitations of generation are predictable and are removed by engineering, not by phrasing the request to the model. We build a hybrid scheme: scenarios are generated from requirements, then run in the environment, and discrepancies are triaged and turned into template checks. This way coverage grows from iteration to iteration, and the test suite remains reproducible.
Implementation stages: from requirements audit to handover
Generation of AI test scenarios does not begin with a request to the model: if you immediately feed it the API description, you get a set of steps that reads nicely but checks the wrong thing.
Therefore the work proceeds in stages, and each ends with an artifact that can be verified before entering the pipeline.
- Requirements audit. We analyze user stories, specifications, and the request collection. We mark wording from which a verifiable scenario will not emerge — we return them to the authors before the start, otherwise the model will fill in the gaps and we will get false coverage.
- Generation schema. We define what is fed as input (story, contract, test data) and what must be output: steps, expected responses, boundary values, and the format of finished AI test scenarios.
- Rule setup. Mandatory preconditions, data isolation between runs, check priorities, prohibition of comparison with changing values and of dependence on the current time.
- Pilot validation. We take several real enhancements and look at how many scenarios pass without edits, where the model made mistakes, and what data was missing. We fix the schema, not individual scenarios.
- Embedding into the process. Generation is triggered by a change in requirements or contract, and the result goes into the automated test repository; divergence between requirements and code is visible during change review, not after release.
- Handover to the team. We hand over the schema, the set of rules, and an analysis of typical errors, and train on real tasks. After that, the team adds rules and resolves disputed scenarios on its own.
What remains with you: check suites and rules
Autogeneration does not end with a demonstration on a test environment — otherwise the result remains with us, not with you.
We hand over a set of artifacts that lives in your repository and works without our participation: it can be edited, extended, and connected to any check pipeline.
- Check suites — scenarios grouped by requirements: positive paths, negative branches, boundary values, API contract checks, and error code checks.
- Requirements markup rules — conventions by which user stories and API schemas are described: required fields, acceptance criteria, example format. They determine how consistently the generator produces checks.
- Test data — prepared datasets and data generators, including masking of personal information and predictable database states before a run.
- Coverage map — a requirement → check mapping table: it shows which paths are covered and which remain uncontrolled, and this state is measured, not debated.
- Update procedure — what to do when the contract or acceptance criteria change: who initiates regeneration, how divergence with manual edits is checked, and how the suite is synchronized with requirements.
- Documentation — instructions for running, triaging failures, and adding new scenarios so the team can work with the suites without contacting us.
- Naming rules and domain glossary — unified terms and folder structure: six months later, the name of a check makes clear what it validates and which requirement it belongs to.
All of this is handed over together with the sources and a walkthrough on your side — after that, your team develops the suites.
Case study: how manual runs were reduced on a live product
The product is a customer portal with billing, access roles, and a dozen external integrations. Regression relied on manual runs: 320 scenarios took 6–8 hours before release, and critical paths — payment, access recovery, plan change — were covered at about 40%.
Critical paths were reached last, when time had already run out. Per sprint, 11–14 defects reached production, and half were where the logic had been changed but the check had not been rewritten in time: the manual scenario became outdated faster than it was maintained.
We restructured the entry point into testing. The sources became user stories in the tracker and the API specification: from them the generator assembled drafts of steps, checks, and test data, and an engineer edited them and tagged them with the "critical path" label.
Scenarios were exported into ready-made Playwright code — they live in the same repository as the product and run in the build pipeline on every branch merge. There is no separate "run" day anymore.
What the metrics showed
The share of covered critical paths grew from 40% to 88%, and full regression was reduced from 6–8 hours to 45 minutes. Defects reaching users dropped from 11–14 to 2–3 per sprint.
For the first month, manual and generated checks ran in parallel — this confirmed that coverage grew not at the expense of false positives. A scenario's divergence from real behavior is now fixed in one place, and the entire check chain returns to green the same day.
How requirements-based generation differs from action recording
Both techniques solve one task — obtaining a set of checks that runs on every release.
The difference is in the source of truth: action recording captures what the system did at the moment a button was clicked, while requirements-based generation captures what it is obliged to do according to the user story and API contract.
Resilience to UI changes is the first difference. A recorded scenario relies on locators of a specific layout: move a block, change the markup — the suite falls apart, and half of the failures are unrelated to logic.
A generated scenario relies on steps named in the requirements and on the contract description in OpenAPI: a change in one step is picked up by the remaining cases.
Readability of the suites differs most of all. A recorded test is a long chain of clicks and waits; when triaging a failure, the meaning has to be reconstructed step by step.
A generated suite reads like a requirement: you can see which branch is being checked and under what condition it should fail.
The main difference is which defects are caught. Recording confirms that an element is present and a transition occurred, that is, it checks layout and navigation.
Generation from requirements and API description covers logic regression: response codes, body structure, boundary and negative branches that are almost never reproduced manually.
Maintenance cost is distributed differently. Recording is cheaper at the start and becomes more expensive with every release: the more actively the UI changes, the more tests are rewritten.
Generation requires described requirements as input, but it survives redesign and is extended with a new scenario rather than a new script. In practice, the techniques are combined: recording is a quick draft for an unstable area, requirements are long-lived regression.
Common technical concerns before starting
Concerns at this stage are natural: generation does not replace understanding of the system, it accelerates routine.
The solution relies on two sources — textual requirements and API contracts — so the quality of the result directly depends on how unambiguously behavior is described. Below are answers to what is most often asked before work starts.
How accurate is generation — how many scenarios will have to be rewritten manually?
We do not consider a generated scenario ready to run without review. The model relies on explicit conditions: input data, expected response code, body structure, behavior on error.
An engineer verifies the assertions and boundary values, after which the scenario goes into the suite. The share of edits depends on requirements discipline, not on the model, so we immediately show where a wording allows double reading.
What should be done if requirements are incomplete or contradictory?
Incompleteness is a normal input, not a blocker. The generator marks places where data is missing: no expected response, no described error format, no defined behavior for an empty field.
These marks are collected into a list of questions for the analyst — this becomes the main result of the first pass. This way gaps in requirements are identified before development, not at acceptance.
Will we duplicate automated tests that already exist?
Before generation, we inventory the existing suite: we compare by endpoint, check type, and input data, not by file name. Matching paths are cut off; only uncovered branches are added — errors, access permissions, boundary values. As a result, the suite grows due to coverage gaps, not due to repetitions.
What environment access is needed and what about isolation?
A test environment and an account with limited permissions are sufficient. Generation does not access the production environment and does not write data — it reads contract descriptions and runs checks in an isolated environment with prepared fixtures. Tokens and addresses are stored in environment variables, not in scenario code.
Let's discuss the task: what is needed to start
Estimation of AI test scenario generation depends not on the amount of code, but on the completeness of the input data: requirements, service schema, and existing checks.
The more accurate the initial set, the fewer assumptions in the estimate and the faster the sequence of work turns into a working environment.
The difference between a vague brief and a complete data package is usually several clarification iterations and a shifted acceptance point for the result.
What speeds up project estimation:
- examples of user stories and acceptance criteria — they show how formalized the requirements are and whether an intermediate layer for parsing them is needed;
- service schema and contracts of their interaction — determines where generation is tied to real calls and where it remains at the level of textual descriptions;
- the current set of checks: manual checklists, automated tests, their coverage, and the share of unstable runs;
- environment constraints — which environments are available, whether test data exists, and how often it is updated;
- access rules and artifact storage requirements if code and data do not leave your environment;
- release procedure: how often requirements change and who confirms changes;
- the acceptance owner — the same person approves the readiness criteria.
Then the order of work is: requirements analysis, a pilot on one module, comparison of generated scenarios with the reference set, then expansion to the remaining services.
Describe the task — we will come back with an estimate and a plan that records the stages, acceptance points, and scope.
