Playwright/Selenium integration with LLM for autonomous testing

Autonomous testing: where manual regression breaks

Manual regression stops keeping up with release frequency before it becomes visible in reports. While engineers run a fixed set of scenarios, several more changes enter the main branch — and some checks are already executed against a different build.

Release delay grows non-linearly: each enhancement adds cases, while time for a full run remains the same.

The second failure point is maintainability of checks. The same user path becomes overgrown with duplicates in different suites, and when markup changes, edits have to be made in each one separately.

Failures are triaged manually: a test failed — it's unclear whether the product broke or the scenario itself became outdated, and triage takes more time than the run.

Autonomous testing changes the source of checks. Check generation comes from the task statement and actual system behavior, not from manually rewriting steps: the scenario adapts to a changed screen, duplicates collapse, and each failure is accompanied by a root-cause analysis.

Pairing Playwright or Selenium with a language model produces an executable check and a verdict that can be read without diving into test code — outdated steps are marked separately rather than silently failing.

Effect on the release cycle: a full run is reduced from days to tens of minutes, the share of paths confirmed automatically grows, and manual work leaves recurring scenarios.

At the same time, the autonomous loop does not replace exploratory testing — it removes routine, leaving people new scenarios and disputed areas. We break down how this is covered in practice — from coverage collection to failure triage.

What changes after implementation: coverage and speed

Before implementation, coverage is limited by manual labor: an engineer describes a check only if they already know where it breaks.

The pairing of Playwright and a language model parses requirements and traverses scenario branches itself — including negative ones that no one got around to.

On critical paths, coverage grows severalfold, and the report shows which branches are checked and which remain unattended.

A run stops being a half-day event: checks are parallelized, and changes get a verdict in minutes rather than by evening.

This changes team behavior — a change is sent for checking immediately rather than accumulated in a batch for release, and an error is found at the moment when it is still cheap to fix.

Flaky checks are the main reason a red report stops being read. The agent rechecks a failed step: repeats the action, verifies markup and application state, separating a real breakage from a timeout or a shifted selector. The share of false failures drops manyfold, and the signal is again perceived as a signal.

Along with this, the class of failures "missed before release" goes away: critical paths pass in full on every release candidate, not selectively according to a manually compiled list.

The release decision relies on the coverage report and the history of past failures, not on "we seem to have checked." Release stability becomes a measurable quantity, not a feeling.

Ways to integrate checks: comparison of modes

The mode of checks determines how many false failures the team triages every week. Ready-made automation tools are predictable but fragile to markup edits; a language-model-based agent survives interface changes but requires control over step correctness.

The comparison of approaches below is by resilience to changes, scenario coverage, and maintenance effort.

In practice, the base layer is covered by deterministic checks: they give an unambiguous answer "passed or not." The agent is connected selectively in areas where markup changes every week, and visual and programmatic checks are separated into different suites so that a layout failure does not mask a logic error.

Such a hybrid keeps the share of false positives low, and coverage grows without rewriting scenarios.

How implementation proceeds: stages of engineering work

Implementation proceeds in short iterations: each ends with a working stand and a set of checks that can already be run. Thus at any step there remains a result suitable for use, rather than waiting for final delivery.

1. Scenario audit. We review regression suites and the product map, select critical user paths, and mark what is covered by autonomous testing and what consciously remains with manual checking.
2. Run scheme. We fix where checks are launched, how data is isolated, and how the initial state is restored between scenarios: otherwise one failed test drags neighboring ones along.
3. Environment setup. We raise stands, test accounts and data, configure Playwright and a run in the browsers actually used by the product audience.
4. Check generation. Based on the scenario description and hints about page structure, the model produces a draft check, the engineer edits it and keeps only what will survive the next interface edits.
5. Stability verification. We run the suite repeatedly on the same code and clean out steps that produce different results without product changes.
6. Acceptance check. We compare the autonomous run with the manual one on control scenarios and make sure that intentionally introduced defects are actually found.
7. Handover to the team. We hand over configurations, rules for maintaining checks, and instructions for expanding the suite — after that the team maintains it itself.

Implementation artifacts: schemes, configurations, regulations

Implementation of autonomous testing ends not with a demonstration run, but with a set according to which the customer's team works further without us. Everything goes into the customer's repository: configurations, check scheme, access policies, and support regulations. Below is the list of implementation artifacts accepted together with the work.

• Check scheme — a map of critical paths with priorities, a list of preconditions, and criteria by which a scenario is considered passed or failed.
• Launch configurations — profiles for local runs, the pre-release loop, and nightly regression: different scenario suites and different levels of check strictness.
• Access policies — dedicated accounts for runs, prohibition of modifying operations on production data, storage of secrets outside code and outside change history.
• Run documentation — how to read the report, which artifacts are saved on failure (screenshot, log, step recording), and how to reproduce a failure on a local machine.
• Support regulations — timelines for triaging failed scenarios, conditions for moving a test to quarantine, procedure for updating selectors when the interface changes.
• Suite development rules — how to add new scenarios, who approves their priority, and by what criteria a test is removed as outdated.

Acceptance proceeds item by item separately: we show a run on the customer's stand, analyze an intentionally broken scenario from report to cause, and hand over accesses, configurations, and documentation to the engineers who remain on support.

Case study: how manual regression was reduced on an order flow

The customer is an online store with an order flow and releases every two weeks. Regression relied on a manual run: about 320 checks, a full pass took 8–10 hours of work for two engineers and stretched over several days.

Checks could not keep up with changes, and 4–6 missed defects consistently reached release — some surfaced already in checkout.

We kept the scenarios in Playwright as the execution layer and built a model on top of them: it assembles steps from the text description of a check, and when layout changes, it finds an element by meaning rather than by a chain of classes.

Regression runs on every code change and nightly on schedule, and the result comes as a report with steps, screenshots, and the failure point.

Two months after launch: full run — 35–40 minutes instead of 8–10 hours, the share of manual checks fell from 100% to 12% — manually remaining were email proofreading and layout on rare devices, missed defects per release — 0–1 instead of 4–6.

A separate effect is support cost: a product card redesign no longer breaks dozens of tests, some are restored on their own, the rest are fixed selectively.

A new scenario is assembled in 20–30 minutes instead of half a day: an engineer describes the check in words, and the model expands it into steps. Autonomous testing became part of the release cycle, not a separate experiment.

How autonomous testing differs from classic automated tests

A classic automated test is a pre-recorded scenario: steps, selectors, and expected result are defined in code. While the interface is stable, the run is reproducible and gives the same verdict.

It breaks where markup changes: the selector does not find the element, and the check fails not because of a defect but because of a layout edit.

Autonomous testing begins where a goal and rules are specified instead of steps. A language model receives the scenario description and page state, decides itself where to click and what to enter, and after each action compares the result with the expectation.

It finds elements by meaning — by text, role, neighboring blocks — so a cosmetic markup edit does not fail the check. Step generation occurs on every run, not once during writing.

Hence the key difference is the cost of maintaining checks. In the classic approach, an engineer fixes selectors after every interface edit, and this share grows along with the number of scenarios.

In the autonomous approach, the goal description and rules are edited; step code does not need to be rewritten.

The choice of approach is dictated by the frequency of changes and the cost of error, not fashion. Where the verdict must be reproducible step by step — payments, access rights — a deterministic framework remains, and autonomous steps are connected selectively.

Where the interface is rewritten every week, the autonomous run wins precisely due to lower maintenance.

Engineers' concerns: access, data, flaky checks

Engineers ask the same questions before a pilot: where data goes, what about access to environments, how to distinguish a real defect from a flaky check. Below are answers without generalities, at the configuration level.

Is real test data needed?

No. The stand is filled with synthetic data according to your database schema: the generator respects relationships between tables and uniqueness constraints, so scenarios pass the same path as on production records. A depersonalized copy is the second option when reproducibility of specific cases matters.

How do you deal with flaky checks?

Flakiness is treated by waiting for an event, not by a pause: the agent waits for a network response, appearance of an element, or state change, not a fixed delay.

Every failure enters the triage queue — a check that fails at the same step goes into quarantine until fixed and does not block the build.

What about access rights and support after handover?

We work under a separate service account with minimal rights: reading data and running tests, without production secrets.

After handover, configurations, quarantine policies, and the failure triage regulation remain, and support for the first runs takes exactly as long as your team needs to edit selectors independently.

Let's discuss the task: what is needed to start work

To take on autonomous testing, a full technical specification is not needed — a short set of inputs is enough.

The earlier we see accesses, stands, and the list of critical scenarios, the more accurate the estimate of scope and the fewer assumptions in the implementation plan: an agent configured on demo data behaves differently from one on a live environment with real roles and database state.

• A repository with current automated tests and rights to a separate branch for the agent configuration.
• A test environment replicating production in data structure and role sets, with its own accounts.
• A list of critical scenarios and rules: what is considered a correct result and what is a false positive.
• Run regulations: on which event we launch, with what frequency, who triages the report and within what timelines.
• Access to application logs and the issue tracking system — the agent files defects there with reproduction steps.
• Agreed constraints: which data and operations we do not touch, where a manual confirmation step is mandatory.
• A person responsible on the product side who triages disputed findings and confirms acceptance.

Describe the task in free form — we will come back with a scope estimate, access requirements, and a stage-by-stage implementation plan.