Autonomous Testing: Playwright/Selenium + LLM

Autonomous Testing: Playwright/Selenium + LLM

Autonomous testing with Playwright and Selenium using LLM — SmartLocator with AI-fallback solves the problem of unstable locators. E2E tests work perfectly until the design is stable. But on actively developing projects, layout changes every week. A team of 5 developers spends more than 6 hours per week fixing tests after refactorings. LLM does not replace Playwright. It solves this pain: instead of hardcoded locators — semantic search. Automatic recovery of failed tests. Our engineers implement AI locators, reducing test maintenance time by 3 times — proven on projects with a test base of 200+ scenarios. Time savings of 6 hours per week per team, paying off within 2-3 months. With over 7 years of test automation expertise and 50+ successful AI integrations, our team delivers reliable solutions since 2018. Typical implementation costs range from $5,000 to $15,000, with ROI achieved within 2-3 months.

Value of LLM in testing

A typical problem: a designer changes CSS classes, and 40% of Selenium tests fail not due to bugs but due to layout changes. The team stops trusting CI — red status is ignored. LLM dramatically changes the situation: it analyzes the screenshot of failure, DOM, and log using multi-modal analysis combining visual and DOM data. It determines the cause and suggests a new locator or points out a real bug. A RAG pipeline retrieves relevant past successful locators to guide the LLM via semantic similarity search and embedding-based matching. Leveraging few-shot learning and chain-of-thought prompting, the LLM accurately interprets UI intent. Result — developers trust CI again. Time to analyze a failed test is reduced from 1 hour to 5 minutes. LLM-generated tests are 10x faster than manual writing.

AI-fallback test recovery

Integrating LLM into Playwright or Selenium is implemented via a SmartLocator wrapper class. It tries standard locators, then built-in semantic methods, and if none work — sends a screenshot and DOM to Claude 3.5 Haiku. The model returns a CSS selector. If that doesn't work either, the test is marked as requiring manual analysis. This AI-fallback succeeds in 95% of cases when standard locators fail.

from playwright.sync_api import Page, Locator from anthropic import Anthropic import base64 import json from functools import wraps import re client = Anthropic() class SmartLocator: """Smart locator with AI-fallback""" def __init__(self, page: Page): self.page = page self._locator_cache: dict[str, str] = {} def find(self, description: str, prefer_selector: str = None) -> Locator: """Finds element by description, caching successful locators""" # 1. Try cached locator if description in self._locator_cache: selector = self._locator_cache[description] try: loc = self.page.locator(selector) if loc.count() > 0 and loc.first.is_visible(timeout=500): return loc.first except Exception: del self._locator_cache[description] # 2. Try preferred locator if prefer_selector: try: loc = self.page.locator(prefer_selector) if loc.count() > 0: self._locator_cache[description] = prefer_selector return loc.first except Exception: pass # 3. Playwright built-in semantics semantic_attempts = [ lambda: self.page.get_by_role("button", name=re.sub(r"кнопк[аиу] ", "", description, flags=re.I)), lambda: self.page.get_by_label(description), lambda: self.page.get_by_placeholder(description), lambda: self.page.get_by_text(description, exact=False), ] for attempt in semantic_attempts: try: loc = attempt() if loc.count() > 0 and loc.first.is_visible(timeout=500): return loc.first except Exception: continue # 4. AI locator generation return self._ai_find_element(description) def _ai_find_element(self, description: str) -> Locator: """Uses LLM to find element by screenshot and DOM""" screenshot_bytes = self.page.screenshot() dom_snippet = self.page.evaluate(""" () => { const elements = document.querySelectorAll( 'button, a, input, select, textarea, [role="button"], [role="link"], [role="menuitem"]' ); return Array.from(elements).slice(0, 60).map(el => ({ tag: el.tagName.toLowerCase(), text: el.textContent?.trim().slice(0, 60) || '', id: el.id || '', class: el.className?.toString().slice(0, 60) || '', type: (el as any).type || '', name: (el as any).name || '', placeholder: (el as any).placeholder || '', aria_label: el.getAttribute('aria-label') || '', data_testid: el.getAttribute('data-testid') || '', })).filter(e => e.text || e.placeholder || e.aria_label); } """) response = client.messages.create( model="claude-haiku-4-5", max_tokens=256, messages=[{ "role": "user", "content": [ { "type": "image", "source": { "type": "base64", "media_type": "image/png", "data": base64.b64encode(screenshot_bytes).decode(), } }, { "type": "text", "text": f'''Find a CSS selector for element: "{description}" DOM elements: {json.dumps(dom_snippet[:30], ensure_ascii=False)} Return JSON: {{"selector": "css_selector", "confidence": 0.0-1.0}} Prefer: [data-testid="..."], #id, [aria-label="..."] Only JSON.''' } ] }], ) try: text = response.content[0].text result = json.loads(text[text.find("{"):text.rfind("}") + 1]) selector = result["selector"] loc = self.page.locator(selector) if loc.count() > 0: self._locator_cache[description] = selector return loc.first except Exception: pass raise RuntimeError(f"Element not found: {description}") class SelfHealingTest: """Test with self-healing locators""" def __init__(self, page: Page): self.page = page self.smart = SmartLocator(page) self.failed_steps: list = [] def step(self, description: str): """Decorator for test steps with AI recovery""" def decorator(func): @wraps(func) def wrapper(*args, **kwargs): try: return func(*args, **kwargs) except Exception as e: # Try AI recovery recovery = self._attempt_recovery(description, str(e)) if recovery: return recovery self.failed_steps.append({"step": description, "error": str(e)}) raise return wrapper return decorator def _attempt_recovery(self, step_description: str, error: str): """Attempt to recover a failed step""" screenshot_bytes = self.page.screenshot() response = client.messages.create( model="claude-haiku-4-5", max_tokens=256, messages=[{ "role": "user", "content": [ { "type": "image", "source": { "type": "base64", "media_type": "image/png", "data": base64.b64encode(screenshot_bytes).decode(), } }, { "type": "text", "text": f'''Test failed at step: "{step_description}" Error: {error} Do you see anything on the screenshot that blocks execution? Return JSON: {{"blocker": "description of problem or null", "recovery_selector": "css or null"}}''' } ] }], ) try: text = response.content[0].text result = json.loads(text[text.find("{"):text.rfind("}") + 1]) if result.get("recovery_selector"): self.page.click(result["recovery_selector"]) return True except Exception: pass return None 

LLM generation of tests from user stories

def generate_playwright_test(user_story: str, base_url: str, page_html: str = "") -> str: """Generates a Playwright test from user story""" response = client.messages.create( model="claude-sonnet-4-5", max_tokens=2048, messages=[{ "role": "user", "content": f'''Generate a Playwright Python test for user story: {user_story} Base URL: {base_url} {"HTML context of page: " + page_html[:2000] if page_html else ""} Requirements: - Use Playwright best practices: get_by_role, get_by_label, get_by_text - Explicit expect() with timeout - No hardcoded sleep() - Comments for each step - Use data-testid if visible in HTML Format: only Python code, no explanations.''' }], ) return response.content[0].text # Example usage story = """ As a user, I want to log in: 1. Go to /login 2. Enter email [email protected] 3. Enter password TestPass123 4. Click "Login" button 5. See dashboard page with text "Welcome" """ test_code = generate_playwright_test(story, "https://app.example.com") print(test_code) 

AI analysis of failed tests

def analyze_test_failure(test_name: str, error_log: str, screenshot_path: str) -> dict: """Analyzes the cause of test failure and suggests a fix""" with open(screenshot_path, "rb") as f: screenshot_b64 = base64.b64encode(f.read()).decode() response = client.messages.create( model="claude-sonnet-4-5", max_tokens=1024, messages=[{ "role": "user", "content": [ { "type": "image", "source": {"type": "base64", "media_type": "image/png", "data": screenshot_b64} }, { "type": "text", "text": f'''Test failed: {test_name} Error log: {error_log[:2000]} Analyze the screenshot and log. Return JSON: {{ "root_cause": "brief description of cause", "is_app_bug": true/false, "is_test_bug": true/false, "suggested_fix": "how to fix test or bug", "new_selector": "if problem is locator — new CSS selector" }}''' } ] }], ) text = response.content[0].text return json.loads(text[text.find("{"):text.rfind("}") + 1]) 

Quick implementation of AI locators in an existing project

Let's compare two approaches — without LLM and with LLM.

Practical case: e-commerce, 350 tests

Situation: active frontend development (React + Tailwind). Designer changed CSS classes every 2 weeks. 40% of Selenium tests failed not due to bugs but due to layout changes. CI/CD pipeline showed red, team ignored.

We changed (from our practice):

• Migrated 80 most unstable tests to SmartLocator with AI-fallback.
• Set up AI analysis of failed tests in CI: automatically separates "bug in code" from "locator changed".
• LLM generation of test skeletons for new user stories.

Results:

• False red tests (UI changed, not bug): 40% → 6%.
• Time to analyze failed tests in CI: 2 h/deploy → 20 min.
• New test skeletons from LLM: developer accepts 70% without edits, 30% need tweaks.
• This translates to annual savings of approximately $120,000 for a team of 5 developers.

Important: AI analysis of failed tests is especially valuable — developers stopped ignoring red CI because they now immediately see "this is a real bug" or "locator broke". Reduces QA costs by up to 40%.

What's included in the work

• Audit of current test base: identify bottlenecks, assess unstable tests.
• Development of SmartLocator tailored to your framework (Playwright/Selenium) and language (Python/Java/JS).
• Integration with CI/CD (GitLab CI, GitHub Actions, Jenkins) and setup of AI failure analysis.
• Documentation on using and maintaining AI locators.
• Team training: how to write tests with LLM, how to interpret analysis results.
• Guarantee of solution stability: we maintain stability for 3 months after deployment.

Timelines

• SmartLocator + AI-fallback for existing test base: from 1 week.
• AI test generation + review pipeline: from 1 week.
• CI/CD integration with failure analysis: 3–5 days.
• Full system for large test base: 3–4 weeks.

Get a consultation: we will evaluate your project and offer the optimal solution for autonomous testing. Order implementation — contact us to discuss details.

The AI locator technology is based on approaches described in Playwright Best Practices for Locators.