Autonomous Testing: Playwright/Selenium + LLM

Autonomous Testing: Playwright/Selenium + LLM

Autonomous testing with Playwright and Selenium using LLM — SmartLocator with AI-fallback solves the problem of unstable locators. E2E tests work perfectly until the design is stable. But on actively developing projects, layout changes every week. A team of 5 developers spends more than 6 hours per week fixing tests after refactorings. LLM does not replace Playwright. It solves this pain: instead of hardcoded locators — semantic search. Automatic recovery of failed tests. Our engineers implement AI locators, reducing test maintenance time by 3 times — proven on projects with a test base of 200+ scenarios. Time savings of 6 hours per week per team, paying off within 2-3 months. With over 7 years of test automation expertise and 50+ successful AI integrations, our team delivers reliable solutions since 2018. Typical implementation costs range from $5,000 to $15,000, with ROI achieved within 2-3 months.

Value of LLM in testing

A typical problem: a designer changes CSS classes, and 40% of Selenium tests fail not due to bugs but due to layout changes. The team stops trusting CI — red status is ignored. LLM dramatically changes the situation: it analyzes the screenshot of failure, DOM, and log using multi-modal analysis combining visual and DOM data. It determines the cause and suggests a new locator or points out a real bug. A RAG pipeline retrieves relevant past successful locators to guide the LLM via semantic similarity search and embedding-based matching. Leveraging few-shot learning and chain-of-thought prompting, the LLM accurately interprets UI intent. Result — developers trust CI again. Time to analyze a failed test is reduced from 1 hour to 5 minutes. LLM-generated tests are 10x faster than manual writing.

AI-fallback test recovery

Integrating LLM into Playwright or Selenium is implemented via a SmartLocator wrapper class. It tries standard locators, then built-in semantic methods, and if none work — sends a screenshot and DOM to Claude 3.5 Haiku. The model returns a CSS selector. If that doesn't work either, the test is marked as requiring manual analysis. This AI-fallback succeeds in 95% of cases when standard locators fail.

from playwright.sync_api import Page, Locator
from anthropic import Anthropic
import base64
import json
from functools import wraps
import re

client = Anthropic()

class SmartLocator:
    """Smart locator with AI-fallback"""
    def __init__(self, page: Page):
        self.page = page
        self._locator_cache: dict[str, str] = {}

    def find(self, description: str, prefer_selector: str = None) -> Locator:
        """Finds element by description, caching successful locators"""
        # 1. Try cached locator
        if description in self._locator_cache:
            selector = self._locator_cache[description]
            try:
                loc = self.page.locator(selector)
                if loc.count() > 0 and loc.first.is_visible(timeout=500):
                    return loc.first
            except Exception:
                del self._locator_cache[description]

        # 2. Try preferred locator
        if prefer_selector:
            try:
                loc = self.page.locator(prefer_selector)
                if loc.count() > 0:
                    self._locator_cache[description] = prefer_selector
                    return loc.first
            except Exception:
                pass

        # 3. Playwright built-in semantics
        semantic_attempts = [
            lambda: self.page.get_by_role("button", name=re.sub(r"кнопк[аиу] ", "", description, flags=re.I)),
            lambda: self.page.get_by_label(description),
            lambda: self.page.get_by_placeholder(description),
            lambda: self.page.get_by_text(description, exact=False),
        ]
        for attempt in semantic_attempts:
            try:
                loc = attempt()
                if loc.count() > 0 and loc.first.is_visible(timeout=500):
                    return loc.first
            except Exception:
                continue

        # 4. AI locator generation
        return self._ai_find_element(description)

    def _ai_find_element(self, description: str) -> Locator:
        """Uses LLM to find element by screenshot and DOM"""
        screenshot_bytes = self.page.screenshot()
        dom_snippet = self.page.evaluate("""
            () => {
                const elements = document.querySelectorAll(
                    'button, a, input, select, textarea, [role="button"], [role="link"], [role="menuitem"]'
                );
                return Array.from(elements).slice(0, 60).map(el => ({
                    tag: el.tagName.toLowerCase(),
                    text: el.textContent?.trim().slice(0, 60) || '',
                    id: el.id || '',
                    class: el.className?.toString().slice(0, 60) || '',
                    type: (el as any).type || '',
                    name: (el as any).name || '',
                    placeholder: (el as any).placeholder || '',
                    aria_label: el.getAttribute('aria-label') || '',
                    data_testid: el.getAttribute('data-testid') || '',
                })).filter(e => e.text || e.placeholder || e.aria_label);
            }
        """)
        response = client.messages.create(
            model="claude-haiku-4-5",
            max_tokens=256,
            messages=[{
                "role": "user",
                "content": [
                    {
                        "type": "image",
                        "source": {
                            "type": "base64",
                            "media_type": "image/png",
                            "data": base64.b64encode(screenshot_bytes).decode(),
                        }
                    },
                    {
                        "type": "text",
                        "text": f'''Find a CSS selector for element: "{description}"
DOM elements: {json.dumps(dom_snippet[:30], ensure_ascii=False)}
Return JSON: {{"selector": "css_selector", "confidence": 0.0-1.0}}
Prefer: [data-testid="..."], #id, [aria-label="..."]
Only JSON.'''
                    }
                ]
            }],
        )
        try:
            text = response.content[0].text
            result = json.loads(text[text.find("{"):text.rfind("}") + 1])
            selector = result["selector"]
            loc = self.page.locator(selector)
            if loc.count() > 0:
                self._locator_cache[description] = selector
                return loc.first
        except Exception:
            pass
        raise RuntimeError(f"Element not found: {description}")

class SelfHealingTest:
    """Test with self-healing locators"""
    def __init__(self, page: Page):
        self.page = page
        self.smart = SmartLocator(page)
        self.failed_steps: list = []

    def step(self, description: str):
        """Decorator for test steps with AI recovery"""
        def decorator(func):
            @wraps(func)
            def wrapper(*args, **kwargs):
                try:
                    return func(*args, **kwargs)
                except Exception as e:
                    # Try AI recovery
                    recovery = self._attempt_recovery(description, str(e))
                    if recovery:
                        return recovery
                    self.failed_steps.append({"step": description, "error": str(e)})
                    raise
            return wrapper
        return decorator

    def _attempt_recovery(self, step_description: str, error: str):
        """Attempt to recover a failed step"""
        screenshot_bytes = self.page.screenshot()
        response = client.messages.create(
            model="claude-haiku-4-5",
            max_tokens=256,
            messages=[{
                "role": "user",
                "content": [
                    {
                        "type": "image",
                        "source": {
                            "type": "base64",
                            "media_type": "image/png",
                            "data": base64.b64encode(screenshot_bytes).decode(),
                        }
                    },
                    {
                        "type": "text",
                        "text": f'''Test failed at step: "{step_description}"
Error: {error}
Do you see anything on the screenshot that blocks execution?
Return JSON: {{"blocker": "description of problem or null", "recovery_selector": "css or null"}}'''
                    }
                ]
            }],
        )
        try:
            text = response.content[0].text
            result = json.loads(text[text.find("{"):text.rfind("}") + 1])
            if result.get("recovery_selector"):
                self.page.click(result["recovery_selector"])
                return True
        except Exception:
            pass
        return None

LLM generation of tests from user stories

def generate_playwright_test(user_story: str, base_url: str, page_html: str = "") -> str:
    """Generates a Playwright test from user story"""
    response = client.messages.create(
        model="claude-sonnet-4-5",
        max_tokens=2048,
        messages=[{
            "role": "user",
            "content": f'''Generate a Playwright Python test for user story: {user_story}
Base URL: {base_url}
{"HTML context of page: " + page_html[:2000] if page_html else ""}
Requirements:
- Use Playwright best practices: get_by_role, get_by_label, get_by_text
- Explicit expect() with timeout
- No hardcoded sleep()
- Comments for each step
- Use data-testid if visible in HTML
Format: only Python code, no explanations.'''
        }],
    )
    return response.content[0].text

# Example usage
story = """
As a user, I want to log in:
1. Go to /login
2. Enter email [email protected]
3. Enter password TestPass123
4. Click "Login" button
5. See dashboard page with text "Welcome"
"""
test_code = generate_playwright_test(story, "https://app.example.com")
print(test_code)

AI analysis of failed tests

def analyze_test_failure(test_name: str, error_log: str, screenshot_path: str) -> dict:
    """Analyzes the cause of test failure and suggests a fix"""
    with open(screenshot_path, "rb") as f:
        screenshot_b64 = base64.b64encode(f.read()).decode()
    response = client.messages.create(
        model="claude-sonnet-4-5",
        max_tokens=1024,
        messages=[{
            "role": "user",
            "content": [
                {
                    "type": "image",
                    "source": {"type": "base64", "media_type": "image/png", "data": screenshot_b64}
                },
                {
                    "type": "text",
                    "text": f'''Test failed: {test_name}
Error log: {error_log[:2000]}
Analyze the screenshot and log. Return JSON: {{
    "root_cause": "brief description of cause",
    "is_app_bug": true/false,
    "is_test_bug": true/false,
    "suggested_fix": "how to fix test or bug",
    "new_selector": "if problem is locator — new CSS selector"
}}'''
                }
            ]
        }],
    )
    text = response.content[0].text
    return json.loads(text[text.find("{"):text.rfind("}") + 1])

Quick implementation of AI locators in an existing project

Let's compare two approaches — without LLM and with LLM.

Practical case: e-commerce, 350 tests

Situation: active frontend development (React + Tailwind). Designer changed CSS classes every 2 weeks. 40% of Selenium tests failed not due to bugs but due to layout changes. CI/CD pipeline showed red, team ignored.

We changed (from our practice):

• Migrated 80 most unstable tests to SmartLocator with AI-fallback.
• Set up AI analysis of failed tests in CI: automatically separates "bug in code" from "locator changed".
• LLM generation of test skeletons for new user stories.

Results:

• False red tests (UI changed, not bug): 40% → 6%.
• Time to analyze failed tests in CI: 2 h/deploy → 20 min.
• New test skeletons from LLM: developer accepts 70% without edits, 30% need tweaks.
• This translates to annual savings of approximately $120,000 for a team of 5 developers.

Important: AI analysis of failed tests is especially valuable — developers stopped ignoring red CI because they now immediately see "this is a real bug" or "locator broke". Reduces QA costs by up to 40%.

What's included in the work

• Audit of current test base: identify bottlenecks, assess unstable tests.
• Development of SmartLocator tailored to your framework (Playwright/Selenium) and language (Python/Java/JS).
• Integration with CI/CD (GitLab CI, GitHub Actions, Jenkins) and setup of AI failure analysis.
• Documentation on using and maintaining AI locators.
• Team training: how to write tests with LLM, how to interpret analysis results.
• Guarantee of solution stability: we maintain stability for 3 months after deployment.

Timelines

• SmartLocator + AI-fallback for existing test base: from 1 week.
• AI test generation + review pipeline: from 1 week.
• CI/CD integration with failure analysis: 3–5 days.
• Full system for large test base: 3–4 weeks.

Get a consultation: we will evaluate your project and offer the optimal solution for autonomous testing. Order implementation — contact us to discuss details.

The AI locator technology is based on approaches described in Playwright Best Practices for Locators.