AI-driven test failure analysis and error classification system

Note: When dozens of tests fail simultaneously in CI, root cause analysis takes hours. Engineers scroll through logs trying to understand what broke: infrastructure, regression, or a flaky test. We trained a model to automatically group failures by real cause and route tickets to the responsible developer. Our AI system processes stack traces 5–10 times faster than manual analysis, reducing time from failure to fix from hours to minutes. Implementing this approach saves a team up to 80 person-hours per month on CI failure triage.

The system doesn't just find errors — it prioritizes defects and automatically routes notifications. Instead of alerting the entire team, only the person who can actually fix the problem receives the notification.

How does error classification work?

The pipeline consists of four stages: parsing, classification, clustering, and routing. Input: raw test logs. Output: structured reports with priority and owner.

Parsing — supports JUnit XML, Allure, pytest JSON, Cypress Mochawesome. Normalizes into a unified schema: {test_id, status, duration, error_message, stack_trace, timestamp}.

Classification — a multi-label classifier based on fine-tuned CodeBERT, trained on labeled stack traces. Output classes:

• INFRASTRUCTURE — timeouts, connection refused, OOM
• REGRESSION — test passed, broke after a commit
• FLAKY — unstable (passes with retry)
• TEST_BUG — error in the test itself, not in the code
• NEW_BUG — real defect in the product
• ENVIRONMENT — problem with test environment

Clustering — stack trace embeddings via CodeBERT → HDBSCAN clustering. Tests in the same cluster likely share the same root cause.

from sentence_transformers import SentenceTransformer
import hdbscan

class ErrorClusterer:
    def __init__(self):
        self.encoder = SentenceTransformer('microsoft/codebert-base')
        self.clusterer = hdbscan.HDBSCAN(
            min_cluster_size=3,
            metric='cosine',
            cluster_selection_method='eom'
        )

    def cluster_failures(self, failures: list[dict]) -> list[int]:
        texts = [f['error_message'] + '\n' + f['stack_trace'][:500] for f in failures]
        embeddings = self.encoder.encode(texts, batch_size=32)
        return self.clusterer.fit_predict(embeddings)

Why use AI test analysis?

Manual failure triage is time-consuming and error-prone. Fine-tuned CodeBERT processes stack traces 5 times faster than a human, and clustering reveals cascading failures that appear as dozens of independent errors. We guarantee: after deployment, you will halve your CI failure triage time. Our experience shows that within 2–3 weeks the system reaches target quality metrics.

How HDBSCAN clustering reveals cascading failures?

Cascading failures — one breakdown (e.g., database outage) causes dozens of test failures. Clustering groups them by stack trace and error message similarity. Even if errors appear different (connection timeout, NULL pointer, 500 error), HDBSCAN merges them into one cluster if embeddings are close enough. This immediately reveals that all failures stem from a single issue.

Flaky test detection

Flaky tests form a separate category. The system analyzes the history of the last 30 runs for each test and computes a flakiness score:

flakiness_score = std(pass_rate_per_day) * frequency_of_status_changes 

A test with flakiness_score > 0.3 is marked as flaky and automatically quarantined — it continues running for statistics but does not block CI.

Notification routing

The system determines the owner of the failing test via git blame on the test file and files from the stack trace. Notifications are sent not to the whole team, but:

• REGRESSION/NEW_BUG → author of the last commit on affected files + team lead
• FLAKY → test owner from git blame
• INFRASTRUCTURE → DevOps channel
• TEST_BUG → QA responsible for that test file

Classification quality metrics

Training is performed on a labeled dataset from the project history. Typical metrics after 2–3 weeks of data accumulation:

What is included in the work?

• Audit of the current CI process and report formats
• Parsing and log normalization setup
• Classifier training on your historical data
• Integration with GitLab, GitHub Actions, or Jenkins
• Dashboard (Grafana) and webhook notifications deployment
• Documentation and team training

Deployment process

1. Analytics — metric collection, defining error types
2. Design — stack selection, pipeline configuration
3. Implementation — writing parsers, model training
4. Testing — A/B comparison with manual analysis
5. Deployment — staging, then production

Estimated timelines

Pricing is calculated individually after an audit. We will estimate your project in one business day — contact us for a consultation. Get a demo access to the system and verify its effectiveness on your own data.