AI-Generated Unit Tests: Automating Coverage and Regression

The codebase grows, coverage drops, refactoring becomes a risky venture. Teams spend up to 40% of each sprint writing tests—and still miss edge cases. We automated this process via AI generation: the system analyzes AST, extracts functions, arguments, exceptions, and return types, then an LLM (Claude Sonnet 4.5) generates pytest tests. Unlike manual approaches, AI doesn't forget boundary conditions—null arguments, empty collections, invalid combinations. Result: 80% time savings for QA (and corresponding cost reduction on manual testing). ROI on AI generation investment is less than 3 months under typical team load.

How AI Handles Legacy Code Without Types?

Even if the code is written without type annotations, the AST parser extracts signatures and return values. We additionally analyze docstrings, if-conditions, and raise expressions. This information is passed to the model together with the context of existing tests (if any) to keep the style consistent. The model returns a ready test file.

from anthropic import Anthropic import ast import inspect from pathlib import Path from typing import Optional import subprocess client = Anthropic() class TestGenerator: def __init__(self, project_root: str): self.project_root = project_root def extract_function_info(self, source_code: str, function_name: str) -> dict: """Extracts function metadata via AST""" tree = ast.parse(source_code) for node in ast.walk(tree): if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)): if node.name == function_name: return { "name": node.name, "args": [arg.arg for arg in node.args.args], "decorators": [ast.unparse(d) for d in node.decorator_list], "is_async": isinstance(node, ast.AsyncFunctionDef), "has_return": any( isinstance(n, ast.Return) and n.value for n in ast.walk(node) ), "raises": [ ast.unparse(n.exc) for n in ast.walk(node) if isinstance(n, ast.Raise) and n.exc ], "source": ast.unparse(node), } return {} def find_related_tests(self, source_file: str) -> str: """Finds existing tests to understand style""" source_path = Path(source_file) test_candidates = [ source_path.parent / f"test_{source_path.name}", source_path.parent.parent / "tests" / f"test_{source_path.name}", source_path.parent / "tests" / f"test_{source_path.name}", ] for test_file in test_candidates: if test_file.exists(): return test_file.read_text()[:2000] return "" def generate_tests( self, source_file: str, function_name: Optional[str] = None, ) -> str: """Generates tests for a file or specific function""" source_code = Path(source_file).read_text() existing_tests = self.find_related_tests(source_file) if function_name: func_info = self.extract_function_info(source_code, function_name) context = f"Function to test:\n```python\n{func_info.get('source', '')}\n```" else: context = f"File to test:\n```python\n{source_code[:4000]}\n```" existing_context = "" if existing_tests: existing_context = f"\nExisting test style (follow this pattern):\n```python\n{existing_tests}\n```" response = client.messages.create( model="claude-sonnet-4-5", max_tokens=4096, system="""You are a senior developer writing pytest tests. Rules: - Test behavior, not implementation - One test = one assertion (AAA: Arrange, Act, Assert) - Name tests as: test_<function>_<scenario>_<expectation> - Cover: happy path, edge cases, errors/exceptions, boundary values - Use pytest.mark.parametrize for similar tests - For async functions — pytest-asyncio - Mock external dependencies via pytest-mock""", messages=[{ "role": "user", "content": f"""{context}{existing_context}\n\nGenerate a complete test file with pytest. Return only code, no explanations.""" }] ) return response.content[0].text 

Why Mutation Testing Is the Only Objective Criterion?

Generated tests need to be verified: do they catch real bugs? Mutation testing introduces mutations into the source code—changes > to <, True to False, removes calls. If a test doesn't fail, the mutation survives. The higher the mutation score (ratio of killed mutants), the more reliable the tests. Our target is 80% and above.

import subprocess from pathlib import Path def evaluate_test_quality(source_file: str, test_file: str) -> dict: """Runs mutation testing to evaluate test quality""" result = subprocess.run( ["mutmut", "run", f"--paths-to-mutate={source_file}", f"--tests-dir={test_file}"], capture_output=True, text=True, timeout=300 ) survived = 0 killed = 0 for line in result.stdout.splitlines(): if "survived" in line.lower(): survived += 1 elif "killed" in line.lower(): killed += 1 total = survived + killed mutation_score = killed / total if total > 0 else 0 return { "mutation_score": mutation_score, "killed_mutants": killed, "survived_mutants": survived, "verdict": "excellent" if mutation_score > 0.8 else "good" if mutation_score > 0.6 else "needs_improvement" } 

Comparison of Approaches: Manual vs AI Generation

AI generation is 80% faster than manual test writing with comparable coverage. And with the auto-fix cycle, we achieve a mutation score >0.8—a level rarely achieved manually.

Work Stages

1. Codebase analysis. AST traversal of all files: extract function signatures, decorators, raise expressions, docstrings. Estimate volume: an average project has 50-100 functions per module.
2. Test generation. Each file gets a separate test file with parameterized tests covering happy path, edge cases, and exceptions.
3. Auto-run and fix cycle. Up to 3 iterations: run pytest, parse errors, refine tests via LLM. Tests that fail due to external dependencies are flagged for manual tuning.
4. Mutation score evaluation. Run mutmut, analyze surviving mutants. If score <0.8, generate additional tests for weak spots.
5. CI integration. Ready script for GitHub Actions or GitLab CI with coverage gate and automatic report.