SWT-Bench introduces a novel approach for automated test generation using LLM-based Code Agents to formalize user-reported issues into test cases and validate bug fixes.

• Creates test cases directly from natural language descriptions of bugs
• Evaluates the capabilities of Code Agents to understand, formalize, and fix software issues
• Provides a benchmark for measuring the effectiveness of automated testing approaches
• Demonstrates practical applications for improving software quality at scale

This research bridges a critical gap in software engineering by combining LLM capabilities with automated testing, potentially reducing development cycles and improving code quality while maintaining security standards.

SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents