Amazon introduced SWE-PolyBench to broaden coding-agent evaluation beyond Python repositories
Amazon introduced SWE-PolyBench on April 23, 2025, with 2,110 repository issues across Java, JavaScript, TypeScript and Python.
Historical reporting: Amazon introduced SWE-PolyBench on April 23, 2025 as a multi-language repository-level coding-agent benchmark. Its AWS post describes 2,110 instances from 21 repositories, four languages, a 500-issue stratified subset, fail-to-pass test selection, pass-rate metrics and file/CST-node localization metrics. Amazon’s public repository provides the stated evaluation harness and dataset structure. Amazon’s comparisons and open-agent findings are publisher-run benchmark results; they do not prove performance in a private codebase, under another agent harness, or with different dependencies, test commands, model versions, context budgets or deployment controls.
Key takeaways
- SWE-PolyBench expanded repository-level coding tasks across Java, JavaScript, TypeScript and Python.
- Its metrics separate a passing patch from locating relevant files and code structures.
- A benchmark result remains tied to its task set, language distribution, containers, test commands and agent configuration.
Amazon dated the benchmark announcement to April 23 and set a multi-language scope[source 1]
Amazon introduced SWE-PolyBench in an AWS developer blog post dated April 23, 2025. The post presents the benchmark as a repository-level evaluation for coding agents across Java, JavaScript, TypeScript and Python, responding to the Python concentration and bug-fix skew Amazon identified in earlier SWE-Bench work. Amazon reported 2,110 instances from 21 repositories and a 500-issue stratified subset for faster experimentation. Those counts and language categories describe Amazon’s published collection; they do not represent all software ecosystems or every kind of maintenance work. The event is a benchmark and harness release, not a claim that an agent can take responsibility for a repository, infer an organization’s unrecorded conventions, or safely merge a change.
The collection used issue-closing pull requests and fail-to-pass tests to define executable tasks[source 1]
Amazon says its data pipeline gathered pull requests that closed issues in real repositories, filtered them, and set them up in containerized environments for consistent test execution. The post distinguishes fail-to-pass from pass-to-pass tests and says only pull requests with at least one fail-to-pass test entered the final data set. That gives each included task an executable behavioral target: a correct patch must change at least one test outcome from failure to passing. It is a stronger target than judging a patch by surface resemblance, while it is not a universal definition of correctness. A test suite may omit performance, security, backward compatibility, migration, user-experience, documentation, operational and review requirements. Container setup and dependency versions also become part of what the score measures.
Amazon separated patch success from file and syntax-tree localization measurements[source 1][source 2]
SWE-PolyBench retains pass rate as its primary outcome: the proportion of tasks whose generated patches pass relevant tests. Amazon also introduced file-level localization and concrete-syntax-tree node retrieval measures. The first asks whether an agent identified the files that needed changes; the second assesses whether it located relevant code structures such as functions or classes. These measures answer different questions. A patch can pass tests after touching the wrong place, and an agent can identify a relevant node without constructing a working patch. Amazon’s public repository identifies the benchmark as a multi-language evaluation project and exposes its data, scripts and evaluation package. The reported measures can help diagnose a failure mode, but they do not independently establish code quality, intent preservation or production readiness.
Amazon’s open-agent results were conditioned on its adapted harnesses and task categories[source 1][source 2]
Amazon reported that its evaluated open-source agents performed best in Python and degraded as tasks required more files or greater structural complexity. It also described adapting test commands, parsers and containerization for the four languages, and reported differences among bug fixes, feature requests and refactoring tasks. These are useful publisher observations because they name the task, language and harness conditions. They are not independent evidence that a model will show the same ranking in a private project. A private repository may have other languages, build systems, fixtures, permissions, security rules, issue quality and acceptance criteria. Anyone using the benchmark to choose an agent should preserve the benchmark revision and run a separate evaluation on representative work before treating a public leaderboard result as an operating decision.
Sources
Continue reading
