SWE-Bench Pro brought private and long-horizon code tasks into agent evaluation
Scale AI introduced SWE-Bench Pro on September 19, 2025, a 1,865-problem coding-agent benchmark spanning public, held-out, and commercial repositories.
Historical reporting: Scale AI announced SWE-Bench Pro on September 19, 2025. The associated paper, posted September 21, describes 1,865 problems from 41 repositories split across public, held-out, and commercial sets. Its launch scores used the SWE-Agent scaffold and are creator-reported results, not universal measures of software-engineering autonomy. The living code repository is later maintenance context only.
Key takeaways
- SWE-Bench Pro measures patch resolution from a codebase and issue, not whether an agent can safely own a production change.
- Private and held-out repositories limit direct public access, so public reproducibility and coverage differ across the benchmark’s three partitions.
- Launch scores depend on the named agent scaffold, tests, task set, and evaluation harness; they should not be compared as tool-free model traits.
The September launch widened the unit of coding evaluation[source 1][source 2]
On September 19, 2025, Scale AI announced SWE-Bench Pro as a benchmark for coding agents. The associated research record describes 1,865 problems from 41 repositories spanning business applications, B2B services, and developer tools. An agent receives a codebase and a natural-language issue, then is asked to produce a patch that resolves the problem and passes the repository test suite. That makes the evaluated output a code change under a defined harness, not a prose answer about how one might fix the issue.
The paper characterizes many tasks as long-horizon work that may take a professional engineer hours or days, often with multi-file patches and substantial modification. “May” is material here: task complexity varies by repository and issue. A benchmark can make a more demanding average workload visible without proving that every task maps to a typical production incident, architectural decision, security review, or release responsibility.
Three partitions separated public access from withheld evaluation[source 1][source 2]
SWE-Bench Pro was partitioned into a public set sourced from 11 repositories, a held-out set from 12 repositories, and a commercial set from 18 proprietary repositories with startup partners. The paper says held-out and commercial problems are not publicly accessible, though commercial-set results are released. This is a response to the risk that familiar public code can already appear in training data, while still leaving one portion available for outside inspection and experimentation.
That partitioning creates an interpretive boundary. Public tasks can be studied and reproduced by more researchers; results on held-out and commercial tasks rely more heavily on the maintainers’ protocol and reporting. A composite headline should identify which subset it represents. It cannot be treated as a fully open dataset score, and a private subset does not automatically guarantee absence from every training source or fairness across every language and codebase.
Human verification improves task definition but does not eliminate evaluation choices[source 2]
The paper says all tasks were human-verified and augmented with sufficient context to ensure resolvability. It also describes selecting strong-copyleft repositories for public and held-out sets and acquiring commercial codebases for the commercial set. Those are concrete data-collection choices intended to reduce contamination and increase realism. They do not turn an issue plus tests into a complete record of production constraints such as deployment windows, customer impact, security policy, undocumented system behavior, or code-owner judgment.
A patch that passes benchmark tests is therefore a defined success, not a release authorization. Software work commonly needs regression analysis, dependency review, compatibility checks, and a human decision about acceptable tradeoffs. SWE-Bench Pro can help differentiate agent performance on its verified tasks; it does not measure all controls that make a code change safe in a specific organization.
Launch scores belonged to one scaffolded evaluation[source 1][source 2]
Scale’s launch article reported public-dataset runs using the SWE-Agent scaffold. It said GPT-5 and Claude Opus 4.1 achieved 23.3% and 23.1% respectively on SWE-Bench Pro, while its paper describes results below 25% Pass@1 under a unified scaffold. These are creator-reported numbers with a named task set and agent wrapper. They should not be repeated as a general percentage of coding work a model can complete without further conditions.
The useful news is methodological: agent evaluation was being pushed toward repository-level, contamination-aware, multi-file problem solving. The limitation remains equally important. Change the scaffold, tool permissions, environment, time budget, tests, repository revision, or subset and the result may move. Current repository maintenance or future benchmark versions need separate attribution rather than being silently used to redefine the September 2025 launch.
Sources
- Scale AI: SWE-Bench Pro launch announcement (September 19, 2025)
- arXiv: SWE-Bench Pro (September 21, 2025)
Continue reading
Related reading
News analysisHumanity’s Last Exam released 3,000 expert-crafted questions for a hard academic test3 min read
News analysisOpenAI introduced SWE-Lancer to test coding agents against freelance software work3 min read
News analysisGLM-4.7 made preserved thinking a named control for multi-turn coding agents3 min read