OpenAI released PaperBench to evaluate agents replicating AI research
OpenAI released PaperBench on April 2, 2025, using ICML papers, hierarchical rubrics and an open implementation to evaluate research replication.
Historical reporting: OpenAI released PaperBench on April 2, 2025. Its launch describes 20 ICML 2024 Spotlight and Oral papers, 8,316 gradable tasks, author-co-developed rubrics, an LLM judge and a human comparison subset. The linked public repository documents a three-container rollout, reproduction and grading path plus dated baseline rows. OpenAI’s reported model and human comparisons are publisher-run results for this setup; they do not show that an agent can independently select, validate, communicate or safely deploy a new research program.
Key takeaways
- PaperBench evaluated a reproducible research artifact, not only a paper summary or code snippet.
- Its staged execution path separated agent creation, reproduction and rubric-based grading.
- Reported scores remain conditional on paper selection, compute, rubric design, judge behavior and agent scaffolding.
OpenAI dated PaperBench to April 2 and framed research replication as the task[source 1]
OpenAI released PaperBench on April 2, 2025 as a benchmark for agents attempting to replicate state-of-the-art AI research. The launch specifies 20 ICML 2024 Spotlight and Oral papers and says an agent must understand a paper’s contribution, build a codebase and execute experiments. That is a material change from asking for a textual explanation or an isolated program. The launch also reports 8,316 individually gradable tasks and rubrics co-developed with paper authors. These details establish what PaperBench claimed to measure on that release date. They do not show that an agent can originate a useful research direction, manage unknown data conditions or deliver a result that another laboratory would accept.
The public implementation made rollout, reproduction and grading separate stages[source 2]
OpenAI’s repository describes three stages. First, an agent works in an Ubuntu container to create a submission. Second, that codebase runs in a fresh GPU-enabled container to produce an executed submission. Third, another container runs the paper’s rubric against the executed result. Separating those stages makes a useful distinction: producing code is not the same as reproducing an experiment, and running an experiment is not the same as satisfying a grading rubric. It also exposes practical dependencies such as data hydration and required access for some papers. A result is therefore inseparable from the container image, GPU setup, retrieved data, permitted credentials, timeout and software revisions used for that run.
Hierarchical rubrics made many research subtasks scorable but did not remove grading judgment[source 1][source 2]
PaperBench decomposes each replication task into smaller rubric criteria and uses an LLM-based judge for scalable grading. OpenAI says it also built a separate benchmark to assess that judge. This makes the judge a component under evaluation, rather than an invisible authority. The approach can surface whether an agent produced artifacts associated with a specific paper’s expected steps, while a rubric still reflects choices about what counts as a faithful replication. A judge’s agreement with a reference process does not prove the result is novel, correct under all conditions or valuable outside that paper. Readers should distinguish the benchmark’s published grading protocol from a peer-review decision or a claim of scientific discovery.
PaperBench separates a reproduction attempt from the rubric that decides which research subtasks count[source 1][source 2]
PaperBench’s result is built from hierarchical rubrics that break each ICML-paper replication into smaller, individually gradable tasks. OpenAI says the rubrics were co-developed with the original authors, then scored at scale with an LLM-based judge whose performance was itself assessed on a separate judge benchmark. That design makes the unit of evidence more specific than a headline replication score: a run can be examined for which artifacts and subtasks matched the rubric, and the grading layer remains an explicit component. The 21.0% best-tested result in the release belongs to that complete setup—selected papers, agent scaffolding, executed artifacts and rubric judge. A later comparison should preserve those elements, because changing a paper set or rubric depth changes what a replication percentage measures.
Sources
- OpenAI: PaperBench launch (April 2, 2025)
- OpenAI: PaperBench public repository
Continue reading
