# OpenAI released PaperBench to evaluate agents replicating AI research OpenAI released PaperBench on April 2, 2025, using ICML papers, hierarchical rubrics and an open implementation to evaluate research replication. Canonical: https://rangoon.ai/insights/archive-openai-paperbench-ai-research-replication-april-2025/ Author: Rangoon Editorial (https://rangoon.ai/insights/editorial/) Published: 2026-10-24 Updated: 2026-10-24 Event date: 2025-04-02 Tags: [OpenAI](https://rangoon.ai/insights/tags/openai/), [AI evaluation](https://rangoon.ai/insights/tags/ai-evaluation/), [agent benchmarks](https://rangoon.ai/insights/tags/agent-benchmarks/), [model evaluation](https://rangoon.ai/insights/tags/model-evaluation/) Source note: Historical reporting: OpenAI released PaperBench on April 2, 2025. Its launch describes 20 ICML 2024 Spotlight and Oral papers, 8,316 gradable tasks, author-co-developed rubrics, an LLM judge and a human comparison subset. The linked public repository documents a three-container rollout, reproduction and grading path plus dated baseline rows. OpenAI’s reported model and human comparisons are publisher-run results for this setup; they do not show that an agent can independently select, validate, communicate or safely deploy a new research program. Image: Research paper replicated through three evaluation stages. Original Rangoon explanatory diagram: Source boundary: dated primary release reporting; original editorial analysis. No vendor interface, photograph, measurement result, or endorsement implied. ## Key takeaways - PaperBench evaluated a reproducible research artifact, not only a paper summary or code snippet. - Its staged execution path separated agent creation, reproduction and rubric-based grading. - Reported scores remain conditional on paper selection, compute, rubric design, judge behavior and agent scaffolding. ## OpenAI dated PaperBench to April 2 and framed research replication as the task [source 1](https://openai.com/index/paperbench/) OpenAI released PaperBench on April 2, 2025 as a benchmark for agents attempting to replicate state-of-the-art AI research. The launch specifies 20 ICML 2024 Spotlight and Oral papers and says an agent must understand a paper’s contribution, build a codebase and execute experiments. That is a material change from asking for a textual explanation or an isolated program. The launch also reports 8,316 individually gradable tasks and rubrics co-developed with paper authors. These details establish what PaperBench claimed to measure on that release date. They do not show that an agent can originate a useful research direction, manage unknown data conditions or deliver a result that another laboratory would accept. ## The public implementation made rollout, reproduction and grading separate stages [source 2](https://github.com/openai/frontier-evals/tree/main/project/paperbench) OpenAI’s repository describes three stages. First, an agent works in an Ubuntu container to create a submission. Second, that codebase runs in a fresh GPU-enabled container to produce an executed submission. Third, another container runs the paper’s rubric against the executed result. Separating those stages makes a useful distinction: producing code is not the same as reproducing an experiment, and running an experiment is not the same as satisfying a grading rubric. It also exposes practical dependencies such as data hydration and required access for some papers. A result is therefore inseparable from the container image, GPU setup, retrieved data, permitted credentials, timeout and software revisions used for that run. ## Hierarchical rubrics made many research subtasks scorable but did not remove grading judgment [source 1](https://openai.com/index/paperbench/) [source 2](https://github.com/openai/frontier-evals/tree/main/project/paperbench) PaperBench decomposes each replication task into smaller rubric criteria and uses an LLM-based judge for scalable grading. OpenAI says it also built a separate benchmark to assess that judge. This makes the judge a component under evaluation, rather than an invisible authority. The approach can surface whether an agent produced artifacts associated with a specific paper’s expected steps, while a rubric still reflects choices about what counts as a faithful replication. A judge’s agreement with a reference process does not prove the result is novel, correct under all conditions or valuable outside that paper. Readers should distinguish the benchmark’s published grading protocol from a peer-review decision or a claim of scientific discovery. ## PaperBench separates a reproduction attempt from the rubric that decides which research subtasks count [source 1](https://openai.com/index/paperbench/) [source 2](https://github.com/openai/frontier-evals/tree/main/project/paperbench) PaperBench’s result is built from hierarchical rubrics that break each ICML-paper replication into smaller, individually gradable tasks. OpenAI says the rubrics were co-developed with the original authors, then scored at scale with an LLM-based judge whose performance was itself assessed on a separate judge benchmark. That design makes the unit of evidence more specific than a headline replication score: a run can be examined for which artifacts and subtasks matched the rubric, and the grading layer remains an explicit component. The 21.0% best-tested result in the release belongs to that complete setup—selected papers, agent scaffolding, executed artifacts and rubric judge. A later comparison should preserve those elements, because changing a paper set or rubric depth changes what a replication percentage measures. ## Sources 1. [OpenAI: PaperBench launch](https://openai.com/index/paperbench/) 2. [OpenAI: PaperBench public repository](https://github.com/openai/frontier-evals/tree/main/project/paperbench) ## Related reading - [OpenAI released BrowseComp for hard-to-find web information tasks](https://rangoon.ai/insights/archive-openai-browsecomp-browsing-agent-benchmark-april-2025/) - [OpenAI introduced GDPval for one-shot professional knowledge-work deliverables](https://rangoon.ai/insights/archive-openai-gdpval-knowledge-work-evaluation-september-2025/) - [METR proposed task-completion time horizons for measuring long software work](https://rangoon.ai/insights/archive-metr-task-completion-time-horizons-march-2025/) ## More information - [Documentation index](https://rangoon.ai/llms.txt) - [AI access and policies](https://rangoon.ai/ai/) - [Source repository](https://github.com/hypler-dev/rangoon)