{
  "schema_version": "rangoon.article.v1",
  "id": "https://rangoon.ai/insights/archive-openai-paperbench-ai-research-replication-april-2025/",
  "url": "https://rangoon.ai/insights/archive-openai-paperbench-ai-research-replication-april-2025/",
  "markdown_url": "https://rangoon.ai/insights/archive-openai-paperbench-ai-research-replication-april-2025/index.md",
  "json_url": "https://rangoon.ai/insights/archive-openai-paperbench-ai-research-replication-april-2025/index.json",
  "title": "OpenAI released PaperBench to evaluate agents replicating AI research",
  "summary": "OpenAI released PaperBench on April 2, 2025, using ICML papers, hierarchical rubrics and an open implementation to evaluate research replication.",
  "language": "en",
  "author": {
    "name": "Rangoon Editorial",
    "url": "https://rangoon.ai/insights/editorial/"
  },
  "publisher": {
    "name": "Rangoon",
    "maintainer": "Hypler",
    "url": "https://rangoon.ai/"
  },
  "format": "News analysis",
  "topic": {
    "slug": "agents",
    "label": "Agent systems",
    "url": "https://rangoon.ai/insights/topics/agents/"
  },
  "tags": [
    {
      "slug": "openai",
      "label": "OpenAI",
      "url": "https://rangoon.ai/insights/tags/openai/"
    },
    {
      "slug": "ai-evaluation",
      "label": "AI evaluation",
      "url": "https://rangoon.ai/insights/tags/ai-evaluation/"
    },
    {
      "slug": "agent-benchmarks",
      "label": "agent benchmarks",
      "url": "https://rangoon.ai/insights/tags/agent-benchmarks/"
    },
    {
      "slug": "model-evaluation",
      "label": "model evaluation",
      "url": "https://rangoon.ai/insights/tags/model-evaluation/"
    }
  ],
  "dates": {
    "published": "2026-10-24",
    "updated": "2026-10-24",
    "event": "2025-04-02"
  },
  "event_source_id": 1,
  "retrospective": true,
  "reading_minutes": 3,
  "takeaways": [
    "PaperBench evaluated a reproducible research artifact, not only a paper summary or code snippet.",
    "Its staged execution path separated agent creation, reproduction and rubric-based grading.",
    "Reported scores remain conditional on paper selection, compute, rubric design, judge behavior and agent scaffolding."
  ],
  "source_note": "Historical reporting: OpenAI released PaperBench on April 2, 2025. Its launch describes 20 ICML 2024 Spotlight and Oral papers, 8,316 gradable tasks, author-co-developed rubrics, an LLM judge and a human comparison subset. The linked public repository documents a three-container rollout, reproduction and grading path plus dated baseline rows. OpenAI’s reported model and human comparisons are publisher-run results for this setup; they do not show that an agent can independently select, validate, communicate or safely deploy a new research program.",
  "sources": [
    {
      "id": 1,
      "title": "OpenAI: PaperBench launch",
      "url": "https://openai.com/index/paperbench/",
      "published": "2025-04-02"
    },
    {
      "id": 2,
      "title": "OpenAI: PaperBench public repository",
      "url": "https://github.com/openai/frontier-evals/tree/main/project/paperbench",
      "published": null
    }
  ],
  "image": {
    "url": "https://rangoon.ai/assets/editorial/archive-openai-paperbench-ai-research-replication-april-2025-diagram-v1.webp",
    "thumbnail_url": "https://rangoon.ai/assets/editorial/archive-openai-paperbench-ai-research-replication-april-2025-diagram-v1-thumb.webp",
    "alt": "Original evaluation explanatory diagram, Research paper replicated through three evaluation stages. Labels: Paper, Agent, Codebase, Run, Rubric, Judge. Source boundary: dated primary release reporting; original editorial analysis.",
    "type": "original-diagram",
    "caption": "Research paper replicated through three evaluation stages. Original Rangoon explanatory diagram: Source boundary: dated primary release reporting; original editorial analysis. No vendor interface, photograph, measurement result, or endorsement implied.",
    "credit": "Rangoon / Hypler",
    "source_url": "https://rangoon.ai/brand/",
    "rights_url": "https://rangoon.ai/terms/#media-use",
    "rights_note": "Hypler artwork permissions do not grant rights to third-party marks or imply endorsement."
  },
  "sections": [
    {
      "id": "launch-scope",
      "url": "https://rangoon.ai/insights/archive-openai-paperbench-ai-research-replication-april-2025/#launch-scope",
      "heading": "OpenAI dated PaperBench to April 2 and framed research replication as the task",
      "paragraphs": [
        "OpenAI released PaperBench on April 2, 2025 as a benchmark for agents attempting to replicate state-of-the-art AI research. The launch specifies 20 ICML 2024 Spotlight and Oral papers and says an agent must understand a paper’s contribution, build a codebase and execute experiments. That is a material change from asking for a textual explanation or an isolated program. The launch also reports 8,316 individually gradable tasks and rubrics co-developed with paper authors. These details establish what PaperBench claimed to measure on that release date. They do not show that an agent can originate a useful research direction, manage unknown data conditions or deliver a result that another laboratory would accept."
      ],
      "bullets": [],
      "source_ids": [
        1
      ]
    },
    {
      "id": "three-stage-path",
      "url": "https://rangoon.ai/insights/archive-openai-paperbench-ai-research-replication-april-2025/#three-stage-path",
      "heading": "The public implementation made rollout, reproduction and grading separate stages",
      "paragraphs": [
        "OpenAI’s repository describes three stages. First, an agent works in an Ubuntu container to create a submission. Second, that codebase runs in a fresh GPU-enabled container to produce an executed submission. Third, another container runs the paper’s rubric against the executed result. Separating those stages makes a useful distinction: producing code is not the same as reproducing an experiment, and running an experiment is not the same as satisfying a grading rubric. It also exposes practical dependencies such as data hydration and required access for some papers. A result is therefore inseparable from the container image, GPU setup, retrieved data, permitted credentials, timeout and software revisions used for that run."
      ],
      "bullets": [],
      "source_ids": [
        2
      ]
    },
    {
      "id": "rubrics-and-judge",
      "url": "https://rangoon.ai/insights/archive-openai-paperbench-ai-research-replication-april-2025/#rubrics-and-judge",
      "heading": "Hierarchical rubrics made many research subtasks scorable but did not remove grading judgment",
      "paragraphs": [
        "PaperBench decomposes each replication task into smaller rubric criteria and uses an LLM-based judge for scalable grading. OpenAI says it also built a separate benchmark to assess that judge. This makes the judge a component under evaluation, rather than an invisible authority. The approach can surface whether an agent produced artifacts associated with a specific paper’s expected steps, while a rubric still reflects choices about what counts as a faithful replication. A judge’s agreement with a reference process does not prove the result is novel, correct under all conditions or valuable outside that paper. Readers should distinguish the benchmark’s published grading protocol from a peer-review decision or a claim of scientific discovery."
      ],
      "bullets": [],
      "source_ids": [
        1,
        2
      ]
    },
    {
      "id": "rubric-reproduction",
      "url": "https://rangoon.ai/insights/archive-openai-paperbench-ai-research-replication-april-2025/#rubric-reproduction",
      "heading": "PaperBench separates a reproduction attempt from the rubric that decides which research subtasks count",
      "paragraphs": [
        "PaperBench’s result is built from hierarchical rubrics that break each ICML-paper replication into smaller, individually gradable tasks. OpenAI says the rubrics were co-developed with the original authors, then scored at scale with an LLM-based judge whose performance was itself assessed on a separate judge benchmark. That design makes the unit of evidence more specific than a headline replication score: a run can be examined for which artifacts and subtasks matched the rubric, and the grading layer remains an explicit component. The 21.0% best-tested result in the release belongs to that complete setup—selected papers, agent scaffolding, executed artifacts and rubric judge. A later comparison should preserve those elements, because changing a paper set or rubric depth changes what a replication percentage measures."
      ],
      "bullets": [],
      "source_ids": [
        1,
        2
      ]
    }
  ],
  "project_context": null,
  "related_links": [],
  "permissions_url": "https://rangoon.ai/permissions.txt",
  "content_hash": "sha256:7bc5b63ff69b0dacec8f7eb3a8315cc14fd4c5fca8286d861cb0d942d026972a"
}