{
  "schema_version": "rangoon.article.v1",
  "id": "https://rangoon.ai/insights/archive-swe-rebench-fresh-software-agent-evaluation-may-2025/",
  "url": "https://rangoon.ai/insights/archive-swe-rebench-fresh-software-agent-evaluation-may-2025/",
  "markdown_url": "https://rangoon.ai/insights/archive-swe-rebench-fresh-software-agent-evaluation-may-2025/index.md",
  "json_url": "https://rangoon.ai/insights/archive-swe-rebench-fresh-software-agent-evaluation-may-2025/index.json",
  "title": "SWE-rebench proposed continuously collected software-agent tasks to limit benchmark contamination",
  "summary": "The May 26, 2025 SWE-rebench paper introduced an automated pipeline for fresh executable Python tasks and a benchmark designed to separate generalization from known public issues.",
  "language": "en",
  "author": {
    "name": "Rangoon Editorial",
    "url": "https://rangoon.ai/insights/editorial/"
  },
  "publisher": {
    "name": "Rangoon",
    "maintainer": "Hypler",
    "url": "https://rangoon.ai/"
  },
  "format": "News analysis",
  "topic": {
    "slug": "agents",
    "label": "Agent systems",
    "url": "https://rangoon.ai/insights/topics/agents/"
  },
  "tags": [
    {
      "slug": "swe-rebench",
      "label": "SWE-rebench",
      "url": "https://rangoon.ai/insights/tags/swe-rebench/"
    },
    {
      "slug": "software-engineering",
      "label": "software engineering",
      "url": "https://rangoon.ai/insights/tags/software-engineering/"
    },
    {
      "slug": "agent-evaluation",
      "label": "Agent evaluation",
      "url": "https://rangoon.ai/insights/tags/agent-evaluation/"
    }
  ],
  "dates": {
    "published": "2026-10-24",
    "updated": "2026-10-24",
    "event": "2025-05-26"
  },
  "event_source_id": 1,
  "retrospective": true,
  "reading_minutes": 3,
  "takeaways": [
    "SWE-rebench paired a 21,336-task Python dataset with a smaller fresh-task benchmark intended to reduce exposure to old public instances.",
    "The pipeline used repository filtering, generated installation recipes, execution checks and automated quality assessment; each stage can affect what survives.",
    "Freshness does not make scores directly comparable when prompts, retries, tools, budgets and task solvability differ."
  ],
  "source_note": "Historical reporting: Source 0 is the pinned arXiv version 1 of SWE-rebench, dated May 26, 2025. Source 1 is the publisher-operated methodology page reviewed on October 24, 2026. It is a current technical record, used to explain stated scaffolding and solvability limits, not to alter the May paper event or replace version-1 findings.",
  "sources": [
    {
      "id": 1,
      "title": "SWE-rebench paper, pinned arXiv version 1",
      "url": "https://arxiv.org/abs/2505.20411v1",
      "published": "2025-05-26"
    },
    {
      "id": 2,
      "title": "SWE-rebench: benchmark methodology and limitations",
      "url": "https://swe-rebench.com/about",
      "published": null
    }
  ],
  "image": {
    "url": "https://rangoon.ai/assets/editorial/archive-swe-rebench-fresh-software-agent-evaluation-may-2025-diagram-v1.webp",
    "thumbnail_url": "https://rangoon.ai/assets/editorial/archive-swe-rebench-fresh-software-agent-evaluation-may-2025-diagram-v1-thumb.webp",
    "alt": "Original evaluation explanatory diagram, Fresh repository changes feed a staged benchmark. Labels: GitHub changes, License filter, Setup recipe, Execution check, Fresh task, Agent run. Source boundary: dated primary release reporting; original editorial analysis.",
    "type": "original-diagram",
    "caption": "Fresh repository changes feed a staged benchmark. Original Rangoon explanatory diagram: Source boundary: dated primary release reporting; original editorial analysis. No vendor interface, photograph, measurement result, or endorsement implied.",
    "credit": "Rangoon / Hypler",
    "source_url": "https://rangoon.ai/brand/",
    "rights_url": "https://rangoon.ai/terms/#media-use",
    "rights_note": "Hypler artwork permissions do not grant rights to third-party marks or imply endorsement."
  },
  "sections": [
    {
      "id": "paper-event",
      "url": "https://rangoon.ai/insights/archive-swe-rebench-fresh-software-agent-evaluation-may-2025/#paper-event",
      "heading": "The release framed freshness as an evaluation problem",
      "paragraphs": [
        "The first SWE-rebench paper appeared on arXiv on May 26, 2025. Its authors argued that static public coding benchmarks can become contaminated when later models encounter their issues, patches or close variants during training. Their response was a continuously collected supply of executable tasks mined from real GitHub repositories. The paper reported a public dataset of 21,336 interactive Python tasks from 3,468 repositories, then described a 294-task benchmark subset from 169 repositories. This distinction is important: the large collection and the evaluation subset answer different questions. A large dataset can support training or analysis, while a time-bounded benchmark slice attempts to measure a configured agent on problems selected for a particular evaluation period."
      ],
      "bullets": [],
      "source_ids": [
        1
      ]
    },
    {
      "id": "collection-method",
      "url": "https://rangoon.ai/insights/archive-swe-rebench-fresh-software-agent-evaluation-may-2025/#collection-method",
      "heading": "Task eligibility depended on four automated stages",
      "paragraphs": [
        "SWE-rebench did not present a raw archive of pull requests as its result. The paper describes preliminary collection, automated installation-instruction configuration, execution-based installation verification and quality assessment. It filtered candidate projects for permissive licenses and Python-heavy codebases, inferred project versions from tags where possible, and generated structured installation recipes with a language model. If scripted setup or tests failed, the method could revise a recipe from errors and source instructions. The paper reports that at least one working recipe was produced for 31 percent of repositories. That rate illustrates why environment construction is part of the measurement: a repository can contain a plausible issue and patch yet fail to provide a usable, repeatable task environment under the chosen procedure."
      ],
      "bullets": [],
      "source_ids": [
        1
      ]
    },
    {
      "id": "scaffolding",
      "url": "https://rangoon.ai/insights/archive-swe-rebench-fresh-software-agent-evaluation-may-2025/#scaffolding",
      "heading": "A scoreboard also depends on agent scaffolding",
      "paragraphs": [
        "The publisher-operated methodology page says comparison becomes difficult when evaluation systems use different prompts, multi-agent arrangements, retry loops, best-of-N sampling and validation machinery. That warning fits the paper’s motivation. A resolved-rate figure is not a property of model weights alone; it is an outcome of the task set, cutoff, installation, tools, system prompt, execution budget and acceptance test. The paper says it shared an exact system prompt for its evaluations, which is useful provenance, but a later run must still state whether it follows that protocol. The fresh-task claim narrows one source of memorization risk. It does not erase differences in agent design or demonstrate that an agent will understand a particular organization’s architecture, review policy or operational constraints."
      ],
      "bullets": [],
      "source_ids": [
        1,
        2
      ]
    },
    {
      "id": "quality-boundary",
      "url": "https://rangoon.ai/insights/archive-swe-rebench-fresh-software-agent-evaluation-may-2025/#quality-boundary",
      "heading": "Automation traded manual curation for broader coverage",
      "paragraphs": [
        "The paper explicitly identifies the trade-off in its method. Automated quality assessment and generated setup instructions can scale across many repositories, but may leave some tasks imperfectly described or unsolvable from the issue alone. It also notes that the initial data is Python-only and that its quality classifier was trained using labels from SWE-bench Verified. The current project page likewise says not every automatically collected problem is guaranteed fully solvable or well described, and that its rates can therefore look lower than on manually curated SWE-bench Verified. These are not defects to hide; they define what a result measures. The May event supports a specific claim about a continuous, automated evaluation design. It does not prove a universal contamination-free score, a complete task-quality audit, or permission to use any source repository outside its stated license and environment terms."
      ],
      "bullets": [],
      "source_ids": [
        1,
        2
      ]
    }
  ],
  "project_context": null,
  "related_links": [],
  "permissions_url": "https://rangoon.ai/permissions.txt",
  "content_hash": "sha256:ecb5591a451629141a68fed2a94d0671036f3cb5a9c0ca5f08f8e2e53095a263"
}