Insight tag
11 related stories.
Amazon released SOP-Bench on August 21, 2026, combining expert-authored procedures, tools and ground-truth task outcomes across business domains.
Read analysisTerminal-Bench released version 2.0 and Harbor on November 7, 2025, rebuilding terminal-agent tasks around stronger task verification.
Read analysisMeta introduced ARE and the Gaia2 benchmark on September 22, 2025, focusing agent evaluation on dynamic, asynchronous environments.
Read analysisXLANG Lab’s July 28, 2025 OSWorld-Verified release reported fixes to 300-plus issues, cloud parallelization and a refreshed basis for comparing computer-use agents.
Read analysisTerminal-Bench’s May 19, 2025 launch introduced an 80-task terminal-agent set, dedicated Docker environments and tests that checked the state left by a configured agent run.
Read analysisThe April 30, 2025 SWE-smith paper described a pipeline that builds executable Python environments, injects test-breaking changes and produces large software-agent training sets.
Read analysisOpenAI released BrowseComp on April 10, 2025, with 1,266 short-answer tasks designed to test persistent web browsing and verification.
Read analysisOpenAI released PaperBench on April 2, 2025, using ICML papers, hierarchical rubrics and an open implementation to evaluate research replication.
Read analysisMETR published its time-horizon study on March 19, 2025, relating agent success to expert human task duration across multi-step tasks.
Read analysisEpoch AI introduced FrontierMath on November 8, 2024, pairing original expert mathematics problems with automated answer verification.
Read analysisMeta published the PARTNR embodied-agent benchmark on October 31, 2024, with 100,000 simulated household tasks designed to expose planning and coordination limits.
Read analysis
