# AI evaluation insights Rangoon Insights reporting and guides tagged AI evaluation. Canonical: https://rangoon.ai/insights/tags/ai-evaluation/ ## Topics - [Agent systems](https://rangoon.ai/insights/topics/agents/): Agent runtimes, handoffs, tools, and the boundaries that make them reviewable. - [Governance](https://rangoon.ai/insights/topics/governance/): Policy, identity, authorization, approvals, and evidence for consequential work. - [Infrastructure](https://rangoon.ai/insights/topics/infrastructure/): Model serving, sandboxes, tool services, and deployment boundaries. - [Models](https://rangoon.ai/insights/topics/models/): Inference choices, local serving, compatibility, and operational tradeoffs. ## Stories - [Amazon released SOP-Bench for agent evaluation on business procedures](https://rangoon.ai/insights/archive-amazon-sop-bench-business-procedure-agent-evaluation-august-2026/) Amazon released SOP-Bench on August 21, 2026, combining expert-authored procedures, tools and ground-truth task outcomes across business domains. - [Microsoft released STATE-Bench to test agent memory in stateful tasks](https://rangoon.ai/insights/archive-microsoft-state-bench-agent-memory-evaluation-may-2026/) Microsoft released STATE-Bench on May 19, 2026, evaluating memory systems through stateful enterprise tasks, repeated runs and user interaction. - [IBM Research launched an open agent leaderboard with Exgentic evaluation tooling](https://rangoon.ai/insights/archive-ibm-open-agent-leaderboard-exgentic-may-2026/) IBM Research launched the Open Agent Leaderboard on May 18, 2026, reporting quality and cost for full agent systems across shared benchmarks. - [Terminal-Bench released version 2.0 with Harbor and renewed task verification](https://rangoon.ai/insights/archive-terminal-bench-two-harbor-verified-terminal-evaluation-november-2025/) Terminal-Bench released version 2.0 and Harbor on November 7, 2025, rebuilding terminal-agent tasks around stronger task verification. - [OpenAI introduced GDPval for one-shot professional knowledge-work deliverables](https://rangoon.ai/insights/archive-openai-gdpval-knowledge-work-evaluation-september-2025/) OpenAI introduced GDPval on September 25, 2025, evaluating model deliverables across 44 occupations with expert comparison and a public gold subset. - [Meta introduced ARE and Gaia2 for asynchronous agent-environment evaluation](https://rangoon.ai/insights/archive-meta-are-gaia2-asynchronous-agent-evaluation-september-2025/) Meta introduced ARE and the Gaia2 benchmark on September 22, 2025, focusing agent evaluation on dynamic, asynchronous environments. - [OpenAI released HealthBench for rubric-based evaluation of health conversations](https://rangoon.ai/insights/archive-openai-healthbench-may-2025/) OpenAI released HealthBench on May 12, 2025, a public benchmark of 5,000 health conversations scored against physician-created rubrics for model evaluation. - [Amazon introduced SWE-PolyBench to broaden coding-agent evaluation beyond Python repositories](https://rangoon.ai/insights/archive-amazon-swe-polybench-multilanguage-coding-benchmark-april-2025/) Amazon introduced SWE-PolyBench on April 23, 2025, with 2,110 repository issues across Java, JavaScript, TypeScript and Python. - [OpenAI released BrowseComp for hard-to-find web information tasks](https://rangoon.ai/insights/archive-openai-browsecomp-browsing-agent-benchmark-april-2025/) OpenAI released BrowseComp on April 10, 2025, with 1,266 short-answer tasks designed to test persistent web browsing and verification. - [OpenAI released PaperBench to evaluate agents replicating AI research](https://rangoon.ai/insights/archive-openai-paperbench-ai-research-replication-april-2025/) OpenAI released PaperBench on April 2, 2025, using ICML papers, hierarchical rubrics and an open implementation to evaluate research replication. - [METR proposed task-completion time horizons for measuring long software work](https://rangoon.ai/insights/archive-metr-task-completion-time-horizons-march-2025/) METR published its time-horizon study on March 19, 2025, relating agent success to expert human task duration across multi-step tasks. - [OpenAI introduced SWE-Lancer to test coding agents against freelance software work](https://rangoon.ai/insights/archive-openai-swe-lancer-freelance-software-benchmark-february-2025/) OpenAI introduced SWE-Lancer on February 18, 2025, using freelance software tasks, end-to-end tests and manager-choice evaluations. - [Epoch AI introduced FrontierMath with unpublished expert mathematics problems](https://rangoon.ai/insights/archive-epoch-frontiermath-expert-mathematics-benchmark-november-2024/) Epoch AI introduced FrontierMath on November 8, 2024, pairing original expert mathematics problems with automated answer verification. - [Magentic-One’s release made multi-agent orchestration an experiment with failure modes](https://rangoon.ai/insights/archive-microsoft-magentic-one-2024/) Microsoft Research introduced Magentic-One on November 4, 2024 as a generalist multi-agent system, reporting benchmark comparisons and publishing technical details. - [SimpleQA narrowed factuality evaluation to short answers with inspectable grading](https://rangoon.ai/insights/archive-openai-simpleqa-factuality-benchmark-2024/) OpenAI introduced the SimpleQA factuality benchmark on October 30, 2024, using short fact-seeking questions and source-supported reference answers for tractable grading. ## More information - [Documentation index](https://rangoon.ai/llms.txt) - [AI access and policies](https://rangoon.ai/ai/) - [Source repository](https://github.com/hypler-dev/rangoon)