IBM Research launched an open agent leaderboard with Exgentic evaluation tooling
IBM Research launched the Open Agent Leaderboard on May 18, 2026, reporting quality and cost for full agent systems across shared benchmarks.
Historical reporting: IBM Research’s Hugging Face post, published May 18, 2026, announces the Open Agent Leaderboard and Exgentic framework. It describes six selected benchmarks, a common task-context-actions protocol, quality and cost reporting, and public code and results. The project’s stated comparisons concern specified agent-model configurations under adapted benchmark environments. They do not define general intelligence, provide an independent model ranking or prove that a listed system will meet a team’s security, reliability or budget requirements.
Key takeaways
- The release treated the complete agent system, including tools and architecture, as the unit of comparison.
- A common protocol made heterogeneous benchmarks more comparable while adding an adaptation layer to inspect.
- Quality and cost columns support tradeoff analysis but do not replace validation in a target workflow.
IBM Research dated the Open Agent Leaderboard launch to May 18, 2026[source 1]
IBM Research’s Hugging Face post is published May 18, 2026 and announces the Open Agent Leaderboard with the Exgentic framework. The release says the project compares full agent systems rather than treating a model name as the entire evaluated object. That is important because planning, tool selection, memory, recovery behavior and context management can alter both outcome and cost. The dated post establishes a public leaderboard and reproduction framework. It does not establish that the leaderboard covers every important agent capability or that a system placed high in its selected suite is generally deployable. The relevant unit is the specified agent-plus-model configuration, not a brand label by itself.
The release assembled several existing task suites under one evaluation protocol[source 1][source 2]
IBM Research says the initial mix covers coding, web research, personal-app actions, customer service and technical support through established benchmark suites. It describes a unified protocol that gives each benchmark a task, context and allowed actions. This lets one agent adapter work across different environments, but the adaptation is also part of the experiment. IBM explicitly notes that different benchmarks have their own assumptions and interaction patterns, and that results can differ from their individual leaderboards. A cross-suite average is therefore a summary of particular implementations and weights. It is not proof that all tasks are equivalent or that a general-purpose agent will preserve performance when tools, policies or user expectations change.
Rows combined success rates, per-task costs and benchmark-level breakdowns[source 1][source 2]
The post says each row identifies a specific agent paired with a specific model and reports average success, average cost per task and per-benchmark breakdowns. That reporting structure discourages a common error: claiming that a model alone caused an outcome when the surrounding agent architecture changed. It also makes cost a comparison dimension instead of a hidden implementation detail. Cost and success still require definition. Token accounting, tool calls, retries, time limits, hardware and aggregator choices can produce different values for the same nominal configuration. Teams should inspect the configuration and underlying run records before deciding that a quality-cost tradeoff matches their constraints.
The Open Agent Leaderboard treats the agent-model pair and its benchmark adapter as the comparison unit[source 1][source 2]
IBM’s release argues that changing tools, planning, memory or recovery can make the same model produce different outcomes and costs. Its leaderboard therefore reports full agent systems, paired with Exgentic for running and reproducing evaluations, rather than assigning a result to a model name alone. That framing supports useful ablations: hold the model fixed while changing an agent component, or hold the agent fixed while changing the model, then inspect how a selected benchmark adapter and allowed actions affect the row. The release’s quality and cost columns are meaningful only for that stated configuration. A cross-benchmark average can summarize a suite, but it does not reveal which agent component drove a change unless the configuration and per-benchmark breakdown remain available. Openness makes those comparisons inspectable.
Sources
- IBM Research: Open Agent Leaderboard launch (May 18, 2026)
- arXiv: General Agent Evaluation (February 26, 2026)
Continue reading
