Terminal-Bench launched an 80-task terminal-agent benchmark with containerized verification
Terminal-Bench’s May 19, 2025 launch introduced an 80-task terminal-agent set, dedicated Docker environments and tests that checked the state left by a configured agent run.
Historical reporting: Source 0 is Terminal-Bench’s official news index, used only as the dated event locator for May 19, 2025. Source 1 is the full first-release announcement, used for the launch dataset, task and harness details; it does not display an independent date header. Source 2 is the project repository reviewed on October 24, 2026 for current oracle, release and contribution mechanics. It is current technical context, not date authority.
Key takeaways
- The first Terminal-Bench release used 80 hand-crafted, human-verified terminal tasks with dedicated Docker environments and tests.
- Its harness recorded agent actions and verified resulting container state, so the environment and verifier are part of every score.
- The project advises running oracle solutions repeatedly on a target sandbox; an agent result without that preflight has an untested baseline.
The first release defined a terminal task set[source 1][source 2]
Terminal-Bench’s official news index dates its first-release announcement to May 19, 2025. That full launch announcement says Terminal-Bench-Core-v0 contained 80 tasks for agents operating through a computer terminal. It described the tasks as hand-crafted and human-verified, covering scientific workflows, network configuration, games, data analysis, API calls and cybersecurity work. Each task carried a Docker environment, a human-verified solution and tests for the agent solution. Those records support a bounded claim about the launch dataset and its intended measurement. They do not establish that a terminal agent can safely use arbitrary shells, production credentials or organization-specific automation. The benchmark’s sandbox is a designed testing condition, not a transfer of authority to execute commands elsewhere.
Task files made the verifier part of the experiment[source 2]
The full launch article laid out a Core task as a directory with a Dockerfile or compose file, task specification, solution script or YAML, test runner, tests and dependencies. This structure exposes why a terminal benchmark is more than a text prompt. The agent receives a particular filesystem and tool surface; a verifier later judges the state it leaves behind. A task asking for a kernel build, an encryption command or a spectrum fit tests a combination of instruction interpretation, command use, dependency handling and output checks. It cannot independently rank all forms of software engineering or office work. A changed base image, test fixture, network policy or tool version can alter both difficulty and pass conditions, even when an agent model has not changed.
The harness constrained how agents reached the terminal[source 2][source 3]
Terminal-Bench’s original harness was designed to orchestrate agents, create multi-container Docker environments, log actions and verify container state. The announcement described three integration paths: installing an agent into the task environment, direct Python integration, or an MCP server exposing a tmux session. Those paths offer different practical affordances. Installation can fail in realistic or old environments; direct integration can expose logging and API infrastructure; the MCP route restricts an agent to the benchmark-defined tools. Therefore a leaderboard result should name more than model family. It should retain the agent implementation, integration route, task revision, resource limit and environment because each can influence whether an action is available, observable or accepted by the test.
A passing score required a working benchmark baseline[source 2][source 3]
The current repository instructs users to run oracle solutions five times before evaluating an agent, specifically to check that all tasks work in the target sandbox. It notes that its own development, continuous integration and leaderboard experiments use Modal, and asks users to report flakes. The repository also describes Terminal-Bench as a continuous benchmark with tagged releases. These current operating facts do not rewrite the May 2025 launch, but they clarify a durable measurement limit: an agent cannot be credited or blamed until the task baseline is known to pass under the selected environment. The historical event was an open benchmark and harness release. It was not an independent safety certification, a coverage claim for every terminal workflow, or proof that benchmark data belongs in training corpora; the launch explicitly warned against that use.
Sources
Continue reading
Related reading
News analysisOSWorld-Verified updated computer-use tasks, infrastructure and comparison rules3 min read
News analysisTerminal-Bench released version 2.0 with Harbor and renewed task verification3 min read
News analysisSWE-smith published a pipeline for turning Python repositories into software-agent training tasks3 min read