Terminal-Bench released version 2.0 with Harbor and renewed task verification
Terminal-Bench released version 2.0 and Harbor on November 7, 2025, rebuilding terminal-agent tasks around stronger task verification.
Historical reporting: Terminal-Bench’s dated news index lists “Terminal-Bench 2.0 and Harbor” on November 7, 2025. The linked release post says version 2.0 responded to discovered version-1 task problems through substantial manual and language-model-assisted verification, and introduced Harbor for container-oriented evaluation and optimization workflows. These are publisher claims about a release and its task process. A benchmark result remains conditional on the selected version, task environment, verifier behavior, agent image, model, time and resource budget; it does not certify an agent for every command-line workflow.
Key takeaways
- Terminal-Bench 2.0 treated benchmark maintenance and task verification as part of evaluation quality.
- Harbor paired the task set with a container-oriented harness for running and optimizing agents.
- Version labels and execution limits are necessary because task repairs can change the meaning of a score.
The Terminal-Bench news index dates version 2.0 and Harbor to November 7, 2025[source 1][source 2]
Terminal-Bench’s public news index lists “Terminal-Bench 2.0 and Harbor” on Friday, November 7, 2025. The linked post says the release paired a harder, better-verified version of the terminal-agent benchmark with Harbor, a package for evaluating and optimizing agents. That dated index and release text establish the event. They do not establish that all command-line tasks are represented, that a current model score maps to the same version, or that the benchmark’s publisher assessment is an independent deployment audit. A reader comparing results needs to start by identifying whether a result belongs to 1.0, 2.0, a later revision or another Terminal-Bench family task set.
The release identified changing external services as one reason task verification matters[source 2]
The release post says the team found problems in some version-1 tasks and names a download task whose success could change with YouTube anti-bot protections. It says version 2.0 used substantial manual and language-model-assisted verification for each task. That is substantive evaluation news because it makes an often-hidden premise explicit: a task’s environment and validator can decay or become unreliable. More checking can improve confidence in the stated dataset, but it does not make a benchmark timeless or flawless. External services, package versions, containers, network policy and test assumptions can still change. A score must therefore travel with the task revision and the environment used to run it.
Harbor positioned the harness as a separate operational layer around agent evaluation[source 2]
The announcement says Harbor was built to address slow container evaluation, large-scale cloud execution and interfaces for rollouts used in supervised fine-tuning, reinforcement learning and prompt optimization. It describes an interface for agents installable in a container. This matters because a benchmark is not only a list of prompts: its harness decides how an agent is installed, what it can access, how a run ends and how outcomes are collected. Harbor’s release does not prove that every hosted environment is comparable, inexpensive or secure. It does make the execution layer visible enough to discuss separately from a model. A reported task resolution rate can reflect harness and resource choices as well as model behavior.
Terminal-Bench 2.0 made task verification and the Harbor execution harness part of the experimental method[source 1][source 2]
Terminal-Bench 2.0 responds to a concrete version-1 problem: its announcement says a task involving YouTube downloads could change as anti-bot protections changed. The release says it added substantial manual and language-model-assisted verification to improve task quality. Harbor then supplies a container-oriented harness with cloud deployment, rollout and optimization interfaces for any agent that can be installed in a container. Together, those changes make the task environment and execution harness visible parts of a terminal-agent result. A score is not only a model response; it depends on the task revision, container image, tool interface and verifier behavior used for the run. Comparing 1.0, 2.0 or different Harbor setups without those details can turn a harness change into a false model-capability claim.
Sources
Continue reading
