METR proposed task-completion time horizons for measuring long software work
METR published its time-horizon study on March 19, 2025, relating agent success to expert human task duration across multi-step tasks.
Historical reporting: METR published its task-completion time-horizon study on March 19, 2025. METR defined the measure through the length of tasks, measured by qualified human completion time, that an agent could complete at a chosen probability, and publicly released analysis code and data-pipeline documentation. METR reported an approximately seven-month doubling trend for its measured 50% horizon. That trend, the task distribution, human-time estimates, curve fit, reliability threshold and later data revisions are all part of METR’s methodology; they do not establish that a model can automate a job, perform all work of a given duration, or preserve the same performance outside the studied task suite.
Key takeaways
- METR framed a capability measure around the duration of coherent tasks completed at a stated success probability.
- The headline trend depends on human baselines, task selection and a fitted success curve, so it is not a direct measure of jobs or general intelligence.
- Public analysis code helps inspect the method, while reproducibility still requires the report version, data and evaluation conditions.
METR’s March 19 study connected agent success to the time an expert takes on a task[source 1]
METR published “Measuring AI Ability to Complete Long Software Tasks” on March 19, 2025. The study proposed describing an agent by the length of tasks that human professionals take to complete and that the agent can complete autonomously with a specified probability. METR reported multi-step software and reasoning tasks, along with completion-time records for people with relevant expertise. This design changes the question from whether a system gets an exam answer right to how success changes as a coherent task gets longer. It does not make every task of the same duration equivalent. A short, poorly specified production incident can be harder than a longer well-bounded exercise, and a model can perform unevenly across tasks even where the human-time estimate is similar.
The time-horizon number came from a fitted success curve, not a stopwatch on one agent run[source 1][source 2]
METR describes fitting a logistic relationship between success and the logarithm of human task duration. Once a success threshold is selected, the corresponding point on that fitted curve becomes the time horizon. Its public analysis repository describes the same path: collect tasks with known human completion times, record agent success or failure, fit the curve, and extract durations at thresholds such as p50 and p80. This gives a method for comparing many runs, but it also makes methodological choices visible. A p50 estimate is not an assurance of near-certain completion, and neither p50 nor p80 says an agent succeeds on every task shorter than that duration. The uncertainty interval and failed-task diagnosis remain necessary when comparing models or forecast claims.
METR reported a seven-month doubling trend while keeping extrapolation separate from observed tasks[source 1]
METR reported that the 50% task-completion horizon for the systems in its analysis had doubled at roughly seven-month intervals over six years. The report also used that historical fit to discuss possible future capability timelines. A fitted historical trend is useful evidence about the measured observations, yet the extrapolation is a model-dependent forecast rather than an observed result. METR’s account notes that task subsets, including shorter software tasks, HCAST, RE-Bench and SWE-Bench Verified, can be noisier or use different time information. Changes in model architecture, agent scaffolding, task contamination, access to tools, and the mix of future tasks can all change whether a prior curve remains informative. The dated study supports the metric and METR’s reported trend, not a calendar guarantee about automation.
A bounded task suite does not measure a whole role, workplace context or accountable decision[source 1][source 2]
METR’s research page explains that its measured tasks are multi-step software and reasoning work, and its code repository records task families, human-minute fields, binary or continuous scores, bootstrap samples and fitted outputs. Those artifacts make the evaluation more inspectable than a bare leaderboard score. They do not provide the prior context, interpersonal coordination, ambiguous goals, ownership, or consequences present in most jobs. A time-horizon result also separates task success from deployment value: an organization still needs to consider supervision, cost, security controls, tool permissions, recovery from errors and whether a task’s success condition is complete. The careful historical reading is that METR introduced a duration-based evaluation lens with public method material, while its external validity remains a question for each target domain.
Sources
Continue reading
