OpenAI released BrowseComp for hard-to-find web information tasks
OpenAI released BrowseComp on April 10, 2025, with 1,266 short-answer tasks designed to test persistent web browsing and verification.
Historical reporting: OpenAI released BrowseComp on April 10, 2025 and said the open benchmark contained 1,266 hard-to-find, short-answer browsing problems. The release describes model screening, search-result checks, human verification and explicit limits around open-ended user queries. Its linked simple-evals repository is the public implementation context. OpenAI’s model results and test-time-compute experiments are supplier records on BrowseComp; they do not prove general research quality, citation reliability, safe browsing or performance on a changing live web.
Key takeaways
- BrowseComp separated hard-to-find fact location from broad, open-ended question answering.
- Short answers made checking practical while narrowing what the benchmark represents.
- Model results depend on browsing policy, search state, tool access, samples and the protected task set.
OpenAI released 1,266 browsing tasks on April 10, 2025[source 1]
OpenAI dated BrowseComp to April 10, 2025 and described an open-source benchmark of 1,266 challenging problems for agents locating hard-to-find information online. The release’s premise is deliberate: a useful browsing agent may need to connect clues across many sites, not merely retrieve a familiar fact. BrowseComp restricts answers to short, single-answer targets so they can be checked. That makes the event a release of a defined browsing evaluation, not a release of a general web-research certification. A task can be difficult to locate and easy to verify while still differing sharply from a reader’s ambiguous, multi-part or time-sensitive request.
The construction process filtered for questions that were hard to find but bounded to grade[source 1]
OpenAI says trainers created questions from evidence-backed facts, then used checks involving contemporary models, simple searches and human attempts to assess difficulty. The release calls this an asymmetry of verification: finding an answer can require broad exploration, while checking a proposed short answer can be direct. That design gives BrowseComp a clear grading target, but it also sets a boundary on the conclusion. It does not evaluate whether an agent writes a useful explanation, handles a disputed source, asks a clarifying question, maintains citations or decides that a query is unsafe. The exact task-writing and filtering process is part of what a BrowseComp score represents.
Tool use and inference-time effort were explicit conditions in OpenAI’s reported runs[source 1][source 2]
The launch compares models with and without browsing and describes experiments that vary browsing effort and aggregate repeated attempts. It reports that additional test-time work affected outcomes, while also noting that the Deep Research model was trained on data aimed at BrowseComp-style tasks. Those facts make it misleading to treat the numbers as attributes of a base model alone. The agent policy, model version, browser capabilities, search availability, maximum effort, number of samples and aggregation rule all help produce a run. A result obtained on the 2025 collection may also become less comparable as the live web changes, sources disappear or benchmark examples leak into training material.
BrowseComp measures hard answer retrieval while deliberately separating it from open-ended browsing help[source 1]
BrowseComp was designed around a particular asymmetry: locating a fact may require searching many sources, while verifying a proposed short answer can remain direct. OpenAI therefore used short, in-principle single-answer questions instead of long responses or disputed conclusions, and it says the benchmark does not aim to measure a common user-query distribution. This makes its 1,266 tasks suited to probing persistent clue-driven retrieval, not to scoring every part of a browsing assistant. The release also compares browsing policies, test-time effort and aggregation of repeated attempts, so a result belongs to an agent configuration as well as a model. Reading the benchmark this way preserves its method: it is evidence about controlled retrieval under its task construction, not a generic score for explanation quality or source judgment.
Sources
Continue reading
