ARC-AGI-2 added calibrated task sets and a cost axis to its reasoning challenge
ARC Prize launched ARC-AGI-2 on March 24, 2025, with grid-based reasoning tasks, 120-task evaluation sets, pass@2 scoring, and cost reporting.
Historical reporting: ARC Prize published the ARC-AGI-2 launch announcement on March 24, 2025. It records 1,000 public training tasks; public, semi-private, and private 120-task evaluation sets; pass@2 scoring; controlled human calibration; and cost reporting. The technical paper appeared later in May and is used only for format detail, not as March launch-date proof.
Key takeaways
- ARC-AGI-2 asks a system to infer transformations from a few grid examples and apply them to unseen inputs; it does not measure broad task execution.
- Pass@2 allows two attempts because tasks can have explicit ambiguity, so scores must retain that scoring rule when compared.
- ARC Prize’s score-and-cost framing makes resource use visible, but its cost estimates and benchmark results remain protocol-specific evidence.
The March event replaced the prior contest dataset with a harder series entry[source 1]
ARC Prize announced ARC-AGI-2 on March 24, 2025, alongside ARC Prize 2025. The organization said it was harder for AI while preserving relative accessibility for people tested under its controlled study. The central claim is about abstract reasoning tasks, where a system must learn a new rule from examples. It is not a general audit of factual knowledge, coding, tool use, or autonomous-agent safety.
The launch description identifies a design target: expose tasks that people can solve but contemporary AI systems struggle with. That complements tests built around specialized knowledge, yet “fluid intelligence” should retain the project’s definition and task format. A score only carries the meaning supplied by its grids, attempt policy, calibration, and evaluation split.
Public and private sets made exposure part of the evaluation design[source 1][source 2]
The March announcement lists 1,000 public training tasks and three 120-task evaluation sets: public, semi-private, and private. ARC Prize said each calibrated evaluation task had been solved pass@2 by at least two people, based on controlled testing with more than 400 human participants. The semi-private set supported live contest and leaderboard reporting, while the private set was reserved for final contest results. This split tries to preserve a test surface after public practice has become possible.
The later technical paper explains the inherited task format: a few input-output grid pairs reveal an unstated transformation rule, and the test taker must apply that rule to unseen grids. Grids use discrete colored cells; that compact representation strips away domain expertise and forces a different kind of inference. It also limits the benchmark’s scope. Success on symbolic visual transformations does not establish reliable reasoning in a domain that needs evidence retrieval, language interpretation, or real-world action.
Two attempts and human calibration changed what a score represents[source 1]
ARC-AGI-2 uses pass@2, allowing two guesses for a task. The project says this accounts for explicit ambiguity and helps detect unintended ambiguity or mistakes in the dataset. It also describes the calibrated sets as independent and identically distributed in principle, so non-overfit scores across public, semi-private, and private sets should be comparable. These are protocol commitments, not a reason to compare a pass@2 result directly with a single-attempt accuracy number from another benchmark.
Human solvability is a valuable check, but it is not an assertion that every person solves every task equally fast or cheaply. The launch reported both a panel average and an at-least-two-humans measure. A model result should likewise retain its prompt, search policy, sampling strategy, and number of attempts; otherwise a percentage hides whether a system paid for broad search or found a concise rule.
The new leaderboard paired score with an efficiency measure[source 1][source 2]
ARC Prize said future reporting would include an efficiency metric and began with cost per task, arguing that brute-force search can raise accuracy without demonstrating an efficient solution. Its launch table reported score and estimated cost under the project’s stated conditions, and contest rules limited internet APIs and provided roughly $50 of compute per submission. These choices make resource use visible alongside solved tasks, rather than treating accuracy as the only outcome.
They do not produce a universal price of intelligence. API prices, hardware, prompts, batching, and estimation methods change, while a contest’s restrictions are not every deployment’s restrictions. ARC-AGI-2’s contribution was to make both correctness and resource use observable within one challenge. Claims about general intelligence, commercial value, or a model’s performance on unrelated work still require evidence beyond its March benchmark score.
Sources
- ARC Prize: ARC-AGI-2 launch announcement (March 24, 2025)
- arXiv: ARC-AGI-2 technical paper (May 17, 2025)
Continue reading
