{
  "schema_version": "rangoon.article.v1",
  "id": "https://rangoon.ai/insights/archive-humanitys-last-exam-january-2025/",
  "url": "https://rangoon.ai/insights/archive-humanitys-last-exam-january-2025/",
  "markdown_url": "https://rangoon.ai/insights/archive-humanitys-last-exam-january-2025/index.md",
  "json_url": "https://rangoon.ai/insights/archive-humanitys-last-exam-january-2025/index.json",
  "title": "Humanity’s Last Exam released 3,000 expert-crafted questions for a hard academic test",
  "summary": "Scale AI and the Center for AI Safety released Humanity’s Last Exam on January 23, 2025: 3,000 multimodal, closed-ended academic questions.",
  "language": "en",
  "author": {
    "name": "Rangoon Editorial",
    "url": "https://rangoon.ai/insights/editorial/"
  },
  "publisher": {
    "name": "Rangoon",
    "maintainer": "Hypler",
    "url": "https://rangoon.ai/"
  },
  "format": "News analysis",
  "topic": {
    "slug": "models",
    "label": "Models",
    "url": "https://rangoon.ai/insights/topics/models/"
  },
  "tags": [
    {
      "slug": "scale-ai",
      "label": "Scale AI",
      "url": "https://rangoon.ai/insights/tags/scale-ai/"
    },
    {
      "slug": "center-for-ai-safety",
      "label": "Center for AI Safety",
      "url": "https://rangoon.ai/insights/tags/center-for-ai-safety/"
    },
    {
      "slug": "humanity-s-last-exam",
      "label": "Humanity’s Last Exam",
      "url": "https://rangoon.ai/insights/tags/humanity-s-last-exam/"
    },
    {
      "slug": "evaluation",
      "label": "Evaluation",
      "url": "https://rangoon.ai/insights/tags/evaluation/"
    }
  ],
  "dates": {
    "published": "2026-10-23",
    "updated": "2026-10-23",
    "event": "2025-01-23"
  },
  "event_source_id": 1,
  "retrospective": true,
  "reading_minutes": 3,
  "takeaways": [
    "HLE was built for difficult, closed-ended academic questions; it does not measure every kind of useful reasoning, agency, or real-world reliability.",
    "Known, verifiable answers enable automated grading, while the question-writing process aimed to avoid answers quickly found by ordinary web retrieval.",
    "Low launch scores measure the authors’ selected questions and evaluation conditions, not a universal boundary on model capability or human expertise."
  ],
  "source_note": "Historical reporting: Scale Labs dated its Humanity’s Last Exam release record January 23, 2025. The release described 3,000 expert-crafted multimodal questions across dozens of subjects, with multiple-choice and short-answer formats suitable for automated grading. The arXiv version appeared January 24, 2025 and is used as corroborating technical context; later dataset revisions are not projected back onto this release.",
  "sources": [
    {
      "id": 1,
      "title": "Scale Labs: Humanity’s Last Exam release record",
      "url": "https://labs.scale.com/papers/humanitys-last-exam",
      "published": "2025-01-23"
    },
    {
      "id": 2,
      "title": "arXiv: Humanity’s Last Exam",
      "url": "https://arxiv.org/abs/2501.14249",
      "published": "2025-01-24"
    }
  ],
  "image": {
    "url": "https://rangoon.ai/assets/editorial/archive-humanitys-last-exam-january-2025-diagram-v1.webp",
    "thumbnail_url": "https://rangoon.ai/assets/editorial/archive-humanitys-last-exam-january-2025-diagram-v1-thumb.webp",
    "alt": "Original research explanatory diagram, Closed-ended academic evaluation. Labels: Expert author, Question item, Text or image, Model answer, Known solution, Accuracy and calibration. Source boundary: dated primary release reporting; original editorial analysis.",
    "type": "original-diagram",
    "caption": "Closed-ended academic evaluation. Original Rangoon explanatory diagram: Source boundary: dated primary release reporting; original editorial analysis. No vendor interface, photograph, measurement result, or endorsement implied.",
    "credit": "Rangoon / Hypler",
    "source_url": "https://rangoon.ai/brand/",
    "rights_url": "https://rangoon.ai/terms/#media-use",
    "rights_note": "Hypler artwork permissions do not grant rights to third-party marks or imply endorsement."
  },
  "sections": [
    {
      "id": "dated-release",
      "url": "https://rangoon.ai/insights/archive-humanitys-last-exam-january-2025/#dated-release",
      "heading": "The January release targeted a benchmark difficulty gap",
      "paragraphs": [
        "Scale Labs dated Humanity’s Last Exam, or HLE, January 23, 2025. The release introduced a multimodal, closed-ended academic benchmark with 3,000 questions across mathematics, humanities, and natural sciences. Its authors framed it as a response to popular evaluations becoming less informative at the frontier, but the event did not create a final exam for every form of intelligence. It created a specific corpus of expert-authored questions designed for automated scoring.",
        "The name is easy to overread. HLE is not a certification that a high score establishes general intelligence, nor does a low score show a system cannot help on a real task. A benchmark converts a broad capability claim into a repeatable test design. Its value depends on whether its questions, grading, and exposure controls remain relevant to the use case being discussed."
      ],
      "bullets": [],
      "source_ids": [
        1,
        2
      ]
    },
    {
      "id": "question-design",
      "url": "https://rangoon.ai/insights/archive-humanitys-last-exam-january-2025/#question-design",
      "heading": "The test joined expert authorship to closed-ended grading",
      "paragraphs": [
        "HLE’s questions were developed globally by subject-matter experts. The launch materials describe multiple-choice and short-answer formats with known, unambiguous, verifiable solutions. That structure lets an evaluator compare answers consistently at scale, while the subject range prevents one narrow discipline from dominating the result. Multimodal questions also mean a text-only system and a system able to interpret an accompanying image are not facing exactly the same input conditions.",
        "The authors additionally stated that questions could not be quickly answered through internet retrieval. This is a design aim, not a permanent contamination warranty. Public release, changing search indexes, tool access, and future model training can alter exposure over time. A current score therefore needs its version, tool policy, modality, and evaluation date beside it; the January release alone cannot freeze those conditions."
      ],
      "bullets": [],
      "source_ids": [
        1,
        2
      ]
    },
    {
      "id": "score-and-calibration",
      "url": "https://rangoon.ai/insights/archive-humanitys-last-exam-january-2025/#score-and-calibration",
      "heading": "Accuracy and calibration describe different failures",
      "paragraphs": [
        "The release says that state-of-the-art models showed low accuracy and calibration on HLE. Accuracy asks whether the submitted answer matches the known solution. Calibration asks whether expressed confidence tracks the chance of being right. A system can answer few questions correctly but still signal uncertainty well, or obtain answers while being poorly calibrated. Combining those observations is useful for risk assessment, but neither metric reveals why a particular answer failed without inspecting the item and response.",
        "Closed-ended grading is also a boundary. It fits questions with an established answer more readily than open research, design review, negotiation, or a software change whose acceptance criteria evolve during work. HLE can show that a model struggled under a fixed academic protocol; it does not report whether a human could use tools, revise a plan, or collaborate with the model to complete a separate task."
      ],
      "bullets": [],
      "source_ids": [
        1,
        2
      ]
    },
    {
      "id": "interpretation-limit",
      "url": "https://rangoon.ai/insights/archive-humanitys-last-exam-january-2025/#interpretation-limit",
      "heading": "Hard questions need version-aware interpretation",
      "paragraphs": [
        "The January materials point to a public release, but a benchmark’s accessibility creates a tension: researchers need enough access to inspect methodology and reproduce evaluation, while broad exposure can make future testing less clean. HLE’s claim to hard, nontrivial questions is strongest when a reported run identifies the dataset version and keeps the selected questions out of the tested system’s training and prompting path as far as the protocol can manage.",
        "For readers of reported launch results, the honest interpretation is bounded: the creators measured systems against their 3,000-question release under their stated grading setup. Results should be described as creator-reported evaluation evidence, not as an all-purpose ranking of models or people. New items, answer corrections, and changes to multimodal handling deserve their own dates and sources instead of being folded invisibly into the January event."
      ],
      "bullets": [],
      "source_ids": [
        1,
        2
      ]
    }
  ],
  "project_context": null,
  "related_links": [],
  "permissions_url": "https://rangoon.ai/permissions.txt",
  "content_hash": "sha256:a9c249680280e0b28ea02396c25b3e0211145ce0368ef598d89ee4a6c0414ff7"
}