FACTS Grounding’s January paper defined a two-stage test for document-faithful answers
Google DeepMind’s January 6, 2025 FACTS Grounding paper described a 1,719-example benchmark for long-form answers tied to supplied documents.
Historical reporting: Google researchers posted the FACTS Grounding paper on January 6, 2025. The technical record describes 1,719 long-context examples, an eligibility check before grounding scoring, and aggregation across multiple judges. Google DeepMind’s December 2024 announcement is earlier launch context; its maintained leaderboard was not used as proof of a January model ranking.
Key takeaways
- FACTS Grounding scores faithfulness to a supplied document and usefulness to a request together; it is not a general truthfulness test.
- The two-stage procedure prevents a concise but unhelpful answer from receiving credit only because it avoided unsupported details.
- Multi-judge aggregation reduces one judge’s influence, but the score still depends on the benchmark’s prompts, judges, and pass rules.
A January research record set out the grounded-answer problem[source 1]
On January 6, 2025, the FACTS Grounding paper described a benchmark for a narrower question than whether a model “knows facts.” Each prompt provides a request and context document; the model must produce an answer using that material. The paper calls for responses to be fully grounded in the document, which makes the evaluation about fidelity to an assigned record rather than recall from training data or a live search result.
That focus matters for systems that summarize policies, contracts, research, or support material. A response can sound fluent while adding an unsupported inference, omitting a constraint, or answering a different question. FACTS Grounding makes those failure modes part of the task definition. It does not certify a document itself as correct, current, complete, or suitable for a decision; a model can be faithful to a flawed source.
The 1,719 examples put long documents inside the measurement[source 1][source 2]
The paper and the original Google DeepMind announcement describe 1,719 examples. Each combines a document, a system instruction to rely exclusively on that document, and a user request. Documents can reach 32,000 tokens and cover finance, technology, retail, medicine, and law; requests include summarization, question generation, rewriting, and fact finding. That design makes source selection unavailable to the tested model within an item: the work is to read, select, and express evidence from the context it was given.
Long context creates a real tradeoff. It can contain the answer and its qualifications, but it also raises the chance that a model overlooks a condition, blends similar entities, or draws on background knowledge that conflicts with the record. Benchmark success therefore says more about a defined document-to-answer pipeline than about performance on an organization’s retrieval system, changing document store, or legal and medical review process.
Eligibility and grounding were deliberately separated[source 1][source 2]
FACTS Grounding first evaluates whether an answer sufficiently addresses the user request. Responses that fail that eligibility check are disqualified. A second phase judges whether an eligible response is factually accurate relative to the provided document and has no hallucinated material. The final factuality score averages judge scores across examples, so an answer must be both responsive and attributable rather than optimizing for one property in isolation.
The project used Gemini 1.5 Pro, GPT-4o, and Claude 3.5 Sonnet as automated judges and selected prompts against a held-out test set. That is a stated mitigation for same-family judging bias, not proof that automated judges cannot make mistakes. A result is an output of this evaluator design: changing the judge prompt, the context length, the response format, or the document domain can change what is counted as grounded.
A leaderboard number is a benchmark result, not an evidence guarantee[source 1][source 2]
Google DeepMind announced FACTS Grounding and a Kaggle leaderboard in December 2024; the January paper supplied the technical description for the benchmark. A reported score should retain that separation. It represents performance on the project’s examples under its judge ensemble and aggregation rule, not an independent finding that a model will accurately summarize every company file, web page, or uploaded record.
For a practical evaluation, use the benchmark as one signal and test the actual source corpus and decision path. Check whether citations point to the right passage, whether exclusions survive summarization, and whether a refusal is appropriate when the record lacks an answer. The benchmark’s useful contribution is a repeatable definition of grounded long-form answering, while source quality, access controls, and human accountability remain outside its score.
Sources
- arXiv: The FACTS Grounding Leaderboard (January 6, 2025)
- Google DeepMind: FACTS Grounding announcement (December 17, 2024)
Continue reading
Related reading
News analysisMicrosoft put PyRIT-backed AI red teaming into Foundry preview, making scan evidence a separate release artifact3 min read
News analysisGoogle’s Big Sleep reported a previously unknown SQLite flaw before release, with human disclosure still central3 min read
News analysisAnthropic released Petri for broad alignment audits, while its generated transcripts still needed human interpretation3 min read