Microsoft released STATE-Bench to test agent memory in stateful tasks
Microsoft released STATE-Bench on May 19, 2026, evaluating memory systems through stateful enterprise tasks, repeated runs and user interaction.
Historical reporting: Microsoft’s Open Source Blog announced STATE-Bench on May 19, 2026 as an open, memory-agnostic benchmark. The release describes 450 tasks across customer support, travel and shopping, pre-populated stateful environments, deterministic assertions for state-changing tasks, repeated-run reliability and user-experience grading. The linked public repository is implementation context. Microsoft’s baseline outcomes and simulator-variance estimate are publisher results under its task and agent setup; they do not prove that any memory architecture is reliable in another enterprise, policy or production database.
Key takeaways
- STATE-Bench assessed whether memory changes task outcomes, not merely whether an agent can retrieve an old fact.
- Repeated runs made consistency a distinct measure alongside average completion.
- Stateful task success still depends on domain procedures, simulated users, scorers and tool permissions.
Microsoft made STATE-Bench publicly available on May 19, 2026[source 1]
Microsoft’s Open Source Blog dated the STATE-Bench announcement May 19, 2026 and called the release open-source and memory-agnostic. Microsoft framed the target question narrowly: whether an agent improves with experience on realistic enterprise tasks, rather than whether a memory layer retrieves a fact from a long conversation. The release lists 450 tasks across customer support, travel and shopping. This establishes a public benchmark and stated scope. It does not establish that memory will improve every agent, that a particular retrieval method is superior or that a live enterprise should permit an agent to alter records without its own controls and review.
The task loop combined multi-turn interaction with stateful tools and bounded simulation[source 1][source 2]
Microsoft describes each task as a self-contained scenario with pre-populated records, a customer problem and deterministic state assertions. The orchestrator runs a multi-turn loop: the agent receives conversation history, calls tools and interacts with a user simulator that reveals information as needed. This is more than a text-retrieval setup because actions can affect a booking, order, cart or account state. The release says the simulator follows task-specific rules and estimates about one percent simulator-induced variance in testing. That estimate belongs to Microsoft’s implementation, not to all conversational-agent evaluation. A task can be reproducible in its supplied environment while differing from a real organization’s identity checks, exceptions and escalation paths.
The release separated completion, repeated-run reliability, efficiency and user experience[source 1][source 2]
STATE-Bench reports four dimensions: task completion, pass-to-the-fifth-power reliability across five runs, efficiency and a user-experience score. For state-mutating tasks, Microsoft says a deterministic scorer compares final state with ground truth; other task types can use an LLM judge for expected process. This separation is a meaningful design choice. A one-off completion rate can conceal fragile behavior, while an efficient run can still provide a poor customer experience. The metrics also have different sources of uncertainty: repeated samples, simulator behavior, state assertions, judge instructions and cost accounting. A single composite headline would hide those choices, so a reader should retain the component results and conditions.
STATE-Bench tests memory through repeated stateful procedures rather than a one-off recall question[source 1][source 2]
STATE-Bench’s method centers on whether memory changes outcomes in pre-populated, stateful tasks. Microsoft runs each task five times, reports average completion and pass-to-the-fifth-power reliability, and uses deterministic final-state comparison for state-mutating tasks; procedural and informational tasks can use an LLM judge. This makes a memory comparison more demanding than retrieving a past chat fact. The same memory layer can change tool choices, the number of turns, consistency and the user interaction that precedes an action. Microsoft’s no-memory baseline is therefore a starting configuration for the supplied customer-support, travel and shopping environments, not a claim about every memory architecture. The meaningful comparison holds task setup, agent loop and scoring method visible while varying the memory component.
Sources
Continue reading
