{
  "schema_version": "rangoon.article.v1",
  "id": "https://rangoon.ai/insights/archive-amazon-sop-bench-business-procedure-agent-evaluation-august-2026/",
  "url": "https://rangoon.ai/insights/archive-amazon-sop-bench-business-procedure-agent-evaluation-august-2026/",
  "markdown_url": "https://rangoon.ai/insights/archive-amazon-sop-bench-business-procedure-agent-evaluation-august-2026/index.md",
  "json_url": "https://rangoon.ai/insights/archive-amazon-sop-bench-business-procedure-agent-evaluation-august-2026/index.json",
  "title": "Amazon released SOP-Bench for agent evaluation on business procedures",
  "summary": "Amazon released SOP-Bench on August 21, 2026, combining expert-authored procedures, tools and ground-truth task outcomes across business domains.",
  "language": "en",
  "author": {
    "name": "Rangoon Editorial",
    "url": "https://rangoon.ai/insights/editorial/"
  },
  "publisher": {
    "name": "Rangoon",
    "maintainer": "Hypler",
    "url": "https://rangoon.ai/"
  },
  "format": "News analysis",
  "topic": {
    "slug": "agents",
    "label": "Agent systems",
    "url": "https://rangoon.ai/insights/topics/agents/"
  },
  "tags": [
    {
      "slug": "amazon",
      "label": "Amazon",
      "url": "https://rangoon.ai/insights/tags/amazon/"
    },
    {
      "slug": "ai-evaluation",
      "label": "AI evaluation",
      "url": "https://rangoon.ai/insights/tags/ai-evaluation/"
    },
    {
      "slug": "agent-evaluation",
      "label": "Agent evaluation",
      "url": "https://rangoon.ai/insights/tags/agent-evaluation/"
    },
    {
      "slug": "agent-benchmarks",
      "label": "agent benchmarks",
      "url": "https://rangoon.ai/insights/tags/agent-benchmarks/"
    }
  ],
  "dates": {
    "published": "2026-10-24",
    "updated": "2026-10-24",
    "event": "2026-08-21"
  },
  "event_source_id": 1,
  "retrospective": true,
  "reading_minutes": 3,
  "takeaways": [
    "SOP-Bench connected written procedures to runnable tools and known task outcomes.",
    "Its evaluation targeted coordinated procedural work, where ambiguity and multi-tool state matter.",
    "Baseline comparisons remain tied to procedure selection, simulated tools, agent design and scoring definitions."
  ],
  "source_note": "Historical reporting: Amazon Science published SOP-Bench on August 21, 2026. The launch describes more than 2,000 tasks in 12 business areas, human-authored procedures, executable tools, ground-truth outputs, two baseline agents and public code. Amazon’s comparisons across 11 frontier models are research results on this framework; they do not show that newer models always regress, that an agent can operate a private process, or that a benchmark outcome substitutes for human oversight and production authorization.",
  "sources": [
    {
      "id": 1,
      "title": "Amazon Science: SOP-Bench launch",
      "url": "https://www.amazon.science/blog/sop-bench-a-new-benchmark-for-evaluating-ai-agents-on-real-business-procedures",
      "published": "2026-08-21"
    },
    {
      "id": 2,
      "title": "Amazon Science: SOP-Bench public repository",
      "url": "https://github.com/amazon-science/sop-bench",
      "published": null
    }
  ],
  "image": {
    "url": "https://rangoon.ai/assets/editorial/archive-amazon-sop-bench-business-procedure-agent-evaluation-august-2026-diagram-v1.webp",
    "thumbnail_url": "https://rangoon.ai/assets/editorial/archive-amazon-sop-bench-business-procedure-agent-evaluation-august-2026-diagram-v1-thumb.webp",
    "alt": "Original evaluation explanatory diagram, Written procedure transformed into tool-grounded evaluation. Labels: SOP, Tools, Agent, Actions, Outcome, Ground truth. Source boundary: dated primary release reporting; original editorial analysis.",
    "type": "original-diagram",
    "caption": "Written procedure transformed into tool-grounded evaluation. Original Rangoon explanatory diagram: Source boundary: dated primary release reporting; original editorial analysis. No vendor interface, photograph, measurement result, or endorsement implied.",
    "credit": "Rangoon / Hypler",
    "source_url": "https://rangoon.ai/brand/",
    "rights_url": "https://rangoon.ai/terms/#media-use",
    "rights_note": "Hypler artwork permissions do not grant rights to third-party marks or imply endorsement."
  },
  "sections": [
    {
      "id": "august-release",
      "url": "https://rangoon.ai/insights/archive-amazon-sop-bench-business-procedure-agent-evaluation-august-2026/#august-release",
      "heading": "Amazon Science published SOP-Bench on August 21, 2026",
      "paragraphs": [
        "Amazon Science dated the SOP-Bench article August 21, 2026 and introduced an open benchmark for agents carrying out business procedures. The release says the collection spans 12 business areas and more than 2,000 tasks, with procedures authored by domain experts, functioning tools and ground-truth answers. That combination distinguishes the event from a generic claim about business automation: the stated target is execution of supplied procedures under a runnable evaluator. It does not establish that an agent can interpret every organization’s undocumented practice, act lawfully in a regulated setting or take responsibility for a consequential operational decision. Those require separate evidence and controls."
      ],
      "bullets": [],
      "source_ids": [
        1
      ]
    },
    {
      "id": "runnable-procedure",
      "url": "https://rangoon.ai/insights/archive-amazon-sop-bench-business-procedure-agent-evaluation-august-2026/#runnable-procedure",
      "heading": "Each procedure connected text, tool specifications and test cases with known outcomes",
      "paragraphs": [
        "Amazon describes SOP-Bench as a framework in which a procedure contains SOP text, callable tools, tool specifications and test cases with known answers. An agent uses the tools and the framework records its interactions before grading outcomes against ground truth. This makes an operational detail visible: a procedure is not evaluated only as prose comprehension but through the actions needed to reach a stated result. The launch describes expert authors setting context and reviewing AI-assisted conversion of procedures into mock APIs, tools and datasets. That review supports Amazon’s stated construction process. It does not show that synthetic or mock interfaces reproduce every error, permission boundary, data quality issue or exception in a live business system."
      ],
      "bullets": [],
      "source_ids": [
        1,
        2
      ]
    },
    {
      "id": "metrics-and-baselines",
      "url": "https://rangoon.ai/insights/archive-amazon-sop-bench-business-procedure-agent-evaluation-august-2026/#metrics-and-baselines",
      "heading": "Amazon reported outcome-aware metrics and two baseline agent designs",
      "paragraphs": [
        "The article names execution-complete rate, completed-task success rate and task success rate as outcome-aware measures. It says Amazon ran a function-calling agent and a reasoning-style agent across 11 frontier models, and presents them as baselines rather than a final best system. The reported patterns include cases where a newer model did not outperform an older one and where extra tools could lower success. These are useful reasons to test a concrete workflow, yet they are not universal laws of model progress or tool use. The scores depend on selected procedures, the two agent implementations, task mix, tool specifications, model versions and the benchmark’s definition of a completed task."
      ],
      "bullets": [],
      "source_ids": [
        1,
        2
      ]
    },
    {
      "id": "procedure-adherence",
      "url": "https://rangoon.ai/insights/archive-amazon-sop-bench-business-procedure-agent-evaluation-august-2026/#procedure-adherence",
      "heading": "SOP-Bench scores whether an agent follows a runnable procedure with tools and known outcomes",
      "paragraphs": [
        "Amazon’s SOP-Bench method turns expert-authored procedures into tasks with four linked elements: procedure text, callable tools, tool specifications and test cases with known answers. Agents execute tool interactions, and the framework retains a record of calls and reasoning before grading outcomes against ground truth. That shifts evaluation from fluent procedure summaries to adherence across multi-step work, including ambiguous instructions, tool choice and recovery. The release also compares a function-calling agent and a reasoning-style agent across the same procedures, showing why the model and agent design both belong in the result. Amazon reports that no one pairing won across all procedures, so an aggregate score should be read with the procedure-level breakdown. The benchmark’s method supports targeted evaluation of operational steps, not a single universal agent rank."
      ],
      "bullets": [],
      "source_ids": [
        1,
        2
      ]
    }
  ],
  "project_context": null,
  "related_links": [],
  "permissions_url": "https://rangoon.ai/permissions.txt",
  "content_hash": "sha256:f9143d4fd6663ac1dcac2889e18369d480bd92537bfede553697ed0f29bbf0af"
}