# Cloudflare’s Pay Per Crawl proposed HTTP-priced access for AI crawlers Cloudflare introduced Pay Per Crawl in private beta on July 1, 2025, using HTTP 402 and authenticated crawler requests for paid access. Canonical: https://rangoon.ai/insights/archive-cloudflare-pay-per-crawl-july-2025/ Author: Rangoon Editorial (https://rangoon.ai/insights/editorial/) Published: 2026-10-23 Updated: 2026-10-23 Event date: 2025-07-01 Tags: [Cloudflare](https://rangoon.ai/insights/tags/cloudflare/), [Pay Per Crawl](https://rangoon.ai/insights/tags/pay-per-crawl/), [AI Crawl Control](https://rangoon.ai/insights/tags/ai-crawl-control/), [HTTP 402](https://rangoon.ai/insights/tags/http-402/) Source note: Historical reporting: Cloudflare introduced Pay Per Crawl as a private beta on July 1, 2025. Current Cloudflare documentation describes its later closed-beta state and operating boundary. This article does not claim access, enrollment, crawler verification, a payout, a price, a legal license, or payment success for any website or AI system. Image: Authenticated crawler access and pricing decision. Original Rangoon explanatory diagram: Source boundary: dated primary release reporting; original editorial analysis. Not measured results, official guidance, vendor interface, or event photograph. ## Key takeaways - Pay Per Crawl used HTTP negotiation to make a crawler’s paid-access request visible at the web edge. - Crawler identity and request signatures are separate from a crawler’s declared user agent string. - A 200 response after payment intent is an access event, not proof of downstream model use or content licensing scope. ## The July announcement made paid crawler access a private-beta experiment [source 1](https://blog.cloudflare.com/introducing-pay-per-crawl/) Cloudflare introduced Pay Per Crawl on July 1, 2025 as a private-beta experiment for charging AI crawlers to access content. The announcement positioned it between an unconditional allow rule and an outright block, using existing HTTP mechanisms rather than a separate crawler marketplace protocol. A crawler could present payment intent in request headers for successful access or receive an HTTP 402 Payment Required response with pricing. Cloudflare also stated that it would act as merchant of record and provide underlying technical infrastructure. The request and response make the price decision observable at the same layer that delivers or denies the requested page. ## Access policy has an order of operations [source 1](https://blog.cloudflare.com/introducing-pay-per-crawl/) [source 2](https://developers.cloudflare.com/ai-crawl-control/features/pay-per-crawl/what-is-pay-per-crawl/) The announcement described three publisher choices for a crawler: allow, charge, or block. It also said the allow-or-charge rules operate after existing WAF and bot-management policies, a precedence rule repeated in current documentation. That means an access design must keep denial rules distinct from payment rules: a blocked crawler does not reach a charge flow. Current documentation calls the feature closed beta, which is later operating context rather than evidence that the July 2025 private beta had the same enrollment terms or interface. A policy review must therefore inspect both the security rule and the pricing rule before explaining why a particular request was served or refused. ## Payment requests depend on a verified crawler identity [source 1](https://blog.cloudflare.com/introducing-pay-per-crawl/) Cloudflare’s launch description said crawler operators generate an Ed25519 key pair, publish a JWK-formatted public key directory, register the directory and user-agent information, then sign requests with HTTP Message Signatures. The named signature headers provide an identity path that is stronger than trusting a request’s user-agent field alone. A verifier still needs key-fetch failure behavior, signature freshness, nonce handling, key revocation, and host-to-crawler binding rules. Without those checks, a pricing decision can be detached from the entity that should be charged. Signature verification also needs to bind the method, target authority, and selected request fields so a valid signature cannot be moved to a different resource. ## A paid response is a bounded accounting signal [source 1](https://blog.cloudflare.com/introducing-pay-per-crawl/) [source 2](https://developers.cloudflare.com/ai-crawl-control/features/pay-per-crawl/what-is-pay-per-crawl/) Cloudflare described recording a billing event when an authenticated crawler presents payment intent, receives a 200-level response, and sees a crawler-charged header. That event can support settlement records, but it does not establish what an AI system later retained, trained on, displayed, or attributed. Content owners still need separate terms, policy, and audit choices for different uses, while crawler operators need their own budget and retry controls. The protocol-level signal is useful precisely because it is narrow: it describes one requested-and-served access event, not every downstream consequence of that copy. Retaining response status, request identity, price decision, and timestamp separately gives both sides a smaller, auditable reconciliation unit. It also exposes failed payment or signature flows for debugging without confusing them with a publisher block. Separate records can prevent a retried request from appearing as a second paid delivery. ## Sources 1. [Cloudflare Blog: Introducing pay per crawl: Enabling content owners to charge AI crawlers for access](https://blog.cloudflare.com/introducing-pay-per-crawl/) 2. [Cloudflare AI Crawl Control docs: What is Pay Per Crawl?](https://developers.cloudflare.com/ai-crawl-control/features/pay-per-crawl/what-is-pay-per-crawl/) ## Related reading - [Cloudflare’s Markdown for Agents made content negotiation an AI-reading interface](https://rangoon.ai/insights/archive-cloudflare-markdown-for-agents-february-2026/) - [Cloudflare introduced its Agents SDK for stateful agent applications on Workers](https://rangoon.ai/insights/archive-cloudflare-agents-sdk-february-2025/) - [NVIDIA’s memory codesign post highlights a practical boundary between capacity and usable agent throughput](https://rangoon.ai/insights/nvidia-memory-full-stack-codesign-2026/) ## More information - [Documentation index](https://rangoon.ai/llms.txt) - [AI access and policies](https://rangoon.ai/ai/) - [Source repository](https://github.com/hypler-dev/rangoon)