Cloudflare’s Pay Per Crawl is a newly launched feature designed to give website owners and content creators control and monetization power over AI crawlers that scrape their content. This initiative responds to the growing problem of AI companies scraping vast amounts of web content without permission or compensation, which has disrupted traditional revenue models for publishers.
Key Features of Pay Per Crawl
New websites using Cloudflare will automatically block AI crawlers from accessing their content unless explicitly allowed by the site owner. Existing customers can enable this blocking with a single click in their dashboard. This shifts the web from an opt-out to an opt-in model for AI crawling. Website owners can set a price for each crawl by an AI bot. AI companies that want to access the content must pay per crawl or be blocked. This creates a marketplace where publishers define rates and AI firms decide whether to pay or not. Publishers can allow or block specific AI crawlers based on their identity and purpose (e.g., training, search, content generation). They can also set different prices for different types of content or pathways, enabling dynamic pricing in the future.
Cloudflare uses the HTTP 402 status code to implement the payment mechanism. When a crawler requests access, it either pays the set price upfront or receives a “payment required” response, turning each crawl into a transactional event.
Cloudflare is working with AI firms and major publishers (like Condé Nast, The Atlantic, Associated Press, Reddit, Pinterest, and others) to create a sustainable ecosystem where AI tools get high-quality, licensed content and publishers are compensated fairly. Both publishers and AI companies have Cloudflare accounts where publishers set crawl prices and AI firms register, view pricing, and decide on access. Cloudflare acts as the intermediary, facilitating payments and distribution to content owners.
Motivation and Impact
AI crawlers currently scrape content extensively without driving traffic back to websites, undermining traditional ad revenue models tied to search referrals. Cloudflare powers about 20% of the web, so this initiative could significantly reshape how AI companies access web data. This model aims to create a fairer, permission-based approach to web crawling, ensuring content creators get paid and AI systems access fresh, high-quality data.
The program is still in beta and expected to evolve, potentially including features like previewing content freshness and machine-optimized formats for AI consumption.
