Cloudflare announces new controls that affect how AI-related crawlers and agents access publisher websites. Starting September 15, Cloudflare will allow traditional search indexing on eligible sites while blocking AI training and AI agent access by default on pages that use advertising, unless publishers choose otherwise. The policy is aimed at separating “search” crawling from “AI training/agent” crawling, giving content owners more control over how their material is used.
In parallel, Cloudflare expands its monetization approach for publishers. The company promotes a pay-per-use model intended to compensate publishers when their content contributes to AI-generated answers, rather than content being accessed only for crawling. Cloudflare frames the changes as a way for publishers to remain discoverable online without necessarily providing their work freely to AI systems.
Across the reports, the common elements are the September 15 timeline, the default blocking of AI training and agent access on ad-supported pages, and the introduction or expansion of payment-based mechanisms for AI-related use.
