Cloudflare announces a policy update intended to give publishers more control over how AI companies access their sites. The change takes effect for new Cloudflare-managed sites beginning September 15. Under the new controls, search indexing is allowed while access for AI training and for AI agents is blocked by default on many ad-supported publisher pages, unless the publisher explicitly permits it. The policy requires AI providers to separate web crawlers used for traditional search from crawlers used to train AI models and to operate AI agents. Cloudflare also expands its approach to monetization by introducing a pay-per-use model aimed at compensating publishers when their content contributes to AI-generated answers, rather than being accessed only through crawling. Cloudflare says the intent is to avoid forcing publishers to choose between being discoverable online and allowing their work to be used freely by AI systems. TechCrunch reports the September 15 deadline and the requirement to distinguish crawler purposes, while Slashdot adds details about the default blocking behavior on ad-supported pages and the pay-per-use compensation concept.