Major AI companies, including Google, Microsoft, OpenAI and Anthropic, announce support for an upcoming website crawling opt-out standard. The aim is to give website operators a way to block AI developers from using their content for training and model development.
The proposal builds on existing web practices: many sites already use a robots.txt file to instruct automated crawlers what content may be accessed. Outlets report that the new approach is designed to be recognized consistently by participating AI companies, rather than requiring multiple separate blocking instructions for different bots.
Coverage also emphasizes timing and use cases. Reporting notes the standardized mechanism will not fully take effect immediately and is expected to arrive over the coming months. Several outlets frame the announcement in the context of longstanding criticism and legal disputes between publishers and AI companies over scraping and data use, while describing the standard as a more streamlined control for publishers and other websites.
