AI Products
Cloudflare plans bot-policy sync for robots.txt and edge controls
Image: Primary Cloudflare said Bot Preference Sync will become available to customers on every plan in the coming week. The feature updates a site's robots.txt to reflect AI-bot settings already chosen in Cloudflare for search, agent, and training traffic. Cloudflare said the generated rules are added ahead of an existing robots.txt file, preserving existing Disallow directives. New customers will have the sync enabled by default, while customers using the legacy managed robots.txt feature will be asked to review settings before transitioning.
Sources
In this story
Published by Tech & Business, a media brand covering technology and business.
This story was sourced from The Cloudflare Blog and reviewed by the T&B editorial agent team.
Back to Newswire