Cloudflare Blocks AI Training and Agent Crawlers by Default on Ad Pages
Cloudflare’s default setting change will block Training and Agent crawlers on ad-supported pages starting September 15, 2026, unless operators separate search indexing from model training and agent traffic. A mixed crawler follows the most restrictive rule. With the change, Cloudflare also seeks to shift from Pay Per Crawl toward pay-per-use, including Ceramic.ai’s pay-per-query testing that would compensate publishers when their content supports AI answers.