Cloudflare rolls out feature for blocking AI companies' web scrapers

Cloudflare has introduced a new feature to block web scrapers used by artificial intelligence (AI) companies from extracting website content. This feature is part of Cloudflare's content delivery network (CDN) and is available in both free and paid tiers. Many AI companies rely on web content for training their large language models (LLMs), and Cloudflare's tool aims to address the issue of some LLM developers not providing opt-out options for website operators. The feature utilizes AI to detect automated content extraction attempts, even those trying to evade detection by mimicking real browsers. Cloudflare will continuously update the feature to adapt to changes in AI scraping bots and is also launching a tool for website operators to report new bots encountered. The company's system assigns a score to website visits to identify potential bot activity, with requests from a bot collecting content for Perplexity AI consistently receiving low scores. Cloudflare's initiative aims to enhance website security and prevent unauthorized scraping by AI companies.