Anthropic accused of 'egregious' data scraping

AI start-up Anthropic is facing accusations of aggressive data scraping from various websites to train its AI models, potentially violating publishers' terms of service. The company, founded by former OpenAI researchers, has been described as "the most aggressive scraper" by Matt Barrie, CEO of Freelancer.com, which reported 3.5 million visits from an Anthropic-linked crawler in just four hours. Other web publishers have echoed similar concerns, stating that Anthropic's bots have ignored requests to cease data collection, leading to significant traffic spikes that disrupt their services and revenue.

Anthropic claims to respect publishers' requests and aims to minimize disruption, asserting that its crawlers comply with standard protocols like 'robots.txt'. However, the practice of data scraping has intensified in recent years due to the AI industry's growth, imposing new costs on website operators. For instance, Kyle Wiens, CEO of iFixit.com, reported one million hits from Anthropic bots in 24 hours, highlighting the strain on their resources.

While scraping publicly available data is generally legal, it raises ethical concerns and can breach terms of service. As competition among AI companies escalates, the backlash against aggressive scraping practices may grow, prompting calls for more respectful engagement with content providers. Anthropic, which aims to develop responsible AI, has not publicly announced partnerships with publishers, unlike some of its competitors.