Cloudflare Launches AI Bot Blocking Feature to Protect Website Content

Cloudflare Launches AI Bot Blocking Feature to Protect Website Content
Image credit: Cloudflare

Cloudflare has introduced a new feature for web hosting customers to block AI bots from scraping website content without permission, aiming to protect content creators and maintain internet safety. This move responds to customer concerns about AI bots dishonestly visiting their sites.

Previously, website owners could use the robots.txt file to deter automated web crawlers. However, this method, akin to the ignored Do Not Track browser header, can be bypassed without repercussions. Many AI companies, including OpenAI and Google, have provided guidance on blocking their crawlers using robots.txt, but this is not always respected.

Cloudflare's new feature offers a more robust solution, claiming that AI bots now frequent about 39% of the top one million web properties it serves. The company acknowledges that bot operators often try to disguise their bots as real browsers using spoofed user agents, but Cloudflare's global machine-learning model has been effective at identifying these bots.

The detection method relies on digital fingerprinting, analyzing technical details through network interactions to identify automated tools and frameworks used by these bots. Cloudflare processes an average of 57 million requests per second, giving it extensive data to recognize untrustworthy fingerprints.

Customers, including those on the free tier, can now easily block AI scrapers and crawlers by toggling a setting in the Security -> Bots menu. Cloudflare warns that some AI companies might persist in evading detection, but it promises to continuously update its models and add more bot blocks to help content creators maintain control over how their content is used.

This proactive measure by Cloudflare follows reports of AI bots ignoring robots.txt directives, as evidenced by Amazon's investigation into bots from Perplexity, an AWS client, which allegedly reproduced website content without proper credit or permission. Despite Perplexity's denial of intentional wrongdoing, third-party bots used by the company were found to be scraping pages against webmaster wishes.

Overall, Cloudflare's new feature aims to counter the pervasive issue of unauthorized AI bot scraping, ensuring content creators can thrive and control the use of their content in AI models.