Cloudflare Blocks AI Bots from Scraping Web Content Without Permission
On Tuesday, the internet infrastructure company Cloudflare announced that it will block AI bots from scraping data from its sites without opt-in permission. Cloudflare hosts about 20 percent of the Web, and the move is seen as a win for the publishing industry. Previously, website owners using Cloudflare could choose to block AI bots, also known as crawlers, when setting up a domain, but if they did nothing, crawlers were permitted to scrape the sites. Such free access to online content has enabled AI companies to create massive data training sets for large language models. The new policy, however, will shift the default to require site owners to actively allow crawlers. Cloudflare is also experimenting with a plan called “pay per crawl” that lets publishers set fees for AI companies to access their sites.
Related Knowledge Center: Artificial intelligence
On top of a failing business model, newsrooms must navigate the powerful — and not fully reliable — new capabilities of artificial intelligence and the deployment of large language models.
State of the business model | Copyright and intellectual property
Related Knowledge Center: Copyright and intellectual property
With copyright questions already complicated by the emergence of the web in the 1990s, artificial intelligence raised the stakes to a new level.