Cloudflare will block AI scraping by default and launches new ‘Pay Per Crawl’ marketplace

Today, Cloudflare became the first major internet infrastructure company to block AI scraping by default. Every new domain registered with Cloudflare will be asked upfront if they want AI crawlers to scrape their site. The shift from an “opt-out” model to an “opt-in” model means AI companies need to receive explicit permission from a website serviced by Cloudflare before scraping.

The company also launched a private beta of “Pay Per Crawl,” a new marketplace where publishers can request compensation from AI companies each time one of their pages is crawled. Rather than simply allowing or blocking crawlers, “pay per crawl” opens up a third option: charging. Publishers can set their own rates on the marketplace, and AI companies can choose whether or not to accept them.

“Pay Per Craw” builds on a micropayment model already established by startups like TollBit. Through Cloudflare’s customer base, though, it has the potential to operate on an unprecedented scale. The company is already accepting new sign-ups.

At launch today, more than a dozen major digital news publishers had endorsed Cloudflare’s new “permission-based” model, including Condé Nast, Dotdash Meredith, Ziff Davis, The Associated Press, Gannett, The Atlantic, Fortune, and Time. They’re joined by social media companies like Reddit and Pinterest, and major copyright holders including Universal Music Group.

“When AI companies can no longer take anything they want for free, it opens the door to sustainable innovation built on permission and partnership,” said Roger Lynch, CEO of Condé Nast, in a statement. “This is a critical step toward creating a fair value exchange on the Internet that protects creators, supports quality journalism and holds AI companies accountable.”

Cloudflare services traffic for 20% of the web and manages trillions of requests each day. The company has long offered bot management tools to help publishers differentiate between human users and crawlers. In recent years, though, generative AI developers have been criticized for using crawlers to scrape content to train and service their products without permission or compensation.

Last year, this debate boiled over when AI companies like Perplexity were called out for evading the robots.txt protocol of digital news publishers like Wired. The honor system has long been the standard for managing crawlers on the internet. In the fall, Cloudflare introduced a “single-click” option to block AI crawlers and begin to clamp down on this unpermitted scraping. The company says this “opt-out” option has since been used by more than one million customers.

Today’s policy changes will make blocking crawlers easier. In addition to the new “opt-in” model, Cloudflare also announced updates to its crawler identification tools. AI companies can now state their purpose for crawling a site, allowing publishers to know whether a given crawler is being used to train an AI model, used for AI inference, or used to index for search. This could give publishers more information as they decide how best to shield their content from illicit scraping.

“We can now limit access to our content to those AI partners willing to engage in fair arrangements,ˮ said Neil Vogel, CEO of Dotdash Meredith.