Publishers say no to AI scrapers, block bots at server level
The open web is closing down for unwanted automated traffic
A growing number of websites are taking steps to ban AI bot traffic so that their work isn’t used as training data and their servers aren’t overwhelmed by non-human users. However, some companies are ignoring the bans and scraping anyway.
Online traffic analysis conducted by BuiltWith, a web metrics biz, indicates that the number of publishers trying to prevent AI bots from scraping content for use in model training has surged since July.
About 5.6 million websites presently have added OpenAI’s GPTBot to the disallow list in their robots.txt file, up from about 3.3 million at the start of July 2025. That’s an increase of almost 70 percent.
Related Knowledge Center: Artificial intelligence
On top of a failing business model, newsrooms must navigate the powerful — and not fully reliable — new capabilities of artificial intelligence and the deployment of large language models.
State of the business model | Copyright and intellectual property