Inside The Atlantic’s AI bot blocking strategy
The Atlantic has built a scorecard for AI crawlers, identifying which bots actually send readers back and which just strip content. Only those with value get through.
This approach led it to block a single AI crawler that tried to recrawl its site 564,000 times in the past seven days.
Some publishers have taken a hard line on AI crawlers, blocking all that they don’t have a licensing deal with. The Atlantic has a licensing deal with OpenAI, which it doesn’t block, but has determined that other AI crawlers must drive back traffic or bring in new subscribers for it to unblock them. The hope is that if the AI engines want access to its content to improve their LLMs’ outputs, then they’ll pay a licensing fee.
Continue reading this article on digiday.com. Sign up for Digiday newsletters to get the latest on media, marketing and the future of TV.
Related Knowledge Center: State of the business model
The traditional journalism business model has long since collapsed, and news organizations have shrunk or disappeared entirely. While a number of promising experiments are under way, the public has not shown a willingness to buy what news organizations are selling.
Related Knowledge Center: Artificial intelligence
On top of a failing business model, newsrooms must navigate the powerful — and not fully reliable — new capabilities of artificial intelligence and the deployment of large language models.
State of the business model | Copyright and intellectual property