AI crawlers and fetchers are blowing up websites, with Meta and OpenAI the worst offenders
Cloud services giant Fastly has released a report claiming AI crawlers are putting a heavy load on the open web, slurping up sites at a rate that accounts for 80 percent of all AI bot traffic, with the remaining 20 percent used by AI fetchers. Bots and fetchers can hit websites hard, demanding data from a single site in thousands of requests per minute.
According to the report [PDF], Facebook owner Meta’s AI division accounts for more than half of those crawlers, while OpenAI accounts for the overwhelming majority of on-demand fetch requests.
”AI bots are reshaping how the internet is accessed and experienced, introducing new complexities for digital platforms,” Fastly senior security researcher Arun Kumar opined in a statement on the report’s release. “Whether scraping for training data or delivering real-time responses, these bots create new challenges for visibility, control, and cost. You can’t secure what you can’t see, and without clear verification standards, AI-driven automation risks are becoming a blind spot for digital teams.”
Related Knowledge Center: Artificial intelligence
On top of a failing business model, newsrooms must navigate the powerful — and not fully reliable — new capabilities of artificial intelligence and the deployment of large language models.
State of the business model | Copyright and intellectual property
Related Knowledge Center: Social media
Social media platforms are the source of news for millions of people — and they have a fraught and ever-shifting relationship with the enterprises that provide that news.
The TikTok ban | The Meta-FTC trial