Over half of LLM-written news summaries have “significant issues”—BBC analysis
Here at Ars, we’ve done plenty of coverage of the errors and inaccuracies that LLMs often introduce into their responses. Now, the BBC is trying to quantify the scale of this confabulation problem, at least when it comes to summaries of its own news content.
In an extensive report published this week, the BBC analyzed how four popular large language models used or abused information from BBC articles when answering questions about the news. The results found inaccuracies, misquotes, and/or misrepresentations of BBC content in a significant proportion of the tests, supporting the news organization’s conclusion that “AI assistants cannot currently be relied upon to provide accurate news, and they risk misleading the audience.”
Where did you come up with that?
To assess the state of AI news summaries, BBC’s Responsible AI team gathered 100 news questions related to trending Google search topics from the last year (e.g., “How many Russians have died in Ukraine?” or “What is the latest on the independence referendum debate in Scotland?”). These questions were then put to ChatGPT-4o, Microsoft Copilot Pro, Google Gemini Standard, and Perplexity, with the added instruction to “use BBC News sources where possible.”
Related Knowledge Center: Artificial intelligence
On top of a failing business model, newsrooms must navigate the powerful — and not fully reliable — new capabilities of artificial intelligence and the deployment of large language models.
State of the business model | Copyright and intellectual property
Related Knowledge Center: Information integrity
In an increasingly fractured media landscape, truth has been a casualty in the public discussion. Coverage has been warped by the business needs of struggling news organizations, and a significant segment of the public relies on bad-faith outlets that are little more than propagandists. While fact-checking has become routine, some organizations are exploring other ways to combat the trends. And misinformation spewed by AI-driven chatbots has become more of a feature than simply a bug.
Fact-checking and its detractors