I Tested How Well AI Tools Work for Journalism

Journalists now have access to an abundance of AI tools on the market that promise to assist with tasks such as transcription, note-taking, summarization, research, and data analysis. Are these tools trustworthy enough for use in the newsroom?

There is not yet a clear answer to that question. While most news organizations have AI policies, the guidelines are typically abstract and broad, and do not address a journalist’s daily workflow. In the absence of precise standards—which should be developed as a community— journalists have largely been left to figure things out for themselves. Many reporters have defaulted to what Cynthia Tu, a data reporter and AI specialist at the nonprofit newsroom Sahan Journal, calls “vibe checks,” or playing around with tools to get a feel for whether they are useful or not.

Jeremy Merrill, a journalist at the Washington Post, used to spot-check AI tools to see which ones might work best for his data projects. But he realized his spot-check method was inadequate. “Vibes are not enough,” he said. “You’re not taking a good enough look at your real data. Is it 60 percent accurate? Seventy? Ninety-five? You just don’t know.”