Beyond pageviews: Small news nonprofits develop their own metrics to measure impact

Minneapolis — What’s the right metric to measure a news organization’s success?

A panel of news leaders, most representing small, local outlets, proposed answers to this question in a breakout session at the Institute for Nonprofit News’ annual INN Days conference last week. It was a conversation as much about nonprofit messaging as about newsroom mission, tailored to smaller outlets that don’t equate audience scale with success.

Panelists explained how they think about, track, and communicate success to their communities and to funders. Their approaches are distinct, but share a common theme: A shift from measuring the reach or success of a unit of news — pageviews, awards, and the like — to defining success by impact on people served. It’s one perspective in a broader, thorny debate about how to evaluate the success/impact/relevance of nonprofit news.

“Ultimately, our goal is not to save journalism,” said Charlottesville Tomorrow editor-in-chief and CEO Angilee Shah. “Our goal is to create something useful and vital for our community.”

“We think there’s been too much emphasis on growth metrics and quantitative metrics,” said Amy Kovac-Ashley, executive director of the Tiny News Collective, “and not enough on what we call impact metrics.” Though this panel focused on the latter, she hoped attendees would walk away seeing these metrics as complementary, “like peanut butter and jelly.”

Charlottesville Tomorrow

Charlottesville Tomorrow’s market in central Virginia tops out around 400,000 people, Shah said. That number is key to putting the outlet’s quantitative data in perspective. Four hundred users of a precinct-specific candidate Q&A may not seem like much, but it’s impressive if the precinct has just 700 registered voters.

Charlottesville Tomorrow prioritizes tracking community input. It directs users to a contact form and does frequent one-on-one community listening. The organization, which receives around 130 notes from community members a day, has developed a Slack workflow where staffers can easily share input for inclusion in its impact tracker. Where applicable, a community editor gives input an impact points score on a five-point scale:

• 1 point: Glad you’re here (a note expressing gratitude for Charlottesville Tomorrow generally, or a specific story);
• 2: Reported impact on an individual;
• 3: The individual took an action;
• 4: The individual took an action and got others involved;
• 5: Something changed in the community (like a law, developer’s plan, or city council budget).

Charlottesville Tomorrow is in the process of building an internal dashboard that will highlight impact scores, not just metrics like pageviews, for its stories.

Understanding that metrics are proxies, and what they are proxies for, is critical, Shah said. If you spend most of your time on the Google Analytics dashboard, she wondered, does that measure how much your community values your work — or how good you are at juicing web traffic? One may influence the other, but “in news, we often mistake the proxy to mean something it doesn’t,” she said.

If a funder asks for data that isn’t aligned with the outlet’s mission, Shah shares it — but tells the funder up front that it’s not how Charlottesville Tomorrow measures success.

“Going big on Google Search is never going to be the thing that gives us sustainability, or the thing that makes us useful to the communities we’re serving,” she said.

She pushed back against the idea that funders need standardized metrics to measure local news in different communities on the same terms. “The desire to standardize does not take into account that every community operates very differently and has different needs,” she said.

“If you don’t decide how you’re going to measure your success,” she said, “somebody else will decide for you.”

Baltimore Beat

The Baltimore Beat, a nonprofit news organization that honors “the tradition of the Black press and the spirit of alt-weekly journalism,” is even leaner than Charlottesville Tomorrow. It has four full-time employees, a new Report For America intern, and an RFA reporter joining this summer, said editor-in-chief and cofounder Lisa Snowden.

Collaborations have always been important to the Beat. Its most successful partnership is with Wide Angle Youth Media, which introduces 14- to 24-year-olds to careers in the arts. Young people interested in design are the Beat’s “de facto designers” for its biweekly print product. Young people also helped with the Beat’s Freddie Gray coverage and collaborated on two youth voter guides last year.

The Beat’s partnerships extend into the city beyond media. A local brewery recently designed a beer called The Beat Goes On (“notes of black cherry and basil”); a dollar from every sale goes to the Beat. Earlier this year, ahead of the 10-year anniversary of Freddie Gray’s death, The Beat partnered with the Baltimore Museum of Art on community conversations and a photography exhibit.

In 2020, partnerships and a funding bump spurred the Beat to create “Beat boxes,” a nod to the Baltimore City Paper’s iconic bright yellow metal boxes. In addition to holding print copies of the Baltimore Beat, the wooden boxes (cheaper than metal) also serve as “community exchange” spots for resources like snacks and Covid and drug tests. Community members have “adopted” Beat boxes to stock, and now the Beat is looking at ways to expand them.

The Beat is intentional about “saying who we are and what we do, and not trying to be all things,” Snowden said.

Invisible Institute

The Chicago-based Invisible Institute (which won two Pulitzers last year) primarily investigates police misconduct. Its roots date back to the 1990s, when founder Jamie Kalven reported on the end of high-rise public housing and tried to support residents — whether that meant helping them file complaints about broken elevators or police complaints.

“We often say in our grant reports that we measure impact by thinking about what effect we’ve had on policy and practice,” said Maira Khwaja, the Invisible Institute’s director of public strategy.

The organization believes creating lasting impact in Chicago requires “the sustained work of accompaniment” — helping community members seek justice, including by connecting them to legal resources offered by other organizations.

The Institute thinks about its work as “a part of a wider network of attorneys and organizers and people most affected,” Khwaja said.

Khwaja described two of the Institute’s most impactful stories. Since a 2016 investigation into drug trade and corrupt officers was published, more than 212 convictions have been overturned, she said, thanks both to sustained relationships with the people in the story and legal support from organizations like The Exoneration Project. A 2020 wrongful conviction investigation introduced new evidence into Robert Johnson’s case; he was released a few months ago.

Meanwhile, an investigation into the 2018 police killing of Harith Augustus spurred conversation and attention and inspired an Oscar-nominated documentary — but didn’t impact policy “in any sustained way,” Khwaja said. The Augustus family lost its lawsuit and received no compensation from the city, and transparency rollbacks mean it would be impossible to replicate the investigation today.

Growth metrics, conversations, and events may be important, she added, but the Institute’s bar for impact is: “Does this lead to changed behavior of the police or oversight bodies? And specifically, are the most affected people feeling any difference? This is more than just policy change on paper.”

Still, Khwaja said “funders don’t love the presentation that I just gave,” and this approach “cannot be the bulk of the grant report.” Foundation boards usually still want to see growth metrics and awards, she said, and news nonprofits must equip foundation program officers to make a case that will resonate to “get that funding again.”

Open Campus

Open Campus is a national news outlet that partners with 16 local news orgs to provide them with higher education beat reporting. That’s important because most people go to college within 50 miles of home, but most local newsrooms lack a dedicated higher ed reporter, said local network managing editor Colleen Murphy.

Earlier this year, the outlet revamped its process for tracking impact. It previously had an impact-tracking Slack channel, but it filled with media mentions and “praise, which is not the same thing as impact,” Murphy said. As part of its impact tracking revamp, Open Campus rebranded that channel as “high fives,” and separated out “formal” impact tracking.

Now, Open Campus uses Airtable to track tangible impact at the individual, community, and institution levels, applying the Impact Architects framework.

Reporters fill out short forms recording impact at least once a month (ideally as soon as something happens). Murphy reviews the entries, often adding more information, and works them into narratives. In addition to an annual impact report, she compiles quarterly impact briefs for the team, funders, partner newsrooms, and the public, and creates location-specific impact reports for partners.

Though Open Campus tracks media mentions and republications, it has deprioritized them. They matter “from a brand awareness perspective,” Murphy said, but “don’t actually reflect engagement or reach into our communities.”

Murphy gave three examples of the kinds of impact Open Campus is proud of. A reporter at WBEZ Chicago told the story of a young woman working to pay off her student loans; people who read and heard the story on the radio paid the remaining $10,000. At Mirror Indy, after a reporter covered how Indiana University’s construction of its Indianapolis campus displaced generations of Black families, former neighbors reconnected with one another on social media. And when a reporter at the Texas Tribune covered a young mother caring for a toddler, working overnight at a laundromat, and attending college full-time, readers asked how they could send money to support her.

“How is your reporting increasing empathy,” Murphy said, “and building social cohesion in your communities?”