An OpenAI Math Test Raises Eyebrows

When OpenAI previewed o3, its latest reasoning model, in December, one of the ways the company demonstrated its strength was by showing off its 25.2% score on FrontierMath, a test of AI models’ math abilities that Epoch AI developed. (You might remember Epoch, the nonprofit research organization, from their work on AI scaling laws.)

That’s not the kind of grade I would have bragged about after taking a math test, but the problems in FrontierMath are extremely hard. Before the o3 announcement, the top scores were below 2%. But over the past weekend, it came to light that OpenAI had funded the development of the test and had access to the problems and their solutions, sparking skepticism about o3’s score, along with frustration from AI observers.

“It came off as being a totally independent benchmark that was going to be equal to all firms,” said Ozzie Gooen, Executive Director at the Quantified Uncertainty Research Institute, which develops Squiggle AI, a probabilistic software. “A lot of the frustration comes out from people thinking it was an independent thing while it was actually a proprietary set.”

Mathematicians who contributed to the project also expressed concern. “I try to evaluate all my research projects from the point of view of ‘do they contribute at least as much to safety as to capabilities’” of AI models, said one mathematician who contributed. They figured that Epoch had an industry partner, and they were not surprised to learn it was OpenAI, but the industry connection made the research seem less valuable, they said. “Is this even positive for humanity? It’s very unclear,” they said.