New study shows why simulated reasoning AI models don’t yet live up to their billing
There’s a curious contradiction at the heart of today’s most capable AI models that purport to “reason”: They can solve routine math problems with impressive accuracy, yet when faced with formulating deeper mathematical proofs found in competition-level challenges, they often fail.
That’s the finding of eye-opening preprint research into simulated reasoning (SR) models, initially listed in March and updated in April, that mostly fell under the news radar. The research serves as an instructive case study on the mathematical limitations of SR models, despite sometimes grandiose marketing claims from AI vendors.
What sets simulated reasoning models apart from traditional large language models (LLMs) is that they have been trained to output a step-by-step “thinking” process (often called “chain-of-thought”) to solve problems. Note that “simulated” in this case doesn’t mean that the models do not reason at all but rather that they do not necessarily reason using the same techniques as humans. That distinction is important because human reasoning itself is difficult to define.
Related Knowledge Center: Artificial intelligence
On top of a failing business model, newsrooms must navigate the powerful — and not fully reliable — new capabilities of artificial intelligence and the deployment of large language models.
State of the business model | Copyright and intellectual property