Each Sunday, NPR host Will Shortz, The New York Occasions’ crossword puzzle guru, will get to quiz hundreds of listeners in a long-running phase referred to as the Sunday Puzzle. Whereas written to be solvable with out too a lot foreknowledge, the brainteasers are normally difficult even for expert contestants.
That’s why some consultants suppose they’re a promising solution to check the boundaries of AI’s problem-solving talents.
In a new examine, a group of researchers hailing from Wellesley School, Oberlin School, the College of Texas at Austin, Northeastern College, and startup Cursor created an AI benchmark utilizing riddles from Sunday Puzzle episodes. The group says their check uncovers shocking insights, like that so-called reasoning fashions — OpenAI’s o1, amongst others — typically “give up” and supply solutions they know aren’t right.
“We wanted to develop a benchmark with problems that humans can understand with only general knowledge,” Arjun Guha, a pc science undergraduate at Northeastern and one of many co-authors on the examine, informed TechCrunch.
The AI business is in a little bit of a benchmarking quandary in the mean time. Many of the assessments generally used to judge AI fashions probe for abilities, like competency on PhD-level math and science questions, that aren’t related to the common consumer. In the meantime, many benchmarks — even benchmarks launched comparatively not too long ago — are rapidly approaching the saturation level.
The benefits of a public radio quiz sport just like the Sunday Puzzle is that it doesn’t check for esoteric information, and the challenges are phrased such that fashions can’t draw on “rote memory” to unravel them, defined Guha.
“I think what makes these problems hard is that it’s really difficult to make meaningful progress on a problem until you solve it — that’s when everything clicks together all at once,” Guha stated. “That requires a combination of insight and a process of elimination.”
No benchmark is ideal, in fact. The Sunday Puzzle is U.S.-centric and English-only. And since the quizzes are publicly obtainable, it’s potential that fashions skilled on them and might “cheat” in a way, though Guha says he hasn’t seen proof of this.
“New questions are released every week, and we can expect the latest questions to be truly unseen,” he added. “We intend to keep the benchmark fresh and track how model performance changes over time.”
On the researchers’ benchmark, which consists of round 600 Sunday Puzzle riddles, reasoning fashions comparable to o1 and DeepSeek’s R1 far outperform the remaining. Reasoning fashions totally fact-check themselves earlier than giving out outcomes, which helps them keep away from among the pitfalls that usually journey up AI fashions. The trade-off is that reasoning fashions take slightly longer to reach at options — usually seconds to minutes longer.
Not less than one mannequin, DeepSeek’s R1, provides options it is aware of to be mistaken for among the Sunday Puzzle questions. R1 will state verbatim “I give up,” adopted by an incorrect reply chosen seemingly at random — habits this human can actually relate to.
The fashions make different weird selections, like giving a mistaken reply solely to instantly retract it, try and tease out a greater one, and fail once more. In addition they get caught “thinking” perpetually and provides nonsensical explanations for solutions, or they arrive at an accurate reply instantly however then go on to contemplate various solutions for no apparent purpose.
“On hard problems, R1 literally says that it’s getting ‘frustrated,’” Guha stated. “It was funny to see how a model emulates what a human might say. It remains to be seen how ‘frustration’ in reasoning can affect the quality of model results.”
The present best-performing mannequin on the benchmark is o1 with a rating of 59%, adopted by the not too long ago launched o3-mini set to excessive “reasoning effort” (47%). (R1 scored 35%.) As a subsequent step, the researchers plan to broaden their testing to extra reasoning fashions, which they hope will assist to establish areas the place these fashions is perhaps enhanced.
“You don’t need a PhD to be good at reasoning, so it should be possible to design reasoning benchmarks that don’t require PhD-level knowledge,” Guha stated. “A benchmark with broader access allows a wider set of researchers to comprehend and analyze the results, which may in turn lead to better solutions in the future. Furthermore, as state-of-the-art models are increasingly deployed in settings that affect everyone, we believe everyone should be able to intuit what these models are — and aren’t — capable of.”
