The Arc Prize Basis, a nonprofit co-founded by outstanding AI researcher François Chollet, introduced in a weblog put up on Monday that it has created a brand new, difficult check to measure the overall intelligence of main AI fashions.
Thus far, the brand new check, known as ARC-AGI-2, has stumped most fashions.
“Reasoning” AI fashions like OpenAI’s o1-pro and DeepSeek’s R1 rating between 1% and 1.3% on ARC-AGI-2, in line with the Arc Prize leaderboard. Highly effective non-reasoning fashions together with GPT-4.5, Claude 3.7 Sonnet, and Gemini 2.0 Flash rating round 1%.
The ARC-AGI checks encompass puzzle-like issues the place an AI has to establish visible patterns from a set of different-colored squares, and generate the right “answer” grid. The issues have been designed to pressure an AI to adapt to new issues it hasn’t seen earlier than.
The Arc Prize Basis had over 400 individuals take ARC-AGI-2 to ascertain a human baseline. On common, “panels” of those individuals obtained 60% of the check’s questions proper — a lot better than any of the fashions’ scores.
In a put up on X, Chollet claimed ARC-AGI-2 is a greater measure of an AI mannequin’s precise intelligence than the primary iteration of the check, ARC-AGI-1. The Arc Prize Basis’s checks are geared toward evaluating whether or not an AI system can effectively purchase new abilities outdoors the info it was educated on.
Chollet stated that not like ARC-AGI-1, the brand new check prevents AI fashions from counting on “brute force” — in depth computing energy — to seek out options. Chollet beforehand acknowledged this was a significant flaw of ARC-AGI-1.
To deal with the primary check’s flaws, ARC-AGI-2 introduces a brand new metric: effectivity. It additionally requires fashions to interpret patterns on the fly as an alternative of counting on memorization.
“Intelligence is not solely defined by the ability to solve problems or achieve high scores,” Arc Prize Basis co-founder Greg Kamradt wrote in a weblog put up. “The efficiency with which those capabilities are acquired and deployed is a crucial, defining component. The core question being asked is not just, ‘Can AI acquire [the] skill to solve a task?’ but also, ‘At what efficiency or cost?’”
ARC-AGI-1 was unbeaten for roughly 5 years till December 2024, when OpenAI launched its superior reasoning mannequin, o3, which outperformed all different AI fashions and matched human efficiency on the analysis. Nonetheless, as we famous on the time, o3’s efficiency beneficial properties on ARC-AGI-1 got here with a hefty price ticket.
The model of OpenAI’s o3 mannequin — o3 (low) — that was first to succeed in new heights on ARC-AGI-1, scoring 75.7% on the check, obtained a measly 4% on ARC-AGI-2 utilizing $200 value of computing energy per job.
The arrival of ARC-AGI-2 comes as many within the tech business are calling for brand new, unsaturated benchmarks to measure AI progress. Hugging Face’s co-founder, Thomas Wolf, just lately instructed TechCrunch that the AI business lacks enough checks to measure the important thing traits of so-called synthetic common intelligence, together with creativity.
Alongside the brand new benchmark, the Arc Prize Basis introduced a brand new Arc Prize 2025 contest, difficult builders to succeed in 85% accuracy on the ARC-AGI-2 check whereas solely spending $0.42 per job.
