ARC-AGI benchmark results and reports from AI research publications

OpenAI's 'o3' Surpasses Human Performance; 'o3-mini' Launching January 2025

Dec 20, 2024, 06:42 PM

OpenAI has announced 'o3' and 'o3-mini', their next-generation reasoning models that significantly surpass previous AI models in benchmarks. The 'o3' model achieved breakthrough performance on the ARC-AGI benchmark, scoring 75.7% in low-compute mode and an impressive 87.5% in high-compute mode, exceeding the human performance threshold of 85%. It also set new records on other benchmarks, including solving 25.2% of Frontier Math problems (surpassing the previous best of 2%), scoring 96.7% on the American Invitational Mathematics Examination (AIME), and achieving 71.7% on SWE-Bench verified. The model achieved a Codeforces rating of 2727, placing it in the top 0.05% of competitive programmers. OpenAI's 'o3' models are designed to 'think' before responding via a 'private chain of thought,' representing a significant leap in AI's ability to adapt to novel tasks and marking a qualitative shift in AI capabilities. The company skipped 'o2' due to potential trademark issues with telecommunications firm O2. The 'o3-mini' model is planned to be released publicly by the end of January 2025, with the full 'o3' model to follow shortly after.