On Friday, December 20, 2024, during Day 12 of its “12 days of OpenAI,” OpenAI CEO Sam Altman announced its latest AI reasoning models, o3 and o3-mini. He presented test results that suggest their latest model has surpassed crucial human performance benchmarks. Greg Brockman, OpenAI, announced on X “o3, our latest reasoning model, is a breakthrough”.
o3
O3 architecture introduces “simulated reasoning” – an internal dialogue system where the AI evaluates and plans responses before outputting them. OpenAI refers to this as “private chain of thought.” This approach moves beyond traditional LLM capabilities. Industry observers note O3’s enhanced performance in coding, logic, and human-like reasoning processes. A key feature is O3’s ability to tackle new problems without specific training data, suggesting progress toward AGI capabilities.
O3 Mini is a streamlined version of O3, requiring less computational resources while still outperforming O1. OpenAI plans to release these models initially for safety testing and research, not immediate public access.
o3 testing
The O3 model demonstrated notable results on multiple reasoning benchmarks. The most significant result came from the Arc AGI test, specifically designed to evaluate genuine intelligence rather than mere pattern recognition or memorization. While O3 scored 76% under standard testing conditions (within the $10,000 compute budget), it achieved an extraordinary 87.5% when allowed unlimited computational resources. This score exceeds the 85% threshold that Arc AGI had previously established as indicative of artificial general intelligence.
Its mathematical abilities are evident through a 96.7% score on the 2024 American Mathematics Invitational Exam, missing only one question. On GPQA Diamond, testing graduate-level science knowledge, O3 reached 87.7%. The model significantly outperformed others on the Frontier Math benchmark, solving 25.2% of problems compared to the previous ceiling of 2%.
Human intelligence has now been surpassed
The results indicate a fundamental shift in how we measure AI capability. Previously, benchmarks typically showed a gradual approach toward human-level performance. Now, we’re seeing models potentially exceeding peak human capability in certain domains, necessitating new frameworks for understanding and evaluating AI progress.
ARC Challenge
François Chollet created the ARC Challenge in 2019, which tests AI systems using visual grid puzzles with colors. These puzzles check if AI can spot patterns and use basic reasoning. While humans find these puzzles straightforward, AI systems typically struggle with them. The challenge helps measure how well AI can think and solve problems compared to humans. The ARC-AGI version specifically looks at how well AI handles new situations. So far, AI models have performed poorly – GPT-3 scored zero, GPT-4 nearly zero, and GPT-4 with vision capabilities reached only 5% success.
The cost of intelligence
The computational demands for achieving these results are substantial. Analysis suggests that reaching the 87.5% score required approximately 5.7 billion tokens and 13.8 minutes per task, with estimated costs potentially reaching $347,000 for the complete test suite.
Future work
The rapid progress demonstrated between O1 and O3 suggests we’re entering a new phase of AI development. The focus has shifted from pre-training larger models to optimizing inference computation and thought processes.
Jason Wei, a researcher in OpenAI says: “progress from o1 to o3 was only three months, which shows how fast progress will be in the new paradigm of RL (reinforcement learning) on chain of thought to scale inference compute. Way faster than pretraining paradigm of new model every 1-2 tears”.
Feedback to o3 results
Despite these breakthrough results, experts remain measured in their assessment.
This milestone, whether or not we label it as true AGI, marks a significant moment in artificial intelligence development.
François Chollet, the architect behind the Arc AGI prize, acknowledges these results: “these capabilities are new territory and they demand serious scientific attention”. This assessment carries particular weight given Chollet’s historically cautious stance on AI capabilities. Chollet notes that while O3 represents a significant milestone, some simple Arc AGI tasks still prove challenging for the model.
Two things are needed to adapt to novelty. First, knowledge is needed – a collection of reusable functions or programs to build upon. LLMs have more than enough of this. Second, the ability to combine these functions into a new program when faced with a new task – a program that models the current task. Program synthesis. LLMs have long lacked this property. The O-series models fix this.
O3’s core mechanism appears to be natural language program search and execution in the token space – during testing, the model searches through the space of possible chains of thought (CoTs) that describe the steps needed to solve the task. For O3, the search is presumably guided by some type of evaluation model. While one-shot generation LLMs struggle with novelty, O3 overcomes this by generating and executing its own programs, where the program itself (CoT) becomes an artifact of knowledge recombination.
O3 represents a form of deep learning-guided program search. During testing, the model performs a search in the space of “programs” (in this case natural language programs – the space of CoTs describing the steps to solve the current task), guided by deep learning prior knowledge (base LLM). The reason why solving a single ARC-AGI task might take tens of millions of tokens and cost thousands of dollars is that this search process has to explore a huge number of paths through program space – including backtracking.
While not everyone agrees that O3 is AGI, they acknowledge that its current performance represents a significant achievement and clear confirmation that intuition-guided test-time search in program space is a powerful paradigm for building AI systems that can adapt to arbitrary tasks.
Selected comments from Youtube users
@SmarttStuff:
AGI is a spectrum, a paradigm, not a definitive line. We are in the AGI paradigm, and have been for a while. I began in the 80’s working with ‘Expert Systems’, which were actually simple binary calculators. With narrow knowledge they were pretty smart, smarter than the average domain practitioner, at the time (simply on a memory-recall basis). Many current ai models in use today, are vastly smarter than those primitive expert systems. We adjust to the is new normal almost arrogantly. I use current models to do extremely advanced mathematical and scientific cognitive work right now. The level is at least at PHD level, and often very novel and innovative on top of that. As Ilya indicates a more unpredictable model comes with superior intelligence-just like us. So from my perspective, in scientific areas, I am already seeing those AGI sparks, especially since 30 June 2024. This announcement of o3 confirms the trajectory I am seeing. The thing is the speed, which has caught everyone flat-footed. 2025 will be a very wild ride across the AGI frontier, and into the ASI zone.
@zrandomz-t3n:
I’ll bet you anything this isn’t AGI. This just beat the ARC AGI benchmark (at an enormous cost by the way). Similar to the Turing test, just because it passes doesn’t make it AGI.
@beerkegaard:
Ultimate goal post is when it takes YOUR job
@IllusionDX:
… can it learn on real-time instead of having to be trained every time you want to add new info, so that it can learn to do a task it never has done before without being taught to? Does it have effective long-term memory, and doesn’t have the limit of a context window? No? Then it’s not AGI
@eng3d:
… there is a certification to be a CISCO technical (network stuff) which consists of an exam. A lot of people pass the exam by memorizing the content, so officially they are certified. Are they able to work in the real world? not really. It is the same with AGI. It aims to break the turning test, but the turning test is just a generic test and not the real world.
@byrnemeister2008:
@Pabz2030 Just because you have the knowledge to pass the test doesn’t mean you are able to be a doctor. Today’s AI models have the knowledge/facts but they cannot yet apply that in the same way a human does and they often get things wrong when applying the knowledge in novel situations. They are getting better but not as general as humans. Just look at the ARC results as an example. It’s fine to take lots of guesses in a benchmark but when that is a medical issue a doctor is dealing with it needs to be right first time.
@peace5850:
@IllusionDX Do you have an unlimited context window? Can you learn to do all tasks you’ve never done before without being taught (how) to? Then you’re not AGI, right?
@ChaseFreedomMusician:
So, 03 is impressive—don’t get me wrong. The whole “compute at runtime” thing, where it basically reasons deeply on the fly to come up with an answer, feels like a big deal. It’s almost like it’s running simulations in real time to backtrack and figure stuff out. But here’s where it falls short for me: it doesn’t actually learn from that process. Once it figures something out, that’s it—it doesn’t embed that insight back into itself. If you ask the same question again, it’s got to go through the whole process all over, using just as much compute.
To me, that’s not AGI. True AGI would take that insight, update its internal “beliefs,” and grow from the experience. It wouldn’t just solve the problem once; it’d carry that knowledge forward and get better as it goes. It’s like in Star Trek: the computer is brilliant—it can answer anything you ask, no problem. But it’s not Data. Data learns, evolves, and adapts. He builds on what he experiences, and that’s what makes him feel like a real intelligence, not just a really advanced tool.
03 feels more like the Star Trek computer to me. It’s incredibly powerful, but it’s still in that input-output mode. True AGI would be more like Data—learning in real time, adapting to the world, and growing from every interaction. Until we see something like that, I’m not calling it AGI. 03 is a huge step forward, but it’s not there yet.
REFERENCES
https://arcprize.org/blog/oai-o3-pub-breakthrough
https://www.reddit.com/r/OpenAI/comments/1hiptxb/openai_o3_performance_on_arcagi/
https://x.com/gdb/status/1870176891828875658?ref_src=twsrc%5Etfw%7Ctwcamp%5Etweetembed%7Ctwterm%5E1870176891828875658%7Ctwgr%5E8aa14df5139564f3d71ea0157a4aaba00df3cc1a%7Ctwcon%5Es1_&ref_url=https%3A%2F%2Fcdn.embedly.com%2Fwidgets%2Fmedia.html%3Ftype%3Dtext2Fhtmlkey%3Da19fcc184b9711e1b4764040d3dc5c07schema%3Dtwitterurl%3Dhttps3A%2F%2Fx.com%2Fgdb%2Fstatus%2F18701768918288756583Fref_src3Dtwsrc255Etfw257Ctwcamp255Etweetembed257Ctwterm255E1870176891828875658257Ctwgr255E0e1cf1afe60066e9d25f76596371f7f979adf318257Ctwcon255Es1_26ref_url3Dhttps253A252F252Ftechcrunch.com252F2024252F12252F20252Fopenai-announces-new-o3-model252Fimage%3D
https://arstechnica.com/information-technology/2024/12/openai-announces-o3-and-o3-mini-its-next-simulated-reasoning-models/
https://www.youtube.com/watch?v=T7Kx1jLspfc