The launch of OpenAI’s o1 series of models heralds a new approach to pursuing AI - the one that is more reliant on the test-time compute, and not completely constrained by the capabilities of the underlying foundational models. However, the quality of the base models still matters, and there has been no indication that the pursuit of the next generation frontier models had come to a standstill. Quite the opposite - aside from all the obvious and nonobvious “leaks” (one former high ranking OpenAI employee prominently mentions his involvement with GPT5 in his X bio), pretty much all the top publicly available LLMs are distillations of the next level frontier models using the previous generation’s architecture, or in many cases even simpler architecture.
Scaling laws are not dead, but they imply an exponential growth in terms of resources - not only to train, but also to deploy and run. Hence the importance of model distillation in the future, especially for models that need to be turned into viable products. Llama 3.1 7B and 70B have been fine tuned on Llama 405B. GPT 4o has been trained on GPT 4.5-5. Claude 3.5 Sonnet perhaps on Claude 4.0 Opus? Gemini 1.5 on Gemini 2?
The next generation of AI models will be available at some point soon, perhaps as soon as next month, but almost certainly by the end of the year/beginning of the next year. Just like o1, almost all of them will incorporate elements beyond LLMs, and indeed beyond deep learning (Q*/Strawberry). With the increased “cognitive” power of the AI systems come exponentially higher challenges - safety, infrastructure (compute and electric power), usability, regulations, etc. The AI trend is not slowing down, but each subsequent leap forward requires at least an order of magnitude more of resources, perhaps much more than that.
I am not an AI expert, just studied some math, I kinda frown, but still have an open mind. I have some questions/doubts too thought. I am not going to divert into what applications and versions of AI apps can do what, again, I am not an expert in that field. However "the quality of the base models still matters" where would one base that on, how does one determine the quality of a basemodel unbiased, or even absolutely? Fundamentally, 'the' base models, a mathematical model if you wish, determining the constraints, limitations of what is possible, and the subjective choice of such model (probably still a statistical/stochastical approach) very much imposes limitations on a "foundational model" whatever that definition is/means. In my opinion, one would be very much constraint simply because of logical issues/hurdles, and severe limitations of such models chosen. One simply can not overcome basic fundamental constraints in math, logic and set theory.
On the topic of scale, I wonder if anyone really understands scale, let alone being able to determine 'laws' about scale. Over centuries, mathematicians have been driven to insanity, even to the point of becoming suicidal, just contemplating scale. So proposing a "law of scale", seems a little far fetched, ignorant even. But hey .....
"The AI trend is not slowing down, but each subsequent leap forward requires at least an order of magnitude ...." well ... that exactly has always been the problem in the last 5,6,7 iterations, in the last several decades, of AI. If we just had more speed, ram, storage, power, more sophisticated languages/models (which is a math thing), we could get there. It seems though that every step forward creates a new order of magnitude of issues, that doesnt seem like something that is helpful and can be overcome.
It is like this thing we call conjectures in math, it seems so easy to understand and within grasp, we just didn't get it quite yet. (The math way of biting of more than we can chew).