scaling laws
Whether more compute keeps buying more capability, and where the curve now bends -- pre-training, post-training, test-time.
FilterEveryone, all time90 from 44 people
- RN
Ramez Naam quoted
Author and clean-energy investor who has been publishing…
Their wordsGiven our best current data, the AI self-improvement loop would need to be roughly 5–10× stronger to sustain itself, let alone run away.
- 8 days earlier
- NL
Nathan Lambert quoted
Machine-learning researcher on open language models
Their wordsall of our scaling laws show that you need exponential compute and resources to make linear improvements in intelligence.
↗Why I still haven’t bought into true RSIinterconnects.ai 1st of 3 in this piece
- 2 weeks earlier
- FC
François Chollet quoted
Creator of the Keras deep-learning library and the ARC-AGI benchmark
Their wordsSeems like test time scaling has gained a 3rd axis: latent space reasoning iterations in looped transformers.
- 1 day earlier
- FL
Fei-Fei Li quoted
Sequoia Professor of Computer Science at Stanford
Their wordsI think three of us have total conviction about the scaling law. That that I think we do. I do think the exact architecture choices and data mixtures is where the the devils are in the details.
↗Why Fei-Fei Li Is Betting on Spatial Intelligenceyoutube.com 7th of 13 in this recording
- 1 day earlier
- AR
Armin Ronacher quoted
Creator of Flask and Jinja
Their wordsSo far we haven't really slowed down with what these models do and I don't think it's going to stop right now.
- 7 days earlier
- FC
François Chollet quoted
Their wordsIn verifiable domains, model capability scaling should remain unbounded. Models will simply keep improving by "absorbing more and more of the computational universe", which is infinite by construction.
- 1 day earlier
- DH
David Heinemeier Hansson quoted
Creator of Ruby on Rails, CTO of 37signals
Their wordsSo we should also have the humility. As amazing as the LLMs are now, it could be that they eventually plateau. We haven't seen any evidence of it yet, and I think this is also why we're seeing this absolute gobsmacking levels of investment, because so far the scaling laws are true, and the more billions are poured in, the more intelligence comes out.
↗DHH: Future of Programming, AI, Agentic Engineering, Vibe Coding & Linux | Lex Fridman Podcast #501youtube.com 35th of 41 in this recording
- 2 weeks earlier
- CF
Chelsea Finn quoted
Assistant professor of computer science at Stanford
roboticsreinforcement learning
Their wordswe see that the across the board the single PIO like pre-trained PIO7 model matches or outperforms the fine-tuned specialists that were developed with reinforcement learning post-training for those downstream tasks.
↗Chelsea Finn: This is the State of the Art in Roboticsyoutube.com 11th of 22 in this recording
- 1 day earlier
- SH
Samuel Hammond quoted
Senior economist at the Foundation for American Innovation
Their wordsIn short, RLVR gives us no reason to expect the normative representations learned in pre-training will acquire motivational force over a given action, especially when post-training repeatedly selects trajectories for terminal task success.
↗The Sorcerer’s Apprenticesecondbest.ca 3rd of 3 in this piece
- RG
Ryan Greenblatt quoted
Chief scientist at Redwood Research
Their wordsBasically, the story would end up being that to get five years of AI progress, you're probably going to need around I would say like maybe eight years of algorithmic progress very roughly. Um, which is a lot a lot of algorithmic progress.
↗Ryan Greenblatt – What happens once AI can automate AI research?youtube.com 4th of 26 in this recording
- RG
Ryan Greenblatt quoted
Our readingScaling up spending on expert human data has not been a major driver of progress in AI research.
Their wordsSo my sense is that scaling up the amount of effort spent on getting expert human data has not been hugely important for AI R&D in general.
↗Ryan Greenblatt – What happens once AI can automate AI research?youtube.com 5th of 26 in this recording
- RG
Ryan Greenblatt quoted
Their wordsLike the thing that I think is most likely to be sort of the bottleneck in terms of like the AI are really good at verifiable domains but not not at doing the actual thing is just like big experiments. You only get a few tries um well a few is maybe a bit understated but like basically like historically R&D has been driven by doing near frontier scale experiments and that has been pretty important and like actually doing the one big training run where you decide exactly what to include in that.
↗Ryan Greenblatt – What happens once AI can automate AI research?youtube.com 8th of 26 in this recording
- RG
Ryan Greenblatt quoted
Their wordsI think one reason why um the the AIs have been scaled up less than you would have otherwise expected and like for example cost of of per token hasn't increased as much as you might have thought is because there's a benefit to doing more of your um work at small scale where you can run more training runs and get more cycles in
↗Ryan Greenblatt – What happens once AI can automate AI research?youtube.com 9th of 26 in this recording
- RG
Ryan Greenblatt quoted
Their wordsAnd so there's some like deep underlying properties of the model that are being sort of transferred between model generations because basically you you train your AI on data from the prior generation and keep going.
↗Ryan Greenblatt – What happens once AI can automate AI research?youtube.com 25th of 26 in this recording
- 4 days earlier
- FC
François Chollet quoted
Our reading · no longer heldThe LLM line of research will reach a capability plateau.
Recalls holdingIn the era of base LLM scaling (2022-2024), I believed the LLM line of research would reach a capability plateau (as later seen with base LLMs). In late 2024, after the o3 test-time compute demo, I changed my views: the new models were showing genuine fluid intelligence, and with this new line of work, the LLM line of research could achieve unbounded capability scaling. "There will be no wall."
SaidIn the era of base LLM scaling (2022-2024), I believed the LLM line of research would reach a capability plateau (as later seen with base LLMs). In late 2024, after the o3 test-time compute demo, I changed my views: the new models were showing genuine fluid intelligence, and with this new line of work, the LLM line of research could achieve unbounded capability scaling. "There will be no wall."
- FC
François Chollet quoted
Their wordsI believe current techniques are 4-6 orders of magnitude away from optimality in terms of data efficiency and test-time compute efficiency. But far future AI will be near-optimal.
- 4 days earlier
- DD
Dmitri Dolgov quoted
Co-CEO of Waymo and a founding member of Google's self-driving car project
Their wordsSo, the lesson here is to bet on a system that's maximally learned and minimally constrained and leverage structure intentionally to boost performance and scaling laws both in training and in evaluation.
↗Waymo Co-CEO Dmitri Dolgov: The Demo Is Only 1% Of The Workyoutube.com 16th of 22 in this recording
- 4 days earlier
- JD
Jeff Dean quoted
Chief Scientist at Google DeepMind and Google Research
Their wordsUm, and that's a very very useful general technique is you know inference time compute to perform search over plausible ways of solving the problem that can get much much higher performance or much more reliability in longunning agent flows.
↗Jeff Dean: The 1% Rule for Building in AIyoutube.com 11th of 22 in this recording
- JD
Jeff Dean quoted
Their wordsUm but I I would also caution that the general models are definitely getting better at a broader and broader range of things. So you have to figure out, you know, is that thing you're working on, is that going to be a durable thing or do you think the models uh at the forefront are going to get better at that in the next six months or 12 months or is it something they're not going to be able to do for a couple years or three years? And you know, you you want to weigh that as you're as you're deciding what to work on.
↗Jeff Dean: The 1% Rule for Building in AIyoutube.com 12th of 22 in this recording
- JD
Jeff Dean quoted
Their wordsAnd if they're completely failing, that's probably a good sign. If they're kind of able to do some of it but not very well, that's maybe not a great sign because that's a probably a a sign that the capability is starting to be present in those models and with more training data or larger scale models or or whatever it's likely to get better. So um you know look for something where the model succeeds 0% or 1% of the time not not 20%.
↗Jeff Dean: The 1% Rule for Building in AIyoutube.com 13th of 22 in this recording
- 2 days earlier
- SA
Sam Altman quoted
Co-founder and CEO of OpenAI
Their wordsI think it will feel like the next six months is like maybe equivalent to the last two years of model progress something like that
↗Sam Altman: "Never a Better Time to Do a Startup"youtube.com 16th of 20 in this recording
- 1 day earlier
- BC
Boris Cherny quoted
Creator of Claude Code at Anthropic
Their wordsLike this is actually like a new form of test time compute. Like when we talk about the scaling laws and kind of we talk about the model getting more intelligent over time, historically, it's been a function of the size of the neural net, the amount of training data, and the number of flops that you put in to the training. And then recently, we also added test time compute. So this is essentially a fancy way a researcher way of saying how many tokens does it generate. And now dynamic workflows are essentially a new way to orchestrate test time compute.
↗Boris Cherny: We Cut 80% of Claude Code's Promptyoutube.com 17th of 20 in this recording
- 1 day earlier
- JH
Jensen Huang quoted
Co-founder and CEO of NVIDIA since 1993
Their wordsI would say the chat GPT moment of robots happened a couple of years ago already.
↗Jensen Huang: The Mindset That Built NVIDIAyoutube.com 15th of 20 in this recording
- 8 days earlier
- SR
Sebastian Raschka quoted
AI research engineer working on large language models
Their wordsa smaller model at a higher reasoning effort can sometimes reach a similar score as a larger model at a lower reasoning effort
↗Controlling Reasoning Effort in LLMsmagazine.sebastianraschka.com 3rd of 3 in this piece
- 3 weeks earlier
- NB
Noam Brown quoted
Research scientist at OpenAI
Their wordsThe problem is we're in a world now where the capability of the model is a function of how much money you put into it. Basically, if you give it a budget of $10,000, it can do a lot more than what it can do with a budget of $10. Give it a budget of $10 million, you can do even more. At what budget should you evaluate these models? The policies that exist today don't really address that question.
↗Really Big Test-Time Compute in AI Changes Benchmarks, Safety and Research with OpenAI's Noam Brownyoutube.com 1st of 18 in this recording
- NB
Noam Brown quoted
Their wordsI think the reason why it doesn't show up as so much better on the benchmarks is because the benchmarks are being presented, the benchmark results are being presented in the wrong way. They're not controlling for the amount of test time compute that is being used on that benchmark question.
↗Really Big Test-Time Compute in AI Changes Benchmarks, Safety and Research with OpenAI's Noam Brownyoutube.com 2nd of 18 in this recording
- NB
Noam Brown quoted
Their wordsBut what we're seeing today with the modern models is that 5.5 and other models can think for if you scaffold them reasonably well, can think for weeks even um before having performance plateau on some of these benchmarks. And so, the point at which they plateau is simply too far out to reasonably test.
↗Really Big Test-Time Compute in AI Changes Benchmarks, Safety and Research with OpenAI's Noam Brownyoutube.com 3rd of 18 in this recording
- NB
Noam Brown quoted
Their wordsmy claim is the proper way to evaluate the models now is you either have some kind of budget for the benchmark whether it's tokens or cost or time or whatever or you plot the performance as a function of the amount of test time compute that's going into the model
↗Really Big Test-Time Compute in AI Changes Benchmarks, Safety and Research with OpenAI's Noam Brownyoutube.com 4th of 18 in this recording
- NB
Noam Brown quoted
Their wordsUm so if you say okay well we're going to instead of just running this model once we're going to run it five times and take the best of the five responses or like ask a judge which one it thinks is best then you can get much higher scores than that model. And so it's really easy to make something that looks a lot better on paper but is actually not better once you control for the amount of test time compute.
↗Really Big Test-Time Compute in AI Changes Benchmarks, Safety and Research with OpenAI's Noam Brownyoutube.com 5th of 18 in this recording
- NB
Noam Brown quoted
Their wordsthe preparedness frameworks and responsible scaling policies, they don't really account for the amount of test time compute.
↗Really Big Test-Time Compute in AI Changes Benchmarks, Safety and Research with OpenAI's Noam Brownyoutube.com 7th of 18 in this recording