Ilya Sutskever – We're moving from the age of scaling to the age of research

Ilya Sutskever · 1h 36m · youtube.com ↗

28 korrents from this recording

Ilya Sutskever did not write this page.

Every claim below is a statement made in this recording, quoted word for word and linked to the second it was said, so you can hear it rather than take our word for it. The wording comes from the transcript published alongside the recording; the sentence above each quote is our reading of the claim, not their wording.

0:04:08 · watch on youtube.com↗

Our reading
Models look far better on evals than they are in the world because researchers, inadvertently, take inspiration from the evals when they build RL environments.

Their wordsAnd one of the one thing you could do, and I think that's something that is done inadvertently, is that people take inspiration from the evals.

Interesting
3 min later
0:06:58 · watch on youtube.com↗

Our reading
Today's models are the student who drilled 10,000 hours for the programming contest, which is exactly why what they learn does not carry anywhere else.

Their wordsThe models are much more like the first student but even more because then we say okay so the model should be good at competitive programming so let's get every single competitive programming problem ever and then let's do some data augmentation so we have even more competitive programming problems

Interesting
3 min later
0:09:36 · watch on youtube.com↗

Our reading
There is no human analogue to pre-training — neither childhood nor evolution is the same thing as it.

Their wordsI don't think there is a human analog to pre-training.

Interesting
9 min later
0:18:35 · watch on youtube.com↗

Our reading
Our built-in emotions were tuned for a world that no longer exists, and hunger is one that now fails to guide us correctly.

Their wordsI think for example our intuitive feeling of hunger is not succeeding in guiding us correctly in this world with an abundance of food.

Interesting
1 min later
0:19:39 · watch on youtube.com↗

Our reading
"Scaling" was powerful because it was one word: naming a research direction is what tells a whole field what to do next.

Their wordsAnd it's just this, this is an example of how language affects thought. Scaling is what just one word, but it's such a powerful word because it informs people what to do.

Interesting
2 min later
0:21:51 · watch on youtube.com↗

Our reading
Another hundredfold of scale would not transform anything; the field is back in the age of research, only now with big computers.

Their wordsLike it would be different for sure but like is the belief that if you just 100x the scale everything would be transformed. I don't think that's true. So it's back to the age of research again just with big computers.

Interesting
3 min later
0:24:40 · watch on youtube.com↗

Our reading
Value functions only make reinforcement learning faster: anything you can do with one you can also do without it, just more slowly.

Their wordsI want to like emphasize that I think the value function is something like it's going to make RL more efficient and I think that makes a difference but I think that anything you can do with a value function you can do without just more slowly.

Interesting
0:25:00 · watch on youtube.com↗

Our reading
The most fundamental thing wrong with today's models is not scale or efficiency but that they generalize dramatically worse than people do.

Their wordsThe thing which I think is the most fundamental is that these models somehow just generalize dramatically worse than people.

Interesting
3 min later
0:27:32 · watch on youtube.com↗

Our reading
A five-year-old's vision is already good enough to drive a car, on a tiny and strikingly undiverse amount of data.

Their wordsAt least for me, when I remember myself being 5 years old, my I was I was very excited about cars back then, and I'm pretty sure my car recognition was more than adequate for self-driving already.

Interesting
1 min later
0:28:09 · watch on youtube.com↗

Our reading
That people are good at maths and coding, which evolution never selected for, shows human learning is not a complicated evolutionary prior but something more fundamental.

Their wordsWhat I meant to say is that language math and coding and especially math and coding suggests that whatever it is that makes people good at learning is probably not so much a complicated prior but something more some fundamental thing.

Interesting
3 min later
0:30:41 · watch on youtube.com↗

Our reading
The human value function is extraordinarily robust — addiction aside — which is why a person can judge their own performance at a skill they have only just started.

Their wordsThey have a general sense which is also by the way extremely robust in people like whatever it is the human value function whatever the human value function is with a few exceptions around addiction it's actually very very robust

Interesting
6 min later
0:36:51 · watch on youtube.com↗

Our reading
The age of scaling sucked the air out of the room and left the field with more companies than ideas, by quite a bit.

Their wordsAnd so because scaling sucked out all the air in the room, everyone started to do the same thing. We got to the point where uh we are in a world where there are more companies than ideas by quite a bit.

Interesting
2 min later
0:39:06 · watch on youtube.com↗

Our reading
Research needs some compute but it is far from obvious that it needs the largest amount of compute in the world.

Their wordsSo there definitely for for research you need like definitely some amount of compute but it's far from obvious that you need the absolutely largest amount of compute ever for research.

Interesting
6 min later
0:45:20 · watch on youtube.com↗

Our reading
Letting the public see a powerful AI does something no essay about AI can do, and that is a real reason not to straight-shot superintelligence.

Their wordsLike basically I think I think that there is a big benefit from AI being in the public and that would be a reason for us to not be quite straight shot.

Interesting
5 min later
0:49:53 · watch on youtube.com↗

Our reading
A human being is not an AGI: we lack a huge amount of knowledge and rely on continual learning instead, so continual learning is what superintelligence should mean.

Their wordsA human being, a human being lacks a huge amount of knowledge. Instead, we rely on continual learning. We rely on continual learning.

Interesting
3 min later
0:53:12 · watch on youtube.com↗

Our reading
Broad deployment of AI that can learn on the job will very likely produce rapid economic growth for some period of time.

Their wordsBut I think the idea of very rapid economic growth for some time, I think it's very possible from broad deployment.

Interesting
3 min later
0:56:10 · watch on youtube.com↗

Our reading
AI should be deployed incrementally and in advance, because a system that does not yet exist is one almost nobody can imagine.

Their wordsone of the one of the ways in which my thinking has been changing is that I now place more importance on AI being deployed incrementally and in advance.

Interesting
1 min later
0:57:34 · watch on youtube.com↗

Our reading
The whole problem of AI and AGI is the power, and nothing else about it is the problem.

Their wordsIndeed, the whole problem, what is the problem of AI and AGI? The whole problem is the power. The whole problem is the power. When the power is really big, what's going to happen?

Interesting
3 min later
1:00:06 · watch on youtube.com↗

Our reading
Once AI starts to feel powerful rather than mistake-prone, every AI company will become far more paranoid about safety than it is today.

Their wordsI do think that at some point the AI will start to feel powerful actually and I think when that happens we will see a big change in the way all AI companies approach safety.

Interesting
1 min later
1:01:29 · watch on youtube.com↗

Our reading
It will be easier to build an AI that cares about all sentient life than one that cares about humans alone, because the AI itself will be sentient.

Their wordsI think in particular it will be there's a case to be made that it will be easier to build an AI that cares about sentient life than an AI that cares about human life alone because the AI itself will be sentient.

Interesting
2 min later
1:03:10 · watch on youtube.com↗

Our reading
Capping the power of the most powerful superintelligence would be materially helpful, whatever the difficulty of doing it.

Their wordsNumber three, I think it would be really materially helpful if the power of the most powerful super intelligence was somehow capped because it would address a lot of these concerns.

Interesting
7 min later
1:10:18 · watch on youtube.com↗

Our reading
The long-run equilibrium with powerful AI may require people to merge with it, so that whatever the AI understands is transmitted wholesale to its human.

Their wordsAnd the solution is if people become part AI with some kind of neural link++ because what will happen as a result is that now the AI understands something and we understand it too like because now the understanding is transmitted wholesale.

Interesting
11 min later
1:21:01 · watch on youtube.com↗

Our reading
The AI companies' strategies will converge, because as AI grows more powerful it becomes clearer to everyone what the strategy should be.

Their wordsI maintain that in the end there will be a convergence of strategies. So I think there will be a convergence of strategies where at some point as AI becomes more powerful it's going to become more or less clearer to everyone what the strategy should be.

Interesting
6 min later
1:26:42 · watch on youtube.com↗

Our reading
Even with human-like learners, competition will split the AI market into specialised niches rather than hand it to one company.

Their wordsI think what's going to happen is that the way competition like competition loves specialization and you see it in the market, you see it in evolution as well. So you're going to have lots of different niches and you're going to have lots of different companies who are occupying different niches

Interesting
2 min later
1:29:00 · watch on youtube.com↗

Our reading
A million copies of one researcher would hit diminishing returns quickly, because what research needs is people who think differently.

Their wordsI think I think there'll definitely be there'll be diminishing returns because you want you want people who think differently rather than the same. I think that if they were literal copies of me, I'm not sure how much more incremental value you'd get.

Interesting
1 min later
1:30:02 · watch on youtube.com↗

Our reading
Models resemble each other because pre-training is the same everywhere; what differentiates labs now is RL and post-training.

Their wordsSo the reason there has been no diversity I believe is because of pre-training. All the pre-trained models are the same pretty much because the pre-train on the same data.

Interesting
1 min later
1:31:07 · watch on youtube.com↗

Our reading
Self-play as it was done in the past is too narrow: it only ever develops a small set of skills like negotiation and strategising.

Their wordsNow the the thing is that selfplay at least the way it was done in the past when you have agents which are somehow compete with each other it's only good for developing a certain set of skills it is too narrow.

Interesting
4 min later
1:34:54 · watch on youtube.com↗

Our reading
A top-down aesthetic belief is what sustains a researcher when the experiments contradict them, because data alone cannot tell a wrong idea from a bug.

Their wordsAnd then the top down belief is the thing that sustains you when the experiments contradict you. Because if you just trust the data all the time, well, sometimes you can be doing a correct thing, but there's a bug.

Interesting