Jakub Pachocki
Chief Scientist at OpenAI. Worked on the research that turned scaled reinforcement learning into reasoning models, and writes about where he thinks that path leads.
Jakub Pachocki did not write this page.
We collected these quotes from things they published elsewhere, and every quote links to where it was said. They have no account here and have not endorsed this site. Quotes are word for word; the short line under each one is our own restatement, not their wording. Their own site. Is this you? Claim it or ask us to remove it. Or tell us what is wrong here.
-
Our reading
The current speed of AI progress could be sustained all the way into recursive self-improvement.
Their wordsBased on internal results, I have a strong expectation that this speed of progress could be sustained into recursive self-improvement.
-
Our reading
Nobody is prepared for the consequences of machine intelligence continuing to rise this fast.
Their wordsI am concerned no one is prepared for the consequences of a continued rapid rise in machine intelligence.
-
Their wordsAI is grown more than designed - it is, to first degree, the product of repeating a straightforward optimization step many times on a hard-to-imagine amount of compute.
-
Their wordsThe fundamental challenge of AI alignment is generalization.
-
Our reading
Chain-of-thought monitoring is getting less reliable, not more, as models grow more capable.
Their wordsHowever, unfortunately our evaluations indicate our ability to rely on CoT monitoring is progressively diminishing.
-
Their wordsStill, I expect general AI progress to increasingly be bottlenecked by confidence in monitoring.
-
Their wordsA clear risk discussed throughout this year is to cybersecurity: the models are becoming superhuman in their ability to break in and out of computer systems.
-
Their wordsWe may be used to thinking of AI as tools, but some agents will be pursuing their own objectives.
-
Their wordsThe strongest argument I see for continuing to train much smarter models quickly is the need to build defensive systems against the dangers posed by other AI.
-
Their wordsThe idea of racing forward at all costs seems absurd once one internalizes the seriousness of the stakes.
-
Our reading
How far AI systems are scaled must be constrained by how confident we are in their safety.
Their wordsScaling AI systems has to be constrained by our confidence in safety.
-
Our reading
No lab has solved alignment and monitoring well enough to keep scaling at maximum speed much longer.
Their wordsCurrently I believe that no lab has solved alignment and monitoring to a sufficient degree to continue responsibly scaling at maximum speed for much longer.
-
Their wordsI expect and hope for voluntary slowdowns to become commonplace until shared safety bars are established.
-
Their wordsAnd I believe that international coordination on future AI development needs to become a top priority for governments around the world.