Machine-learning researcher on open language models; writes the Interconnects newsletter and the RLHF Book, after leading post-training at Ai2.
We collected these quotes from things they published elsewhere, and every quote links to where it was said. They have no account here and have not endorsed this site. Quotes are word for word; the short line under each one is our own restatement, not their wording. Their own site. Is this you? Claim it or ask us to remove it. Or tell us what is wrong here.
Their wordsThe leading open models have all come from Chinese labs since ~2024.
Their wordsChinese AI companies are the clear leaders in open weight models.
Their wordsThe open-closed model gap has reduced in recent years, and is now at roughly 4-6 months.
Their wordsThe capabilities gap from open to closed models available to users has been decreasing over the last 3 years.
Their wordsChinese model makers at the frontier, however, are becoming more restrictive
Their wordsOne thing we see again and again at Interconnects through our open model analysis is that Western open-weight models get astonishingly more adoption than Chinese models with the same performance-to-size footprint (even though the Chinese models usually come out first)!
Their wordsthese open models are probably going to keep coming for the time being, whether or not we want to stop them, and stopping them might make it even worse and harder to prepare.
Their wordsAt the end of the day, this type of safety barely matters when true open-weights are coming.
Mental modelCollective action problems
Their wordsThe capability diffusion is determined by the lowest common denominator.
Their wordsThe leading proponents of this new wave of open-weight fear mongering have effectively been making a falsifiable prediction — that the current open-weight models are going to cause substantial, not seen before AI harms by crippling our cyber infrastructure. Based on all the evidence I have, which I’ve unpacked above, they’re set up to be wrong.
Their wordsOpen-weight models are becoming an essential tool for AI diffusion, and the best path to get ahead of these risks and unbalanced relationships where American companies rely on models built in China is to continue to enable investment in open models in the US.
Their wordsOpen-source AI has a tricky future, as building the best models is extremely capital intensive.
Their wordsThese models still can get substantial adoption, enough to start growing a business, but the costs of building true frontier models are now measured in billions per major model and only huge companies can sustain that.
Their wordsopen models are still incredibly useful, but fill a long-tail ecosystem relative to the closed counterparts that have monopoly ownership stakes in the most valuable areas like knowledge work collaboration, drug discovery, SWE, etc.
Mental modelFeedback loops
Their wordsuntil there are feedback loops of open source AI, it seems like mostly an ideological mission. People like Mark Zuckerberg, which is like America needs this and I agree with him, but in the time where the motivation ideologically is high, we need to capitalize and build this ecosystem around, what benefits do you get from seeing the language model data?
Their wordsAnd for that reason, there's physical constraints to things like AGI, like recursive improvement to kill us all type stuff. For the physical reasons and for how humans have figured things out before, I'm not too worried about AI takeover.
Their wordsI put the probability of complete extinction as being so low it isn't worth discussing, but the probabilities of AI caused disasters - e.g. cyber attacks on critical infrastructure or bio-risks - as being worth debating.
Their wordslossy self-improvement remains my baseline on the trajectory of progress, and the increased discussion of extinction risk seems very misplaced
Mental modelS-curves and diminishing returns
Their wordsall of our scaling laws show that you need exponential compute and resources to make linear improvements in intelligence.
Their wordsRSI is poised to make modern LLMs vastly cheaper. Trends that have shown LLMs get exponentially cheaper at a given intelligence are likely to accelerate.
Their wordsAnd my kind of statement is that I think the cycle from of having an idea and doing the experimentation to test it, I agree will be 10X faster very soon. But I don’t necessarily agree that I would say that AI researchers will be 10X more productive in net, which I would describe as the pace of the field’s complete understanding. And understanding is a different axis from just continuing to scale models. I think that’s one of my core confusions on the AI research side.
Their wordsIt feels very likely in 50 years that the average American's day to day life looks very similar.
Their wordsAnd we'll get into the details of the models and again and again as we try to get deeper into how the models were trained, we will say things like the data processing, data filtering data quality is the number one determinant of the model quality.
Mental modelEmergence
Their wordsAnd these reasoning behaviors emerge naturally. So these things like, "Wait, let me see. Wait, let me check this. Oh, that might be a mistake." And they emerge from only having questions and answers.
Their wordsAnd the history of NLP and language processing instruction, tuning and tasks per language model used to be like one language model did one task, and then in the instruction tuning literature, there's this point where you start adding more and more tasks together where it just starts to generalize to every task. And we don't know where on this curve we are.
Their wordsAnd the important thing to say is that no matter how you want the model to behave, these RLHF and preference-tuning techniques also improve performance. So, on things like math evals and code evals, there is something innate to these, what is called contrastive loss functions.
Their wordsThe scale word gets a lot of attention in this. The interpretation that I use is effectively to avoid adding the human priors to your learning process. And if you read the original essay, this is what it talks about is how researchers will try to come up with clever solutions to their specific problem that might get them small gains in the short term while simply enabling these deep learning systems to work efficiently, and for these bigger problems in the long term might be more likely to scale and continue to drive success.
Their wordsAnd humans are actually very good at reading or judging between two things versus... This goes back to the core of what RLHF and preference tuning is that it's hard to generate a good answer for a lot of problems, but it's easy to see which one is better.
Mental modelIncentives
Their wordsDistillation is standard practice in industry. Whether or not, if you're at a closed lab where you care about terms of service and IP closely, you distill from your own models.
Their wordsI think that they're trying to shift the narrative. They're trying to protect themselves. We saw this years ago when ByteDance was actually banned from some OpenAI APIs for training on outputs. There's other AI startups that most people, if you're in the AI culture, were like they just told us they trained on OpenAI outputs and they never got banned.
Their wordsCode and data is hard, but ideas is easy. Silicon Valley operates on the way that top employees get bought out by other companies for a pay raise, and a large reason why these companies do this is to bring ideas with them.
Their wordsSomething intertwined with this story, which I stumbled upon when thinking about agents, is how your pace of understanding won't increase by using agents.
Their wordsSo, in 2-5 years I still expect the best textbooks to be heavily crafted by the human hand.
Mental modelBottlenecks
Their wordsModels being stagnant in long-form, non-fiction writing should be alarming to those reliant on models autonomously solving grand, open science problems in the near future.