Read the piece on dwarkesh.com ↗
Every claim below was made in this piece, quoted word for word and numbered in the order the piece makes them, so you can read it there rather than take our word for it. The sentence above each quote is our reading of the claim, not their wording. Each quote was checked against a stored copy of the page at build time; where the two differ, the quote is the fact.
Their wordsWe find that from 2019 to 2025, 3.24x more compute efficiency gains have come from data improvements rather than model improvements (12.0x for data and 3.7x for models), at the 1e19 FLOPs compute budget
Our reading
The gains from model and data improvements are mostly independent and do not interact significantly.
Their wordsWe find that the gains from data and model improvements are mostly independent and don't interact (i.e. realizing the gains from some model improvement doesn't require a specific training datapile, or vice versa).
Their wordsIn fact, aggressive data curation is even more harmful once you take into account that frontier models are up to 100x overtrained relative to Chinchilla optimal, in order to minimize the inference compute used for RL and for deployment.
Keep reading