When pre-training data is limited, it is better to train a smaller model for multiple epochs than to train a larger model on unique data once.

Added to korrents 1 Dec 2023 · How quotes work · Something wrong? Tell us

Related korrents

Our reading — they may agree, disagree or merely touch the same thing. Closest first: a shared subject counts for most, then how near the wording is.

• A benchmark result should be reported under a stated budget, or as a curve against test-time compute — never as a single number. NBNoam Brown26 Jun 2026scaling lawsResearch scientist at OpenAI
• A lab should spend most of its compute on research rather than on building the next model, because research is where the tenfold yearly efficiency gains come from. DPDylan Patel13 Mar 2026scaling lawsFounder and chief executive of SemiAnalysis
• After pre-training, post-training and test-time scaling, the fourth scaling law is agentic: multiplying AI by spawning agents, and the whole loop scales on one thing, compute. JHJensen Huang23 Mar 2026scaling lawsCo-founder and CEO of NVIDIA since 1993
• AI scaling laws require exponentially more compute and resources to produce only linear gains in model intelligence. NLNathan Lambert19 Sept 2026scaling lawsMachine-learning researcher on open language models
• As models keep scaling up, modularity will become essential to how they are developed. SRSebastian Ruder23 Feb 2023scaling lawsResearch scientist at Meta working on multilingual models and evaluation

On the map

Open the map on its own page →