korrents
Sebastian Ruder

What Sebastian Ruder thinks about scaling laws

@sebastian-ruder · 23 positions · On ppll ↗

Research scientist at Meta working on multilingual models and evaluation; led the multilingual team at Cohere and was at Google DeepMind before that.

Sebastian Ruder did not write this page.

We collected these quotes from things they published elsewhere, and every quote links to where it was said. They have no account here and have not endorsed this site. Quotes are word for word; the short line under each one is our own restatement, not their wording. Their own site. Is this you? Claim it or ask us to remove it. Or tell us what is wrong here.

Filter scaling laws · Newest first
Topic
All 23LLMs 18scaling laws 5open source 3benchmarks 2innovation 2simplicity 2AI alignment 1AI and science 1AI writing 1
6 more topics
management 1metascience 1neural networks 1OpenAI 1startups 1team size 1
Sort
NewestOldest

Only Sebastian Ruder on scaling laws. See everyone on scaling laws

5 dated positions, 2023 to 2024, in their own words. Our reading of what Sebastian Ruder has said — not written or endorsed by them.

  1. Our readingLanguage models perform better on benchmark problems released before their training data cutoff, indicating data contamination inflates scores.

    LLMsbenchmarks

    Their wordsGPT models perform much better on coding problems released before their pre-training data cut-off.

    ↗The Evolving Landscape of LLM Evaluationruder.io 1st of 2 in this piece

  2. 5 months earlier

  3. Our readingBreakthroughs achieved with massive compute are typically achieved using it inefficiently.

    Their wordswhile massive compute often achieves breakthrough results, its usage is often inefficient.

    ↗NLP Research in the Era of LLMsruder.io 1st of 3 in this piece

  5. Our readingThe largest AI models will remain the most capable ones in the near term.

    Their wordsIn the near term, the largest models using the most compute will continue to be the most capable.

    ↗NLP Research in the Era of LLMsruder.io 3rd of 3 in this piece

  6. 3 weeks earlier

  7. Our readingWhen pre-training data is limited, it is better to train a smaller model for multiple epochs than to train a larger model on unique data once.

    Their wordsIn sum, whenever we don't have infinite amounts of pre-training data, we should train smaller models for more (up to 4) epochs.

    ↗NeurIPS 2023 Primerruder.io 1st of 3 in this piece

  8. 9 months earlier

  9. Our readingAs models keep scaling up, modularity will become essential to how they are developed.

    Their wordsGiven the trend of pre-training larger and larger models, we believe modularity will be crucial.

    ↗Modular Deep Learningruder.io 1st of 2 in this piece