What Sebastian Ruder thinks about simplicity

@sebastian-ruder · 23 positions · On ppll ↗

Research scientist at Meta working on multilingual models and evaluation; led the multilingual team at Cohere and was at Google DeepMind before that.

Sebastian Ruder did not write this page.

We collected these quotes from things they published elsewhere, and every quote links to where it was said. They have no account here and have not endorsed this site. Quotes are word for word; the short line under each one is our own restatement, not their wording. Their own site. Is this you? Claim it or ask us to remove it. Or tell us what is wrong here.

Filter simplicity · Newest first

Topic

All 23LLMs 18scaling laws 5open source 3benchmarks 2innovation 2simplicity 2AI alignment 1AI and science 1AI writing 1

6 more topics

management 1metascience 1neural networks 1OpenAI 1startups 1team size 1

Sort

NewestOldest

Only Sebastian Ruder on simplicity. See everyone on simplicity

2 dated positions, 2023, in their own words. Our reading of what Sebastian Ruder has said — not written or endorsed by them.

2 positions so far — this page is not yet offered to search engines.

1. 1 Dec 2023

   Our readingWhen pre-training data is limited, it is better to train a smaller model for multiple epochs than to train a larger model on unique data once.

   scaling laws

   Their wordsIn sum, whenever we don't have infinite amounts of pre-training data, we should train smaller models for more (up to 4) epochs.

   ↗NeurIPS 2023 Primerruder.io 1st of 3 in this piece

2. 2 weeks earlier

3. 15 Nov 2023

   Our readingIn instruction tuning, a smaller set of high-quality training examples outperforms a larger, noisier dataset.

   Their wordstraining on a small set of high-quality data outperforms instruction-tuning on larger, noisier data.

   ↗An Overview of Instruction Tuning Dataruder.io 1st of 2 in this piece