LLMs 18
- Language models perform better on benchmark problems released before their training data cutoff, indicating data contamination inflates scores.
- Retrieval-augmented generation is the most practical approach for reducing LLM hallucinations.
Sebastian Ruder
Research scientist at Meta working on multilingual models and evaluation; led the multilingual team at Cohere and was at Google DeepMind before that.
Sebastian Ruder did not write this page.
We collected these quotes from things they published elsewhere, and every quote links to where it was said. They have no account here and have not endorsed this site. Quotes are word for word; the short line under each one is our own restatement, not their wording. Their own site. Is this you? Claim it or ask us to remove it. Or tell us what is wrong here.
What Sebastian Ruder says most about
scaling laws 5
- Language models perform better on benchmark problems released before their training data cutoff, indicating data contamination inflates scores.
- The largest AI models will remain the most capable ones in the near term.
open source 3
- The performance gap between closed-source and open-weight language models is closing.
- Leading AI labs disclosing less about their models' architecture and training data is slowing progress in AI development.
benchmarks 2
- Reliance on public benchmark results for evaluating AI models will decrease over time.
- Language models perform better on benchmark problems released before their training data cutoff, indicating data contamination inflates scores.
innovation 2
- Modular AI development could shift model development away from a few dominant institutions and toward the wider community.
- Leading AI labs disclosing less about their models' architecture and training data is slowing progress in AI development.
simplicity 2
- In instruction tuning, a smaller set of high-quality training examples outperforms a larger, noisier dataset.
- When pre-training data is limited, it is better to train a smaller model for multiple epochs than to train a larger model on unique data once.