What Sebastian Ruder thinks about LLMs
Research scientist at Meta working on multilingual models and evaluation; led the multilingual team at Cohere and was at Google DeepMind before that.
Everything they publish, on ppll ↗
Sebastian Ruder did not write this page.
We collected these quotes from things they published elsewhere, and every quote links to where it was said. They have no account here and have not endorsed this site. Quotes are word for word; the short line under each one is our own restatement, not their wording. Their own site. Is this you? Claim it or ask us to remove it. Or tell us what is wrong here.
9 dated positions, 2023 to 2024, in their own words. Our reading of what Sebastian Ruder has said — not written or endorsed by them.
-
Their wordsGPT models perform much better on coding problems released before their pre-training data cut-off.
↗The Evolving Landscape of LLM Evaluationruder.io 1st of 2 in this piece
- 4 weeks earlier
-
Their wordsI'm excited about what this means for the open-source community and research, with the gap between closed-source and open-weight models closing and SOTA-level conversational models being more easily accessible.
-
Our readingRetrieval-augmented generation is the most practical approach for reducing LLM hallucinations.
Their wordsRetrieval-augmented generation (RAG; Lewis et al., 2020), which conditions on the LLM's generation on retrieved documents is the most practical paradigm IMO.
- 4 months earlier
-
Their wordsrecent LLMs are reaching the limits of text data online and repeating data eventually leads to diminishing returns
↗NLP Research in the Era of LLMsruder.io 2nd of 3 in this piece
- 2 weeks earlier
-
Their wordsLLMs are still limited in non-English settings and that making LLMs more multilingual is an important direction.
-
Their wordsIn light of the increasing number of closed-source LLMs, it is important to continue to promote an open culture of sharing knowledge, data, and software, from which the NLP community has benefited greatly.
-
Their wordsFull fine-tuning of LLMs has become prohibitive and requires parameter-efficient methods instead.
- 4 days earlier
-
Their wordsOverall, while initial plans of LLMs can be useful as a starting point, LLM-based planning currently works best mainly in conjunction with external tools.
-
Their wordsOverall, current LLMs still struggle with composing operations into correct reasoning paths.