NLP Research in the Era of LLMs

Sebastian Ruder · 19 Dec 2023 · ruder.io ↗

3 korrents from this piece

Sebastian Ruder did not write this page.

Every claim below was made in this piece, quoted word for word and numbered in the order the piece makes them, so you can read it there rather than take our word for it. The sentence above each quote is our reading of the claim, not their wording. Each quote was checked against a stored copy of the page at build time; where the two differ, the quote is the fact.

Our reading
Breakthroughs achieved with massive compute are typically achieved using it inefficiently.

Their wordswhile massive compute often achieves breakthrough results, its usage is often inefficient.
Our reading
Available online text data is becoming a limiting factor for AI training, and repeating it yields diminishing returns.

Their wordsrecent LLMs are reaching the limits of text data online and repeating data eventually leads to diminishing returns
Our reading
The largest AI models will remain the most capable ones in the near term.

Their wordsIn the near term, the largest models using the most compute will continue to be the most capable.