korrents

korrents · piece

The Evolving Landscape of LLM Evaluation

Sebastian Ruder · 13 May 2024 · ruder.io ↗

2 korrents from this piece

Sebastian Ruder did not write this page.

Every claim below was made in this piece, quoted word for word and numbered in the order the piece makes them, so you can read it there rather than take our word for it. The sentence above each quote is our reading of the claim, not their wording. Each quote was checked against a stored copy of the page at build time; where the two differ, the quote is the fact.

  1. Our reading

    Language models perform better on benchmark problems released before their training data cutoff, indicating data contamination inflates scores.

    Their wordsGPT models perform much better on coding problems released before their pre-training data cut-off.

  2. Our reading

    Reliance on public benchmark results for evaluating AI models will decrease over time.

    Their wordsThe time when benchmarks lasted multiple decades has passed. Going forward, we will rely less on public benchmark results.