korrents
Sebastian Ruder

What Sebastian Ruder thinks about benchmarks

@sebastian-ruder · 23 positions · 0 changes of mind

Research scientist at Meta working on multilingual models and evaluation; led the multilingual team at Cohere and was at Google DeepMind before that.

Everything they publish, on ppll ↗

Sebastian Ruder did not write this page.

We collected these quotes from things they published elsewhere, and every quote links to where it was said. They have no account here and have not endorsed this site. Quotes are word for word; the short line under each one is our own restatement, not their wording. Their own site. Is this you? Claim it or ask us to remove it. Or tell us what is wrong here.

2 dated positions, 2024, in their own words. Our reading of what Sebastian Ruder has said — not written or endorsed by them.

2 positions so far — this page is not yet offered to search engines.

  1. Our readingLanguage models perform better on benchmark problems released before their training data cutoff, indicating data contamination inflates scores.

    LLMsscaling laws

    Their wordsGPT models perform much better on coding problems released before their pre-training data cut-off.

    ↗The Evolving Landscape of LLM Evaluationruder.io 1st of 2 in this piece

  3. Our readingReliance on public benchmark results for evaluating AI models will decrease over time.

    Their wordsThe time when benchmarks lasted multiple decades has passed. Going forward, we will rely less on public benchmark results.

    ↗The Evolving Landscape of LLM Evaluationruder.io 2nd of 2 in this piece