korrents
Eugene Yan

What Eugene Yan thinks about benchmarks

@eugene-yan · 15 positions · 0 changes of mind

Member of technical staff at Anthropic. He has led ML/AI teams at Amazon, Alibaba and Lazada, and writes about LLMs, recommender systems and engineering at eugeneyan.com.

Everything they publish, on ppll ↗

Eugene Yan did not write this page.

We collected these quotes from things they published elsewhere, and every quote links to where it was said. They have no account here and have not endorsed this site. Quotes are word for word; the short line under each one is our own restatement, not their wording. Their own site. Is this you? Claim it or ask us to remove it. Or tell us what is wrong here.

2 dated positions, 2025, in their own words. Our reading of what Eugene Yan has said — not written or endorsed by them.

2 positions so far — this page is not yet offered to search engines.

  1. Our readingThe right bar for an LLM evaluator is human-level performance, not perfect accuracy.

    LLMs

    Their wordsThe benchmark is human performance, not perfection. We sometimes get requirements for 90%+ accuracy.

    ↗Product Evals in Three Simple Stepseugeneyan.com 1st of 3 in this piece

  2. 5 months earlier

  3. Our readingLong-context evaluation benchmarks are likely contaminated by model training data, making them unreliable as a sole evaluation method

    Their wordsSince these datasets are likely already part of model training data, we shouldn't rely solely on them to evaluate our Q&A system.

    ↗Evaluating Long-Context Question & Answer Systemseugeneyan.com 3rd of 3 in this piece