What Nelson Elhage thinks about benchmarks

@nelson-elhage · 19 positions · On ppll ↗

Software engineer who writes the blog Made of Bugs about performance, debugging and understanding computer systems. Previously worked at Anthropic on interpretability, at Stripe on Sorbet, and at Ksplice.

Nelson Elhage did not write this page.

We collected these quotes from things they published elsewhere, and every quote links to where it was said. They have no account here and have not endorsed this site. Quotes are word for word; the short line under each one is our own restatement, not their wording. Their own site. Is this you? Claim it or ask us to remove it. Or tell us what is wrong here.

Filter benchmarks · Newest first

Topic

All 19software performance 6developer tools 5design 3LLMs 2software quality 2benchmarks 1compilers 1documentation 1ideas vs execution 1

3 more topics

neural networks 1startups 1team size 1

Sort

NewestOldest

Only Nelson Elhage on benchmarks. See everyone on benchmarks

1 dated position, 2025, in their own words. Our reading of what Nelson Elhage has said — not written or endorsed by them.

1 position so far — this page is not yet offered to search engines.

1. 9 Mar 2025

   Our readingImpressive-looking benchmark improvements often result from comparing against a poorly-tuned baseline rather than a genuine advance.

   Their wordsIt's easy to get impressive-looking results if you're comparing against a poorly-tuned baseline, and that observation turns out to explain a surprising fraction of supposed improvements.

   ↗Performance of the Python 3.14 tail-call interpreterblog.nelhage.com 1st of 3 in this piece