What Aman Khan thinks about benchmarks

@aman-khan · 6 positions · On ppll ↗

Product leader; director of product at Arize AI in 2024, earlier an AI product manager on Spotify's ML platform, with product roles at Cruise, Zipline and Apple.

Aman Khan did not write this page.

We collected these quotes from things they published elsewhere, and every quote links to where it was said. They have no account here and have not endorsed this site. Quotes are word for word; the short line under each one is our own restatement, not their wording. Their own site. Is this you? Claim it or ask us to remove it. Or tell us what is wrong here.

Filter benchmarks · Newest first

Topic

All 6Anthropic 1benchmarks 1

Sort

NewestOldest

Only Aman Khan on benchmarks. See everyone on benchmarks

1 dated position, 2026, in their own words. Our reading of what Aman Khan has said — not written or endorsed by them.

1 position so far — this page is not yet offered to search engines.

1. 8 Sept 2026, 11:26 UTC

   Our readingBenchmark scores do not tell you whether a new model is better at your own work.

   Their wordsOnce a benchmark becomes a target, it can pull optimization toward the test itself. That’s the concern behind Goodhart’s Law, and it’s a reason to check more than the headline score. None of that tells you whether the new model is better at reviewing your product spec or catching the bug in your pull request.

   ↗Build your own vibe check in an afternoonamankhan1.substack.com 1st of 3 in this piece