What Sayash Kapoor thinks about benchmarks

@sayash-kapoor · 20 positions · On ppll ↗

PhD candidate in computer science at Princeton's Center for Information Technology Policy; researches the societal impact of AI and co-writes AI as Normal Technology with Arvind Narayanan.

Sayash Kapoor did not write this page.

We collected these quotes from things they published elsewhere, and every quote links to where it was said. They have no account here and have not endorsed this site. Quotes are word for word; the short line under each one is our own restatement, not their wording. Their own site. Is this you? Claim it or ask us to remove it. Or tell us what is wrong here.

Filter benchmarks · Newest first

Topic

All 20AI agents 3benchmarks 3AI alignment 1Google 1LLMs 1recursive self-improvement 1reinforcement learning 1startups 1

Sort

NewestOldest

Only Sayash Kapoor on benchmarks. See everyone on benchmarks

3 dated positions, 2026, in their own words. Our reading of what Sayash Kapoor has said — not written or endorsed by them.

3 positions so far — this page is not yet offered to search engines.

1. 22 May 2026

   Our readingBenchmark evaluation is effectively impossible for long-horizon real-world agent tasks, for many reasons including cost.

   Their wordsbenchmark evaluation is effectively impossible for this kind of task for many reasons including cost.

   ↗Did Google’s AI agents really build an operating system for $916?normaltech.ai 3rd of 3 in this piece

2. 5 weeks earlier

3. 16 Apr 2026

   Our readingAnything precise enough to benchmark is also precise enough to optimize for, and modern RL training increasingly resembles the shape of the benchmarks themselves.

   reinforcement learning

   Their wordsBut whatever is precise enough to benchmark is also precise enough to optimize for, and modern RL training runs increasingly resemble the shape of the benchmarks themselves.

   ↗Open-world evaluations for measuring frontier AI capabilitiesnormaltech.ai 2nd of 3 in this piece

5. 16 Apr 2026

   Our readingAverage performance on a benchmark suite is very different from the upper bound of what an AI agent can do.

   Their wordsAverage performance is very different from upper-bound capability elicitation. Trying to run dozens (or 100s) of tasks in a benchmark suite is only necessary to measure average performance.

   ↗Open-world evaluations for measuring frontier AI capabilitiesnormaltech.ai 3rd of 3 in this piece