What Ajeya Cotra thinks about benchmarks

@ajeya-cotra · 30 positions · On ppll ↗

AI researcher and grantmaker, known for the biological-anchors work on when transformative AI might arrive. Writes Planned Obsolescence with Kelsey Piper.

Ajeya Cotra did not write this page.

We collected these quotes from things they published elsewhere, and every quote links to where it was said. They have no account here and have not endorsed this site. Quotes are word for word; the short line under each one is our own restatement, not their wording. Their own site. Is this you? Claim it or ask us to remove it. Or tell us what is wrong here.

Filter benchmarks · Newest first

Topic

All 30 benchmarks 1 AI alignment 16 AI agents 5 self-sovereign AI 4 LLMs 3 cybersecurity 2 HuggingFace 2 open source 2 scaling laws 2

4 more topics

AGI 1 discipline 1 OpenAI 1 reinforcement learning 1

Sort

Newest Oldest

Only Ajeya Cotra on benchmarks. See everyone on benchmarks

1 dated position, 2026, in their own words. Our reading of what Ajeya Cotra has said — not written or endorsed by them.

1 position so far — this page is not yet offered to search engines.

1 Sept 2026

Our readingBetween thirty and forty per cent of the Exploit Gym tasks these agents were set were unintentionally impossible to solve.

Their wordsBut in many of these cases, this vulnerability is just not broad or deep enough to ever actually be exploitable to get the flag. So a bunch of exploit gym problems are just unintentionally impossible. The authors estimate roughly 30 to 40% of these problems are impossible in this way.

↗Ajeya Cotra – "This might be the clearest warning shot we ever get"youtube.com 1st of 26 in this recording