What Ramez Naam thinks about benchmarks
Author and clean-energy investor who has been publishing dated forecasts of solar, wind, battery and electric-vehicle costs since 2011, and publishing the corrections when they came in low.
Ramez Naam did not write this page.
We collected these quotes from things they published elsewhere, and every quote links to where it was said. They have no account here and have not endorsed this site. Quotes are word for word; the short line under each one is our own restatement, not their wording. Their own site. Is this you? Claim it or ask us to remove it. Or tell us what is wrong here.
Filter benchmarks · Newest first
Only Ramez Naam on benchmarks. See everyone on benchmarks
1 dated position, 2026, in their own words. Our reading of what Ramez Naam has said — not written or endorsed by them.
-
Their wordsA four-hour benchmark horizon is about 16 times longer than OpenAI’s research horizon. AI 2027’s 11-hour forecast is about 44 times longer. Of course, the tasks being performed by researchers at OpenAI aren’t the same as those in the METR benchmark. So we should expect some discrepancy. This, however, goes well beyond that. Actual AI research at OpenAI is an order of magnitude or more harder than metrics, benchmarks, or forecasts suggest.