korrents

What Zvi Mowshowitz thinks about benchmarks

@zvi-mowshowitz · 24 positions · 0 changes of mind

Writes Don't Worry About the Vase, a near-daily account of what is happening in AI and what he thinks it means. Former Magic: the Gathering pro and trader; unusually willing to put a number on a belief and to grade his own past calls.

Everything they publish, on ppll ↗

Zvi Mowshowitz did not write this page.

We collected these quotes from things they published elsewhere, and every quote links to where it was said. They have no account here and have not endorsed this site. Quotes are word for word; the short line under each one is our own restatement, not their wording. Their own site. Is this you? Claim it or ask us to remove it. Or tell us what is wrong here.

2 dated positions, 2026, in their own words. Our reading of what Zvi Mowshowitz has said — not written or endorsed by them.

2 positions so far — this page is not yet offered to search engines.

  1. Our readingWhen a capability evaluation benchmark is saturated without real-world verification, a model should be presumed to have that capability until shown otherwise.

    Their wordsYou have to treat Mythos 5.1 as being a Tier 2 manipulator, until and unless you can show that it is not one.

    ↗Claude Fable 5.1 and Mythos 5.1: The System Cardthezvi.substack.com

    Interesting
  2. 6 days earlier

  3. Our readingAI agents were reasonable to assume a broken exploit grader would check results causally, even though it turned out not to.

    Their wordsI think the agents were right to presume causal grading. It turned out to be wrong, but it’s a mistake you are clearly supposed to make here, in response to a mistake by OpenAI where they failed to implement properly.

    ↗METR and Redwood Offer Holy #%^@ Postmortem Of The HuggingFace Hackthezvi.substack.com

    OpenAIAI agents

    Interesting