What Simon Willison thinks about benchmarks

@simon-willison · 54 positions · On ppll ↗

Co-creator of Django and creator of Datasette; writes daily at simonwillison.net.

Simon Willison did not write this page.

We collected these quotes from things they published elsewhere, and every quote links to where it was said. They have no account here and have not endorsed this site. Quotes are word for word; the short line under each one is our own restatement, not their wording. Their own site. Is this you? Claim it or ask us to remove it. Or tell us what is wrong here.

Filter benchmarks · Newest first

Topic

All 54benchmarks 1coding agents 12LLMs 12Anthropic 11OpenAI 6cybersecurity 4design 4prompt injection 4software quality 4

11 more topics

open source 3taste 3AI agents 2vibe coding 2behavioural science 1data centers 1energy 1HuggingFace 1innovation 1journalism 1MCP 1

Sort

NewestOldest

Only Simon Willison on benchmarks. See everyone on benchmarks

1 dated position, 2026, in their own words. Our reading of what Simon Willison has said — not written or endorsed by them.

1 position so far — this page is not yet offered to search engines.

1. 1 Sept 2026, 23:57 UTC

   Our readingThe pelican-drawing benchmark's link to how good models are at other tasks didn't seem to hold as strongly as in 2025.

   Their wordsits connection to how good the models were at other tasks didn’t seem to hold as strongly as it did back in 2025

   ↗Claude Fable 5.1 made me a really nice animated pelicansimonwillison.net