Evals are not the durable asset people think they are: one survives about one to three model generations before it saturates and has to be thrown away.

Added to korrents 27 Jul 2026 · How quotes work · Something wrong? Tell us

Related korrents

Our reading — they may agree, disagree or merely touch the same thing. Closest first: a shared subject counts for most, then how near the wording is.

• Models look far better on evals than they are in the world because researchers, inadvertently, take inspiration from the evals when they build RL environments. ISIlya Sutskever25 Nov 2025similar wording
• Before building on a gap in the general models, work out whether that gap survives six months or three years. JDJeff Dean30 Jul 2026similar wording
• A year from now every coding model on the market will still take a bad instruction at face value and go fix the listed issues, where a human senior engineer would insist on a rewrite. DSDan Shipper24 May 2026similar wording
• You can now generate ten thousand variants of a function faster than you could write it once, and that economics pushes software towards replacement rather than repair. CMCharity Majors12 Aug 2026similar wording
• Running a model until its performance plateaus is no longer a usable evaluation rule, because a well-scaffolded model keeps improving for weeks. NBNoam Brown26 Jun 2026similar wording

See this on the map →