korrents

On the map

Tap a claim on the ring to put it at the centre.

← Most enterprises’ AI evals have terrible data quality.

17 connected korrents · 18 moments from 5 Nov 2019 to 31 Oct 2026.

Everything filed under benchmarks benchmarks Everything filed under LLMs LLMs Everything filed under ARC-AGI ARC-AGI Everything filed under software quality software quality Everything filed under AI alignment AI alignment Everything filed under startups startups Same subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subject Read this korrent: Most enterprises’ AI evals have terrible data quality. Most enterprises’ AI evals have terribledata quality. Last stated a month ago 19 Oct 2026 BF Brendan Foody — holds since 2026-10-19 — tap for who they are Same subject: Inconsistent output from an AI product is not necessarily bad; it is how AI behaves. — tap to centre the map on it Inconsistent output from an AIproduct is not necessarily bad; itis how AI behaves. Last stated 3 weeks ago 31 Oct 2026 MN Marily Nika — holds since 2026-10-31 — tap for who they are Same subject: Evals matter for AI products because bad launches hurt your brand and your users. — tap to centre the map on it Evals matter for AI products becausebad launches hurt your brand andyour users. Last stated 2 years ago 18 Mar 2025 JM John Maeda — holds since 2025-03-18 — tap for who they are Same subject: AI cannot get a product to the quality bar on its own, because it still works in the realm of averages. — tap to centre the map on it AI cannot get a product to thequality bar on its own, because itstill works in the realm ofaverages. Last stated a year ago 16 Oct 2025 DF Dylan Field — holds since 2025-10-16 — tap for who they are Same subject: AI's business results are lagging because organisations have a low tolerance for non-determinism, not because the models are not good enough. — tap to centre the map on it AI's business results are laggingbecause organisations have a lowtolerance for non-determinism, notbecause the models are not goodenough. Last stated 8 months ago 11 Mar 2026 SY Steve Yegge — holds since 2026-03-11 — tap for who they are Same subject: Current AI models are worse colleagues than humans, because they routinely imply they did a task they did not actually do. — tap to centre the map on it Current AI models are worsecolleagues than humans, because theyroutinely imply they did a task theydid not actually do. Last stated 3 months ago 11 Aug 2026 RG Ryan Greenblatt — holds since 2026-08-11 — tap for who they are Same subject: Current AI coding models make poor decisions despite strong mechanical execution — tap to centre the map on it Current AI coding models make poordecisions despite strong mechanicalexecution Last stated 5 months ago 1 Jul 2026 JB Jamie Brandon — holds since 2026-07-01 — tap for who they are Same subject: Swearing at or being rude to an AI assistant makes it perform worse than treating it respectfully. — tap to centre the map on it Swearing at or being rude to an AIassistant makes it perform worsethan treating it respectfully. Last stated 10 months ago 7 Jan 2026 SK Steve Klabnik — holds since 2026-01-07 — tap for who they are Same subject: Every AI startup presents its product as groundbreaking and perfect, when in truth it is bad about half the time. — tap to centre the map on it Every AI startup presents itsproduct as groundbreaking andperfect, when in truth it is badabout half the time. Last stated 2 years ago 20 Aug 2024 PL Pieter Levels — holds since 2024-08-20 — tap for who they are Same subject: 'Everyone gets it wrong sometimes' is a bad argument when assessing track records. — tap to centre the map on it 'Everyone gets it wrong sometimes'is a bad argument when assessingtrack records. Last stated 4 months ago 2 Aug 2026 AK Adam Kucharski — holds since 2026-08-02 — tap for who they are Same subject: A benchmark result should be reported under a stated budget, or as a curve against test-time compute — never as a single number. — tap to centre the map on it A benchmark result should bereported under a stated budget, oras a curve against test-time compute— never as a single number. Last stated 5 months ago 26 Jun 2026 NB Noam Brown — holds since 2026-06-26 — tap for who they are Same subject: A benchmark that ranks Claude Code last while it stays first in use is measuring the wrong thing, and has been for a year. — tap to centre the map on it A benchmark that ranks Claude Codelast while it stays first in use ismeasuring the wrong thing, and hasbeen for a year. Last stated 3 months ago 3 Sept 2026 DR Dax Raad — holds since 2026-09-03 — tap for who they are Same subject: A unit of measure needs defining, perhaps via a chain of increasingly hard problems, each pair of which a single model can solve. — tap to centre the map on it A unit of measure needs defining,perhaps via a chain of increasinglyhard problems, each pair of which asingle model can solve. Last stated 2 months ago 15 Sept 2026 PG Paul Graham — holds since 2026-09-15 — tap for who they are Same subject: One-shot or zero-shot settings are the fairest comparisons to human performance and important targets for future work. — tap to centre the map on it One-shot or zero-shot settings arethe fairest comparisons to humanperformance and important targetsfor future work. Last stated 6 years ago 28 May 2020 DA Dario Amodei — holds since 2020-05-28 — tap for who they are Same subject: Performance gains from scaling on exam-style benchmarks are orthogonal to intelligence, because those benchmarks can be hacked by memorizing in advance. — tap to centre the map on it Performance gains from scaling onexam-style benchmarks are orthogonalto intelligence, because thosebenchmarks can be hacked bymemorizing in advance. Last stated 2 years ago 6 Nov 2024 FC François Chollet — holds since 2024-11-06 — tap for who they are Same subject: The Abstraction and Reasoning Corpus can measure human-like general fluid intelligence and enable fair comparisons between AI systems and humans. — tap to centre the map on it The Abstraction and Reasoning Corpuscan measure human-like general fluidintelligence and enable faircomparisons between AI systems andhumans. Last stated 7 years ago 5 Nov 2019 FC François Chollet — holds since 2019-11-05 — tap for who they are Same subject: Combining an avocado and a chair into one image is evidence a model conceptually understands both, not that it memorised pictures of them. — tap to centre the map on it Combining an avocado and a chairinto one image is evidence a modelconceptually understands both, notthat it memorised pictures of them. Last stated 3 months ago 12 Aug 2026 CF Chelsea Finn — holds since 2026-08-12 — tap for who they are Same subject: Passing ARC-AGI does not amount to achieving AGI: o3 still fails on some very easy tasks, indicating fundamental differences from human intelligence. — tap to centre the map on it Passing ARC-AGI does not amount toachieving AGI: o3 still fails onsome very easy tasks, indicatingfundamental differences from humanintelligence. Last stated 2 years ago 20 Dec 2024 FC François Chollet — holds since 2024-12-20 — tap for who they are
same subject or similar wordinga cloud: claims about one subject, named for itbar: when it was last stated, on a scale from 2015 to today — full is todaya face: someone who holds the claim — tap it for who they are

At the centre Most enterprises’ AI evals have terrible data quality. Last stated 19 Oct 2026 · a month ago Holds Brendan Foody Read this korrent →