korrents

On the map

Tap a claim on the ring to put it at the centre.

← Overfitting to an evaluation set can happen without directly training…

17 connected korrents · 16 moments from 5 Nov 2019 to 5 Nov 2026.

Everything filed under benchmarks benchmarks Everything filed under ARC-AGI ARC-AGI Everything filed under LLMs LLMs Everything filed under product management product management Everything filed under reinforcement learning reinforcement learning Everything filed under health and fitness health and fitness Everything filed under humility humility Everything filed under AI agents AI agents Same subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subject Read this korrent: Overfitting to an evaluation set can happen without directly training on it, contrary to a misconception common even among machine learning experts. Overfitting to an evaluation set canhappen without directly training on it,contrary to a misconception common evenamong machine learning experts. Last stated 2 years ago 6 Nov 2024 FC François Chollet — holds since 2024-11-06 — tap for who they are Same subject: Long-context evaluation benchmarks are likely contaminated by model training data, making them unreliable as a sole evaluation method — tap to centre the map on it Long-context evaluation benchmarksare likely contaminated by modeltraining data, making themunreliable as a sole evaluationmethod Last stated a year ago 22 Jun 2025 EY Eugene Yan — holds since 2025-06-22 — tap for who they are Same subject: Distillation carries a higher risk of overfitting, so it is not a free lunch. — tap to centre the map on it Distillation carries a higher riskof overfitting, so it is not a freelunch. Last stated a month ago 17 Oct 2026 SR Sebastian Raschka — holds since 2026-10-17 — tap for who they are Same subject: AI cannot turn an untested assumption into fact; that takes real customers and real data. — tap to centre the map on it AI cannot turn an untestedassumption into fact; that takesreal customers and real data. Last stated 2 months ago 10 Sept 2026 CV Claire Vo — holds since 2026-09-10 — tap for who they are Same subject: If a model answers every rollout correctly all the time, training might as well stop, because there is no learning signal left. — tap to centre the map on it If a model answers every rolloutcorrectly all the time, trainingmight as well stop, because there isno learning signal left. Last stated a month ago 10 Oct 2026 SR Sebastian Raschka — holds since 2026-10-10 — tap for who they are Same subject: Most people who think they are overtrained are probably only non-functionally overreached and need just a short back-off. — tap to centre the map on it Most people who think they areovertrained are probably onlynon-functionally overreached andneed just a short back-off. Last stated 4 years ago 15 Feb 2023 AG Andy Galpin — holds since 2023-02-15 — tap for who they are Same subject: How much machine learning can help a biological problem is set by the quality of the data around that problem, not by the algorithms. — tap to centre the map on it How much machine learning can help abiological problem is set by thequality of the data around thatproblem, not by the algorithms. Last stated 2 years ago 19 Sept 2024 DL Derek Lowe — holds since 2024-09-19 — tap for who they are Same subject: The argument that RAG and long context make fine-tuning unnecessary holds when information is missing, but not when the model struggles with the task itself. — tap to centre the map on it The argument that RAG and longcontext make fine-tuning unnecessaryholds when information is missing,but not when the model struggleswith the task itself. Last stated 2 weeks ago 5 Nov 2026 AR Aishwarya Naresh Reganti — holds since 2026-11-05 — tap for who they are Same subject: Every machine-learning deployment that has paid off so far left a person making the decision, which is the only reason imperfect models were useful. — tap to centre the map on it Every machine-learning deploymentthat has paid off so far left aperson making the decision, which isthe only reason imperfect modelswere useful. Last stated 3 months ago 12 Aug 2026 CF Chelsea Finn — holds since 2026-08-12 — tap for who they are Same subject: 'Everyone gets it wrong sometimes' is a bad argument when assessing track records. — tap to centre the map on it 'Everyone gets it wrong sometimes'is a bad argument when assessingtrack records. Last stated 4 months ago 2 Aug 2026 AK Adam Kucharski — holds since 2026-08-02 — tap for who they are Same subject: A benchmark result should be reported under a stated budget, or as a curve against test-time compute — never as a single number. — tap to centre the map on it A benchmark result should bereported under a stated budget, oras a curve against test-time compute— never as a single number. Last stated 5 months ago 26 Jun 2026 NB Noam Brown — holds since 2026-06-26 — tap for who they are Same subject: A benchmark that ranks Claude Code last while it stays first in use is measuring the wrong thing, and has been for a year. — tap to centre the map on it A benchmark that ranks Claude Codelast while it stays first in use ismeasuring the wrong thing, and hasbeen for a year. Last stated 3 months ago 3 Sept 2026 DR Dax Raad — holds since 2026-09-03 — tap for who they are Same subject: A unit of measure needs defining, perhaps via a chain of increasingly hard problems, each pair of which a single model can solve. — tap to centre the map on it A unit of measure needs defining,perhaps via a chain of increasinglyhard problems, each pair of which asingle model can solve. Last stated 2 months ago 15 Sept 2026 PG Paul Graham — holds since 2026-09-15 — tap for who they are Same subject: One-shot or zero-shot settings are the fairest comparisons to human performance and important targets for future work. — tap to centre the map on it One-shot or zero-shot settings arethe fairest comparisons to humanperformance and important targetsfor future work. Last stated 6 years ago 28 May 2020 DA Dario Amodei — holds since 2020-05-28 — tap for who they are Same subject: Performance gains from scaling on exam-style benchmarks are orthogonal to intelligence, because those benchmarks can be hacked by memorizing in advance. — tap to centre the map on it Performance gains from scaling onexam-style benchmarks are orthogonalto intelligence, because thosebenchmarks can be hacked bymemorizing in advance. Last stated 2 years ago 6 Nov 2024 FC François Chollet — holds since 2024-11-06 — tap for who they are Same subject: The Abstraction and Reasoning Corpus can measure human-like general fluid intelligence and enable fair comparisons between AI systems and humans. — tap to centre the map on it The Abstraction and Reasoning Corpuscan measure human-like general fluidintelligence and enable faircomparisons between AI systems andhumans. Last stated 7 years ago 5 Nov 2019 FC François Chollet — holds since 2019-11-05 — tap for who they are Same subject: Combining an avocado and a chair into one image is evidence a model conceptually understands both, not that it memorised pictures of them. — tap to centre the map on it Combining an avocado and a chairinto one image is evidence a modelconceptually understands both, notthat it memorised pictures of them. Last stated 3 months ago 12 Aug 2026 CF Chelsea Finn — holds since 2026-08-12 — tap for who they are Same subject: Passing ARC-AGI does not amount to achieving AGI: o3 still fails on some very easy tasks, indicating fundamental differences from human intelligence. — tap to centre the map on it Passing ARC-AGI does not amount toachieving AGI: o3 still fails onsome very easy tasks, indicatingfundamental differences from humanintelligence. Last stated 2 years ago 20 Dec 2024 FC François Chollet — holds since 2024-12-20 — tap for who they are
same subject or similar wordinga cloud: claims about one subject, named for itbar: when it was last stated, on a scale from 2015 to today — full is todaya face: someone who holds the claim — tap it for who they are

At the centre Overfitting to an evaluation set can happen without directly training on it, contrary to a misconception common even among machine learning experts. Last stated 6 Nov 2024 · 2 years ago Holds François Chollet Read this korrent →