Tap a claim on the ring to put it at the centre.
← Overfitting to an evaluation set can happen without directly training…
17 connected korrents · 16 moments from 5 Nov 2019 to 5 Nov 2026.
Everything filed under benchmarks
benchmarks
Everything filed under ARC-AGI
ARC-AGI
Everything filed under LLMs
LLMs
Everything filed under product management
product management
Everything filed under reinforcement learning
reinforcement learning
Everything filed under health and fitness
health and fitness
Everything filed under humility
humility
Everything filed under AI agents
AI agents
Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject
Read this korrent: Overfitting to an evaluation set can happen without directly training on it, contrary to a misconception common even among machine learning experts.
Overfitting to an evaluation set can happen without directly training on it, contrary to a misconception common even among machine learning experts.
Last stated 2 years ago
6 Nov 2024
FC
François Chollet — holds since 2024-11-06 — tap for who they are
Same subject: Long-context evaluation benchmarks are likely contaminated by model training data, making them unreliable as a sole evaluation method — tap to centre the map on it
Long-context evaluation benchmarks are likely contaminated by model training data, making them unreliable as a sole evaluation method
Last stated a year ago
22 Jun 2025
EY
Eugene Yan — holds since 2025-06-22 — tap for who they are
Same subject: Distillation carries a higher risk of overfitting, so it is not a free lunch. — tap to centre the map on it
Distillation carries a higher risk of overfitting, so it is not a free lunch.
Last stated a month ago
17 Oct 2026
SR
Sebastian Raschka — holds since 2026-10-17 — tap for who they are
Same subject: AI cannot turn an untested assumption into fact; that takes real customers and real data. — tap to centre the map on it
AI cannot turn an untested assumption into fact; that takes real customers and real data.
Last stated 2 months ago
10 Sept 2026
CV
Claire Vo — holds since 2026-09-10 — tap for who they are
Same subject: If a model answers every rollout correctly all the time, training might as well stop, because there is no learning signal left. — tap to centre the map on it
If a model answers every rollout correctly all the time, training might as well stop, because there is no learning signal left.
Last stated a month ago
10 Oct 2026
SR
Sebastian Raschka — holds since 2026-10-10 — tap for who they are
Same subject: Most people who think they are overtrained are probably only non-functionally overreached and need just a short back-off. — tap to centre the map on it
Most people who think they are overtrained are probably only non-functionally overreached and need just a short back-off.
Last stated 4 years ago
15 Feb 2023
AG
Andy Galpin — holds since 2023-02-15 — tap for who they are
Same subject: How much machine learning can help a biological problem is set by the quality of the data around that problem, not by the algorithms. — tap to centre the map on it
How much machine learning can help a biological problem is set by the quality of the data around that problem, not by the algorithms.
Last stated 2 years ago
19 Sept 2024
DL
Derek Lowe — holds since 2024-09-19 — tap for who they are
Same subject: The argument that RAG and long context make fine-tuning unnecessary holds when information is missing, but not when the model struggles with the task itself. — tap to centre the map on it
The argument that RAG and long context make fine-tuning unnecessary holds when information is missing, but not when the model struggles with the task itself.
Last stated 2 weeks ago
5 Nov 2026
AR
Aishwarya Naresh Reganti — holds since 2026-11-05 — tap for who they are
Same subject: Every machine-learning deployment that has paid off so far left a person making the decision, which is the only reason imperfect models were useful. — tap to centre the map on it
Every machine-learning deployment that has paid off so far left a person making the decision, which is the only reason imperfect models were useful.
Last stated 3 months ago
12 Aug 2026
CF
Chelsea Finn — holds since 2026-08-12 — tap for who they are
Same subject: 'Everyone gets it wrong sometimes' is a bad argument when assessing track records. — tap to centre the map on it
'Everyone gets it wrong sometimes' is a bad argument when assessing track records.
Last stated 4 months ago
2 Aug 2026
AK
Adam Kucharski — holds since 2026-08-02 — tap for who they are
Same subject: A benchmark result should be reported under a stated budget, or as a curve against test-time compute — never as a single number. — tap to centre the map on it
A benchmark result should be reported under a stated budget, or as a curve against test-time compute — never as a single number.
Last stated 5 months ago
26 Jun 2026
NB
Noam Brown — holds since 2026-06-26 — tap for who they are
Same subject: A benchmark that ranks Claude Code last while it stays first in use is measuring the wrong thing, and has been for a year. — tap to centre the map on it
A benchmark that ranks Claude Code last while it stays first in use is measuring the wrong thing, and has been for a year.
Last stated 3 months ago
3 Sept 2026
DR
Dax Raad — holds since 2026-09-03 — tap for who they are
Same subject: A unit of measure needs defining, perhaps via a chain of increasingly hard problems, each pair of which a single model can solve. — tap to centre the map on it
A unit of measure needs defining, perhaps via a chain of increasingly hard problems, each pair of which a single model can solve.
Last stated 2 months ago
15 Sept 2026
PG
Paul Graham — holds since 2026-09-15 — tap for who they are
Same subject: One-shot or zero-shot settings are the fairest comparisons to human performance and important targets for future work. — tap to centre the map on it
One-shot or zero-shot settings are the fairest comparisons to human performance and important targets for future work.
Last stated 6 years ago
28 May 2020
DA
Dario Amodei — holds since 2020-05-28 — tap for who they are
Same subject: Performance gains from scaling on exam-style benchmarks are orthogonal to intelligence, because those benchmarks can be hacked by memorizing in advance. — tap to centre the map on it
Performance gains from scaling on exam-style benchmarks are orthogonal to intelligence, because those benchmarks can be hacked by memorizing in advance.
Last stated 2 years ago
6 Nov 2024
FC
François Chollet — holds since 2024-11-06 — tap for who they are
Same subject: The Abstraction and Reasoning Corpus can measure human-like general fluid intelligence and enable fair comparisons between AI systems and humans. — tap to centre the map on it
The Abstraction and Reasoning Corpus can measure human-like general fluid intelligence and enable fair comparisons between AI systems and humans.
Last stated 7 years ago
5 Nov 2019
FC
François Chollet — holds since 2019-11-05 — tap for who they are
Same subject: Combining an avocado and a chair into one image is evidence a model conceptually understands both, not that it memorised pictures of them. — tap to centre the map on it
Combining an avocado and a chair into one image is evidence a model conceptually understands both, not that it memorised pictures of them.
Last stated 3 months ago
12 Aug 2026
CF
Chelsea Finn — holds since 2026-08-12 — tap for who they are
Same subject: Passing ARC-AGI does not amount to achieving AGI: o3 still fails on some very easy tasks, indicating fundamental differences from human intelligence. — tap to centre the map on it
Passing ARC-AGI does not amount to achieving AGI: o3 still fails on some very easy tasks, indicating fundamental differences from human intelligence.
Last stated 2 years ago
20 Dec 2024
FC
François Chollet — holds since 2024-12-20 — tap for who they are
same subject or similar wording a cloud: claims about one subject, named for it bar: when it was last stated, on a scale from 2015 to today — full is today a face: someone who holds the claim — tap it for who they are
At the centre
Overfitting to an evaluation set can happen without directly training on it, contrary to a misconception common even among machine learning experts.
Last stated 6 Nov 2024 · 2 years ago
Holds François Chollet
Read this korrent →
Similar wording
Long-context evaluation benchmarks are likely contaminated by model training data, making them unreliable as a sole evaluation method
Last stated 22 Jun 2025 · a year ago
Holds Eugene Yan
Similar wording
Distillation carries a higher risk of overfitting, so it is not a free lunch.
Last stated 17 Oct 2026 · a month ago
Holds Sebastian Raschka
Similar wording
AI cannot turn an untested assumption into fact; that takes real customers and real data.
Last stated 10 Sept 2026 · 2 months ago
Holds Claire Vo
Similar wording
If a model answers every rollout correctly all the time, training might as well stop, because there is no learning signal left.
Last stated 10 Oct 2026 · a month ago
Holds Sebastian Raschka
Similar wording
Most people who think they are overtrained are probably only non-functionally overreached and need just a short back-off.
Last stated 15 Feb 2023 · 4 years ago
Holds Andy Galpin
Similar wording
How much machine learning can help a biological problem is set by the quality of the data around that problem, not by the algorithms.
Last stated 19 Sept 2024 · 2 years ago
Holds Derek Lowe
Similar wording
The argument that RAG and long context make fine-tuning unnecessary holds when information is missing, but not when the model struggles with the task itself.
Last stated 5 Nov 2026 · 2 weeks ago
Holds Aishwarya Naresh Reganti
Similar wording
Every machine-learning deployment that has paid off so far left a person making the decision, which is the only reason imperfect models were useful.
Last stated 12 Aug 2026 · 3 months ago
Holds Chelsea Finn
Same subject: benchmarks
'Everyone gets it wrong sometimes' is a bad argument when assessing track records.
Last stated 2 Aug 2026 · 4 months ago
Holds Adam Kucharski
Same subject: benchmarks
A benchmark result should be reported under a stated budget, or as a curve against test-time compute — never as a single number.
Last stated 26 Jun 2026 · 5 months ago
Holds Noam Brown
Same subject: benchmarks
A benchmark that ranks Claude Code last while it stays first in use is measuring the wrong thing, and has been for a year.
Last stated 3 Sept 2026 · 3 months ago
Holds Dax Raad
Same subject: measuring intelligence
A unit of measure needs defining, perhaps via a chain of increasingly hard problems, each pair of which a single model can solve.
Last stated 15 Sept 2026 · 2 months ago
Holds Paul Graham
Same subject: measuring intelligence
One-shot or zero-shot settings are the fairest comparisons to human performance and important targets for future work.
Last stated 28 May 2020 · 6 years ago
Holds Dario Amodei
Same subject: measuring intelligence
Performance gains from scaling on exam-style benchmarks are orthogonal to intelligence, because those benchmarks can be hacked by memorizing in advance.
Last stated 6 Nov 2024 · 2 years ago
Holds François Chollet
Same subject: ARC-AGI
The Abstraction and Reasoning Corpus can measure human-like general fluid intelligence and enable fair comparisons between AI systems and humans.
Last stated 5 Nov 2019 · 7 years ago
Holds François Chollet
Same subject: ARC-AGI
Combining an avocado and a chair into one image is evidence a model conceptually understands both, not that it memorised pictures of them.
Last stated 12 Aug 2026 · 3 months ago
Holds Chelsea Finn
Same subject: ARC-AGI
Passing ARC-AGI does not amount to achieving AGI: o3 still fails on some very easy tasks, indicating fundamental differences from human intelligence.
Last stated 20 Dec 2024 · 2 years ago
Holds François Chollet