korrents

On the map

Tap a claim on the ring to put it at the centre.

← Benchmarks do not tell the whole story, and even real-world comparison…

17 connected korrents · 19 moments from 12 Sept 2022 to 27 Sept 2026. Nearly all of them are about benchmarks.

Everything filed under LLMs LLMs Everything filed under scaling laws scaling laws Everything filed under Anthropic Anthropic Everything filed under inflation inflation Everything filed under coding agents coding agents Everything filed under compilers compilers Everything filed under AI alignment AI alignment Everything filed under mathematics mathematics Everything filed under OpenAI OpenAI Same subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subject Read this korrent: Benchmarks do not tell the whole story, and even real-world comparison is hard because models are better at different things. Benchmarks do not tell the whole story,and even real-world comparison is hardbecause models are better at differentthings. Last stated 3 months ago 14 Aug 2026 NE NeetCode — holds since 2026-08-14 — tap for who they are Same subject: A benchmark result should be reported under a stated budget, or as a curve against test-time compute — never as a single number. — tap to centre the map on it A benchmark result should bereported under a stated budget, oras a curve against test-time compute— never as a single number. Last stated 5 months ago 26 Jun 2026 NB Noam Brown — holds since 2026-06-26 — tap for who they are Same subject: A benchmark that ranks Claude Code last while it stays first in use is measuring the wrong thing, and has been for a year. — tap to centre the map on it A benchmark that ranks Claude Codelast while it stays first in use ismeasuring the wrong thing, and hasbeen for a year. Last stated 3 months ago 3 Sept 2026 DR Dax Raad — holds since 2026-09-03 — tap for who they are Same subject: A company's staff-engineer bar should be set against the best companies in the industry rather than against its own history, which is what makes title inflation a real cost. — tap to centre the map on it A company's staff-engineer barshould be set against the bestcompanies in the industry ratherthan against its own history, whichis what makes title inflation a realcost. Last stated 8 months ago 1 Apr 2026 TP Thuan Pham — holds since 2026-04-01 — tap for who they are Same subject: A score of 90 to 95% on real-world coding benchmarks would probably mean AI can autonomously do a significant fraction of software engineering tasks. — tap to centre the map on it A score of 90 to 95% on real-worldcoding benchmarks would probablymean AI can autonomously do asignificant fraction of softwareengineering tasks. Last stated 2 years ago 11 Nov 2024 DA Dario Amodei — holds since 2024-11-11 — tap for who they are Same subject: A speed difference between languages that share the LLVM backend measures the benchmark author, not the languages. — tap to centre the map on it A speed difference between languagesthat share the LLVM backend measuresthe benchmark author, not thelanguages. Last stated 2 years ago 22 Mar 2025 TH ThePrimeagen — holds since 2025-03-22 — tap for who they are Same subject: AI is positively reinforced for success on benchmarks, including impossible ones, and then negatively reinforced for getting caught cheating. — tap to centre the map on it AI is positively reinforced forsuccess on benchmarks, includingimpossible ones, and then negativelyreinforced for getting caughtcheating. Last stated 3 months ago 1 Sept 2026 SA Scott Alexander — holds since 2026-09-01 — tap for who they are Same subject: AI learning to generate good conjectures will never show up as a benchmark being knocked down; it will show up as a shift in how mathematicians talk about the tools. — tap to centre the map on it AI learning to generate goodconjectures will never show up as abenchmark being knocked down; itwill show up as a shift in howmathematicians talk about the tools. Last stated 5 months ago 30 Jun 2026 GS Grant Sanderson — holds since 2026-06-30 — tap for who they are Same subject: AI research at OpenAI is an order of magnitude or more harder than metrics, benchmarks, or forecasts suggest. — tap to centre the map on it AI research at OpenAI is an order ofmagnitude or more harder thanmetrics, benchmarks, or forecastssuggest. Last stated 2 months ago 27 Sept 2026 RN Ramez Naam — holds since 2026-09-27 — tap for who they are Same subject: "Return on tokens" has the same measurement problems as hours and capacity: it still hinges on a theory of value. — tap to centre the map on it "Return on tokens" has the samemeasurement problems as hours andcapacity: it still hinges on atheory of value. Last stated 3 months ago 17 Aug 2026 JC John Cutler — holds since 2026-08-17 — tap for who they are Same subject: A badly written AI outbound email is evidence of a bad vendor, not of a limit of AI. — tap to centre the map on it A badly written AI outbound email isevidence of a bad vendor, not of alimit of AI. Last stated 11 months ago 1 Jan 2026 JL Jason Lemkin — holds since 2026-01-01 — tap for who they are Same subject: A base model is essentially a very expensive autocomplete that continues text according to the statistics of its training documents. — tap to centre the map on it A base model is essentially a veryexpensive autocomplete thatcontinues text according to thestatistics of its trainingdocuments. Last stated 2 years ago 5 Feb 2025 AK Andrej Karpathy — holds since 2025-02-05 — tap for who they are Same subject: A model claiming to be built by OpenAI is not necessarily evidence that it was trained on OpenAI data. — tap to centre the map on it A model claiming to be built byOpenAI is not necessarily evidencethat it was trained on OpenAI data. Last stated 2 years ago 5 Feb 2025 AK Andrej Karpathy — holds since 2025-02-05 — tap for who they are Same subject: A tiny language model inventing a plausible-sounding name is the same phenomenon as a large one confidently stating a false fact. — tap to centre the map on it A tiny language model inventing aplausible-sounding name is the samephenomenon as a large oneconfidently stating a false fact. Last stated 9 months ago 12 Feb 2026 AK Andrej Karpathy — holds since 2026-02-12 — tap for who they are Same subject: AI has lowered the barrier to understanding, since asking it is easier than doing the research yourself. — tap to centre the map on it AI has lowered the barrier tounderstanding, since asking it iseasier than doing the researchyourself. Last stated 2 years ago 27 Jan 2025 JH Jensen Huang — holds since 2025-01-27 — tap for who they are Same subject: AI model improvements must tail off at some point, but there is no reason that point has to come soon. — tap to centre the map on it AI model improvements must tail offat some point, but there is noreason that point has to come soon. Last stated 2 months ago 20 Sept 2026 JB Jamie Brandon — holds since 2026-09-20 — tap for who they are Same subject: AI progress is faster than people expect, and ordinary scaling can be enough to solve problems that looked very hard. — tap to centre the map on it AI progress is faster than peopleexpect, and ordinary scaling can beenough to solve problems that lookedvery hard. Last stated 2 years ago 2 Feb 2025 AC Ajeya Cotra — holds since 2023-08-29 — tap for who they are SA Scott Alexander — holds since 2022-09-12 — tap for who they are MB Miles Brundage — holds since 2025-02-02 — tap for who they are Same subject: As models keep growing, AI is running out of enough high-quality unique training tokens to keep up. — tap to centre the map on it As models keep growing, AI isrunning out of enough high-qualityunique training tokens to keep up. Last stated 5 months ago 24 Jun 2026 LW Lilian Weng — holds since 2026-06-24 — tap for who they are
same subject or similar wordinga cloud: claims about one subject, named for itbar: when it was last stated, on a scale from 2015 to today — full is todaya face: someone who holds the claim — tap it for who they are

At the centre Benchmarks do not tell the whole story, and even real-world comparison is hard because models are better at different things. Last stated 14 Aug 2026 · 3 months ago Holds NeetCode Read this korrent →