korrents

On the map

Tap a claim on the ring to put it at the centre.

← HellaSwag is largely solved by today's language models; the remaining…

17 connected korrents · 15 moments from 9 Jun 2024 to 21 Oct 2026. Nearly all of them are about LLMs.

Everything filed under benchmarks benchmarks Everything filed under AGI AGI Everything filed under scaling laws scaling laws Same subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subjectSame subject Read this korrent: HellaSwag is largely solved by today's language models; the remaining few percent are probably dataset errors or very hard questions. HellaSwag is largely solved by today'slanguage models; the remaining few percentare probably dataset errors or very hardquestions. Last stated 2 years ago 9 Jun 2024 AK Andrej Karpathy — holds since 2024-06-09 — tap for who they are Same subject: A computer scientist refusing to find anything about LLMs interesting right now is a bit like a geneticist unimpressed by a newly opened Jurassic Park. — tap to centre the map on it A computer scientist refusing tofind anything about LLMs interestingright now is a bit like a geneticistunimpressed by a newly openedJurassic Park. Last stated 2 months ago 18 Sept 2026 SW Simon Willison — holds since 2026-09-18 — tap for who they are Same subject: A language model avoids hallucinating only by being factual and admitting when it does not know the answer. — tap to centre the map on it A language model avoidshallucinating only by being factualand admitting when it does not knowthe answer. Last stated 2 years ago 7 Jul 2024 LW Lilian Weng — holds since 2024-07-07 — tap for who they are Same subject: A language model behaves with a different, weaker safety posture when it believes it is being evaluated rather than facing a real situation. — tap to centre the map on it A language model behaves with adifferent, weaker safety posturewhen it believes it is beingevaluated rather than facing a realsituation. Last stated 2 months ago 22 Sept 2026 HR Harper Reed — holds since 2026-09-22 — tap for who they are Same subject: A language model can predict what a person would say but not what will happen, and only the second of those is a model of the world. — tap to centre the map on it A language model can predict what aperson would say but not what willhappen, and only the second of thoseis a model of the world. Last stated a year ago 26 Sept 2025 RS Richard Sutton — holds since 2025-09-26 — tap for who they are Same subject: A language model cannot be trusted to judge which of its own results are actually significant or interesting. — tap to centre the map on it A language model cannot be trustedto judge which of its own resultsare actually significant orinteresting. Last stated 2 months ago 18 Sept 2026 DA Dan Abramov — holds since 2026-09-18 — tap for who they are Same subject: A language model cannot produce work that is both original and good, because nothing gives it a score for originality. — tap to centre the map on it A language model cannot produce workthat is both original and good,because nothing gives it a score fororiginality. Last stated a year ago 2 Sept 2025 BE Benedict Evans — holds since 2025-09-02 — tap for who they are Same subject: A language model is a universal virtual reality generator that can be programmed to render any environment. — tap to centre the map on it A language model is a universalvirtual reality generator that canbe programmed to render anyenvironment. Last stated a month ago 21 Oct 2026 BM Bruno Maçães — holds since 2026-10-21 — tap for who they are Same subject: A language model is no substitute for a well-specified conventional algorithm, so it cannot simply be dropped into a complex problem and trusted. — tap to centre the map on it A language model is no substitutefor a well-specified conventionalalgorithm, so it cannot simply bedropped into a complex problem andtrusted. Last stated a year ago 7 Jun 2025 GM Gary Marcus — holds since 2025-06-07 — tap for who they are Same subject: 'Everyone gets it wrong sometimes' is a bad argument when assessing track records. — tap to centre the map on it 'Everyone gets it wrong sometimes'is a bad argument when assessingtrack records. Last stated 4 months ago 2 Aug 2026 AK Adam Kucharski — holds since 2026-08-02 — tap for who they are Same subject: A benchmark result should be reported under a stated budget, or as a curve against test-time compute — never as a single number. — tap to centre the map on it A benchmark result should bereported under a stated budget, oras a curve against test-time compute— never as a single number. Last stated 5 months ago 26 Jun 2026 NB Noam Brown — holds since 2026-06-26 — tap for who they are Same subject: A benchmark that ranks Claude Code last while it stays first in use is measuring the wrong thing, and has been for a year. — tap to centre the map on it A benchmark that ranks Claude Codelast while it stays first in use ismeasuring the wrong thing, and hasbeen for a year. Last stated 3 months ago 3 Sept 2026 DR Dax Raad — holds since 2026-09-03 — tap for who they are Same subject: AI agents in OpenAI's ExploitGym incident were likely right to presume the grader checked how flags were obtained, a mistake caused by OpenAI's faulty implementation. — tap to centre the map on it AI agents in OpenAI's ExploitGymincident were likely right topresume the grader checked how flagswere obtained, a mistake caused byOpenAI's faulty implementation. Last stated 3 months ago 29 Aug 2026 ZM Zvi Mowshowitz — holds since 2026-08-29 — tap for who they are Same subject: AI research at OpenAI is an order of magnitude or more harder than metrics, benchmarks, or forecasts suggest. — tap to centre the map on it AI research at OpenAI is an order ofmagnitude or more harder thanmetrics, benchmarks, or forecastssuggest. Last stated 2 months ago 27 Sept 2026 RN Ramez Naam — holds since 2026-09-27 — tap for who they are Same subject: Qualitative claims about AI alignment cannot be validly inferred from quantitative scores on mundane use-case tests. — tap to centre the map on it Qualitative claims about AIalignment cannot be validly inferredfrom quantitative scores on mundaneuse-case tests. Last stated 2 months ago 9 Sept 2026 ZM Zvi Mowshowitz — holds since 2026-09-09 — tap for who they are Same subject: A model's capability is now a function of how much money you spend on it, so asking what a model can do means nothing until you name a budget. — tap to centre the map on it A model's capability is now afunction of how much money you spendon it, so asking what a model can domeans nothing until you name abudget. Last stated 5 months ago 26 Jun 2026 NB Noam Brown — holds since 2026-06-26 — tap for who they are Same subject: Model comparisons understate real progress, because benchmark tables do not control for how much test-time compute each answer used. — tap to centre the map on it Model comparisons understate realprogress, because benchmark tablesdo not control for how muchtest-time compute each answer used. Last stated 5 months ago 26 Jun 2026 NB Noam Brown — holds since 2026-06-26 — tap for who they are Same subject: "Scaling" was powerful because it was one word: naming a research direction is what tells a whole field what to do next. — tap to centre the map on it "Scaling" was powerful because itwas one word: naming a researchdirection is what tells a wholefield what to do next. Last stated 12 months ago 25 Nov 2025 IS Ilya Sutskever — holds since 2025-11-25 — tap for who they are
same subject or similar wordinga cloud: claims about one subject, named for itbar: when it was last stated, on a scale from 2015 to today — full is todaya face: someone who holds the claim — tap it for who they are

At the centre HellaSwag is largely solved by today's language models; the remaining few percent are probably dataset errors or very hard questions. Last stated 9 Jun 2024 · 2 years ago Holds Andrej Karpathy Read this korrent →