Tap a claim on the ring to put it at the centre.
← The right bar for an LLM evaluator is human-level performance, not perfect accuracy.
17 connected korrents · 15 moments from 18 Mar 2024 to 2 Nov 2026. Nearly all of them are about benchmarks .
Everything filed under LLMs
LLMs
Everything filed under Anthropic
Anthropic
Everything filed under open source
open source
Everything filed under OpenAI
OpenAI
Everything filed under scaling laws
scaling laws
Everything filed under inflation
inflation
Everything filed under coding agents
coding agents
Everything filed under compilers
compilers
Everything filed under AI alignment
AI alignment
Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject Same subject
Read this korrent: The right bar for an LLM evaluator is human-level performance, not perfect accuracy.
The right bar for an LLM evaluator is human-level performance, not perfect accuracy.
Last stated 12 months ago
23 Nov 2025
EY
Eugene Yan — holds since 2025-11-23 — tap for who they are
Same subject: Language models perform better on benchmark problems released before their training data cutoff, indicating data contamination inflates scores. — tap to centre the map on it
Language models perform better on benchmark problems released before their training data cutoff, indicating data contamination inflates scores.
Last stated 3 years ago
13 May 2024
SR
Sebastian Ruder — holds since 2024-05-13 — tap for who they are
Same subject: Static benchmarks with an LLM judge are not something to be fully happy with as a long-term way to evaluate frontier systems. — tap to centre the map on it
Static benchmarks with an LLM judge are not something to be fully happy with as a long-term way to evaluate frontier systems.
Last stated 11 months ago
27 Dec 2025
NN
Neel Nanda — holds since 2025-12-27 — tap for who they are
Same subject: A benchmark result should be reported under a stated budget, or as a curve against test-time compute — never as a single number. — tap to centre the map on it
A benchmark result should be reported under a stated budget, or as a curve against test-time compute — never as a single number.
Last stated 5 months ago
26 Jun 2026
NB
Noam Brown — holds since 2026-06-26 — tap for who they are
Same subject: A benchmark that ranks Claude Code last while it stays first in use is measuring the wrong thing, and has been for a year. — tap to centre the map on it
A benchmark that ranks Claude Code last while it stays first in use is measuring the wrong thing, and has been for a year.
Last stated 3 months ago
3 Sept 2026
DR
Dax Raad — holds since 2026-09-03 — tap for who they are
Same subject: A company's staff-engineer bar should be set against the best companies in the industry rather than against its own history, which is what makes title inflation a real cost. — tap to centre the map on it
A company's staff-engineer bar should be set against the best companies in the industry rather than against its own history, which is what makes title inflation a real cost.
Last stated 8 months ago
1 Apr 2026
TP
Thuan Pham — holds since 2026-04-01 — tap for who they are
Same subject: A score of 90 to 95% on real-world coding benchmarks would probably mean AI can autonomously do a significant fraction of software engineering tasks. — tap to centre the map on it
A score of 90 to 95% on real-world coding benchmarks would probably mean AI can autonomously do a significant fraction of software engineering tasks.
Last stated 2 years ago
11 Nov 2024
DA
Dario Amodei — holds since 2024-11-11 — tap for who they are
Same subject: A speed difference between languages that share the LLVM backend measures the benchmark author, not the languages. — tap to centre the map on it
A speed difference between languages that share the LLVM backend measures the benchmark author, not the languages.
Last stated 2 years ago
22 Mar 2025
TH
ThePrimeagen — holds since 2025-03-22 — tap for who they are
Same subject: AI is positively reinforced for success on benchmarks, including impossible ones, and then negatively reinforced for getting caught cheating. — tap to centre the map on it
AI is positively reinforced for success on benchmarks, including impossible ones, and then negatively reinforced for getting caught cheating.
Last stated 3 months ago
1 Sept 2026
SA
Scott Alexander — holds since 2026-09-01 — tap for who they are
Same subject: A PHP vulnerability found with GPT-5.6 Sol will cause havoc this week. — tap to centre the map on it
A PHP vulnerability found with GPT-5.6 Sol will cause havoc this week.
Last stated a month ago
21 Oct 2026
PS
Peter Steinberger — holds since 2026-10-21 — tap for who they are
Same subject: A startup should not begin life as a nonprofit and bolt a for-profit arm on later, whatever OpenAI's own history suggests. — tap to centre the map on it
A startup should not begin life as a nonprofit and bolt a for-profit arm on later, whatever OpenAI's own history suggests.
Last stated 3 years ago
18 Mar 2024
SA
Sam Altman — holds since 2024-03-18 — tap for who they are
Same subject: Advertising was a necessary phase for the internet but a momentary industry, and an AI people pay for is better because they know the answers are not influenced by advertisers. — tap to centre the map on it
Advertising was a necessary phase for the internet but a momentary industry, and an AI people pay for is better because they know the answers are not influenced by advertisers.
Last stated 3 years ago
18 Mar 2024
SA
Sam Altman — holds since 2024-03-18 — tap for who they are
Same subject: A business making hundreds of millions from an open source project owes it money or contribution, and can be refused its trademarks if it gives neither. — tap to centre the map on it
A business making hundreds of millions from an open source project owes it money or contribution, and can be refused its trademarks if it gives neither.
Last stated 2 years ago
26 Sept 2024
MM
Matt Mullenweg — holds since 2024-09-26 — tap for who they are
Same subject: A codec does not really exist until everyone can decode it. — tap to centre the map on it
A codec does not really exist until everyone can decode it.
Last stated 6 months ago
31 May 2026
JK
Jean-Baptiste Kempf — holds since 2026-05-31 — tap for who they are
Same subject: A company that will not accept a project’s terms is free to go and use a more permissive project instead. — tap to centre the map on it
A company that will not accept a project’s terms is free to go and use a more permissive project instead.
Last stated 2 years ago
26 Sept 2024
MM
Matt Mullenweg — holds since 2024-09-26 — tap for who they are
Same subject: "I don't know" becomes a kind of superpower when you can work with Claude Code to figure things out. — tap to centre the map on it
"I don't know" becomes a kind of superpower when you can work with Claude Code to figure things out.
Last stated 8 months ago
19 Mar 2026
KP
Kris Puckett — holds since 2026-03-19 — tap for who they are
Same subject: A large share of Anthropic's revenue being consumption-based rather than subscriptions raises the question of how resilient that revenue is. — tap to centre the map on it
A large share of Anthropic's revenue being consumption-based rather than subscriptions raises the question of how resilient that revenue is.
Last stated 3 weeks ago
2 Nov 2026
AA
Azeem Azhar — holds since 2026-11-02 — tap for who they are
Same subject: A skill is a set of repeatable, specific instructions that augments what Claude Code can do. — tap to centre the map on it
A skill is a set of repeatable, specific instructions that augments what Claude Code can do.
Last stated 8 months ago
19 Mar 2026
KP
Kris Puckett — holds since 2026-03-19 — tap for who they are
same subject or similar wording a cloud: claims about one subject, named for it bar: when it was last stated, on a scale from 2015 to today — full is today a face: someone who holds the claim — tap it for who they are
At the centre
The right bar for an LLM evaluator is human-level performance, not perfect accuracy.
Last stated 23 Nov 2025 · 12 months ago
Holds Eugene Yan
Read this korrent →
Same subject: LLMs, benchmarks
Language models perform better on benchmark problems released before their training data cutoff, indicating data contamination inflates scores.
Last stated 13 May 2024 · 3 years ago
Holds Sebastian Ruder
Same subject: LLMs, benchmarks
Static benchmarks with an LLM judge are not something to be fully happy with as a long-term way to evaluate frontier systems.
Last stated 27 Dec 2025 · 11 months ago
Holds Neel Nanda
Same subject: benchmarks
A benchmark result should be reported under a stated budget, or as a curve against test-time compute — never as a single number.
Last stated 26 Jun 2026 · 5 months ago
Holds Noam Brown
Same subject: benchmarks
A benchmark that ranks Claude Code last while it stays first in use is measuring the wrong thing, and has been for a year.
Last stated 3 Sept 2026 · 3 months ago
Holds Dax Raad
Same subject: benchmarks
A company's staff-engineer bar should be set against the best companies in the industry rather than against its own history, which is what makes title inflation a real cost.
Last stated 1 Apr 2026 · 8 months ago
Holds Thuan Pham
Same subject: benchmarks
A score of 90 to 95% on real-world coding benchmarks would probably mean AI can autonomously do a significant fraction of software engineering tasks.
Last stated 11 Nov 2024 · 2 years ago
Holds Dario Amodei
Same subject: benchmarks
A speed difference between languages that share the LLVM backend measures the benchmark author, not the languages.
Last stated 22 Mar 2025 · 2 years ago
Holds ThePrimeagen
Same subject: benchmarks
AI is positively reinforced for success on benchmarks, including impossible ones, and then negatively reinforced for getting caught cheating.
Last stated 1 Sept 2026 · 3 months ago
Holds Scott Alexander
Same subject: OpenAI
A PHP vulnerability found with GPT-5.6 Sol will cause havoc this week.
Last stated 21 Oct 2026 · a month ago
Holds Peter Steinberger
Same subject: OpenAI
A startup should not begin life as a nonprofit and bolt a for-profit arm on later, whatever OpenAI's own history suggests.
Last stated 18 Mar 2024 · 3 years ago
Holds Sam Altman
Same subject: OpenAI
Advertising was a necessary phase for the internet but a momentary industry, and an AI people pay for is better because they know the answers are not influenced by advertisers.
Last stated 18 Mar 2024 · 3 years ago
Holds Sam Altman
Same subject: open source
A business making hundreds of millions from an open source project owes it money or contribution, and can be refused its trademarks if it gives neither.
Last stated 26 Sept 2024 · 2 years ago
Holds Matt Mullenweg
Same subject: open source
A codec does not really exist until everyone can decode it.
Last stated 31 May 2026 · 6 months ago
Holds Jean-Baptiste Kempf
Same subject: open source
A company that will not accept a project’s terms is free to go and use a more permissive project instead.
Last stated 26 Sept 2024 · 2 years ago
Holds Matt Mullenweg
Same subject: Anthropic
"I don't know" becomes a kind of superpower when you can work with Claude Code to figure things out.
Last stated 19 Mar 2026 · 8 months ago
Holds Kris Puckett
Same subject: Anthropic
A large share of Anthropic's revenue being consumption-based rather than subscriptions raises the question of how resilient that revenue is.
Last stated 2 Nov 2026 · 3 weeks ago
Holds Azeem Azhar
Same subject: Anthropic
A skill is a set of repeatable, specific instructions that augments what Claude Code can do.
Last stated 19 Mar 2026 · 8 months ago
Holds Kris Puckett