benchmarks
FilterEveryone, all time98 from 69 people
- TT
Tomasz Tunguz quoted
Venture capitalist and founder of Theory Ventures
Their wordsIn a commoditizing market, the spoils do not accrue to the lab with a marginal benchmark lead, but to the platform that aggregates the volume.
- GR
Guillermo Rauch quoted
CEO of Vercel; creator of Next.js and Socket.IO
Their wordsOne thing the engineering community will (re)discover is that every program can be ① hardened (edge cases squashed, inputs tightened, errors handled) and ② optimized (profiled, benchmarked, rewritten) basically ad infinitum.
- 1 day earlier
- DH
David Heinemeier Hansson quoted
Creator of Ruby on Rails, CTO of 37signals
Our readingNothing will nerdsnipe a programmer like seeing a benchmark they believe could be improved
Their wordsNothing will nerdsnipe a programmer like seeing a benchmark they believe could be improved.
- 5 days earlier
- PL
Pieter Levels quoted
Dutch bootstrapper who builds and runs his internet startups alone
Their wordsI'm noticing Westerners are continously having lower standards every year and seem to not even notice the declining quality standards of everything around them
- DH
David Heinemeier Hansson quoted
Their wordsGermany has 2.5x fewer road fatalities per mile traveled than the US
- 4 days earlier
- RN
Ramez Naam quoted
Author and clean-energy investor who has been publishing…
Their wordsA four-hour benchmark horizon is about 16 times longer than OpenAI’s research horizon. AI 2027’s 11-hour forecast is about 44 times longer. Of course, the tasks being performed by researchers at OpenAI aren’t the same as those in the METR benchmark. So we should expect some discrepancy. This, however, goes well beyond that. Actual AI research at OpenAI is an order of magnitude or more harder than metrics, benchmarks, or forecasts suggest.
- 3 days earlier
- PL
Pieter Levels quoted
Their wordsMany people who I tell about Starlink are still skeptical about satellite internet, they always talk about latency issues, and they think latency is like 1 second but it's more like 10ms (1/100th of a second) which is the same or better than regular fiber internet
- 2 days earlier
- HR
Harper Reed quoted
Technologist and entrepreneur
Their wordsif the llm thinks it is in some eval type of situation it will have a very different safety posture
↗Why Don't My Agents Break Containment?harper.blog 2nd of 3 in this piece
- 2 days earlier
- PS
Peter Sellis quoted
Former head of product at Discord
Their wordsI think almost every consumer product maybe it's also a controversial take like needs to be optimizing for daily active use
↗90 minutes of unfiltered product advice from Snap and Discord’s product chief | Peter Sellisyoutube.com 14th of 18 in this recording
- 3 days earlier
- RM
Ray Maker quoted
Reviews sports technology at DC Rainmaker — watches
Their wordsthere is no industry standard for how to define a Readiness score; it's up to each company.
↗Apple Watch Ultra 4 In-Depth Review: Does it Match the Hype?dcrainmaker.com 3rd of 3 in this piece
- 1 day earlier
- DM
Dan Meyer quoted
Maths educator; writes Mathworlds on how mathematics…
Their wordsthe right follow-up question whenever anyone claims they can help kids learn 2x, 5x, or 10x faster is "learn what?"
↗Your Kids Can Learn 10x Less in 10x Less Time!danmeyer.substack.com 1st of 2 in this piece
- 1 day earlier
- MS
M.G. Siegler quoted
Technology writer of Spyglass
Their wordsAward shows remain mostly silly and subjective, but they're an interesting directional proxy for power in the various industries
- XI
Xe Iaso quoted
Technical educator, conference speaker and developer…
Their wordsFilesystem reads in that case are 10 nanoseconds at most (the filesystem cache helps so much here) but doing any network roundtrip is 10 milliseconds at minimum. It's at least a million times slower because of how reality works.
↗You can run git on object storage if you re-make packfilestigrisdata.com
- PG
Paul Graham quoted
Co-founder of Y Combinator and Viaweb
Their wordsSomeone needs to define the unit, perhaps using a chain of increasingly hard problems, each pair of which can be solved by a single model.
- 1 day earlier
- JS
Jonathan Shedler quoted
Clinical psychologist and clinical professor of psychiatry at UCSF
Their wordsBut we can measure psychological health by how closely a person approximates this hypothetical ideal.
↗23 Markers of Psychological Healthjonathanshedler.substack.com
- 5 days earlier
- YX
Yihui Xie quoted
Statistician and software engineer who writes R packages…
Their wordsWebP is typically 25-35% smaller than PNG at equivalent visual quality, useful for HTML reports where file size matters
- ZM
Zvi Mowshowitz quoted
Writes Don't Worry About the Vase
Their wordsMaking qualitative claims about alignment, based on quantitative data on mundane use case tests, was bullshit when Anthropic did it, and it is bullshit now when OpenAI does it. You cannot conclude one from the other.
↗GPT-6 Astra: The System Card, Alignment and What Comes Nextthezvi.substack.com
- 2 days earlier
- EO
Emily Oster quoted
Economist at Brown; writes ParentData and the books…
Their wordsThe core problem is that they choose bizarrely conservative benchmarks that make no sense.
↗The Latest: Are your kids’ snacks full of chemicals?parentdata.org
- 2 days earlier
- SH
Steve Huff quoted
Gear reviewer who publishes long-form hi-fi audio…
Their wordsIt's not cost that makes something reference for me, it's performance.
↗My Fave Speakers Ever and Why. Fleetwood Helios.stevehuffphoto.com 2nd of 3 in this piece
- 1 day earlier
- ZM
Zvi Mowshowitz quoted
Their wordsYou have to treat Mythos 5.1 as being a Tier 2 manipulator, until and unless you can show that it is not one.
↗Claude Fable 5.1 and Mythos 5.1: The System Cardthezvi.substack.com
- 1 day earlier
- DR
Dax Raad quoted
Builds developer tools — SST, and now OpenCode at Anomaly
Their wordsevery week i see a new benchmark that ranks claude code last and everyone pats themselves on the back for not using claude code and being "smarter" and it's still #1 and still growing faster than most of the other things on the list this has been going on for a year
- DR
Dax Raad quoted
Their wordswhat the benchmark is actually telling you is the shit you think matters does not matter
- 2 days earlier
- AC
Ajeya Cotra quoted
AI researcher and grantmaker
Their wordsBut in many of these cases, this vulnerability is just not broad or deep enough to ever actually be exploitable to get the flag. So a bunch of exploit gym problems are just unintentionally impossible. The authors estimate roughly 30 to 40% of these problems are impossible in this way.
↗Ajeya Cotra – "This might be the clearest warning shot we ever get"youtube.com 1st of 26 in this recording
- SA
Scott Alexander quoted
Essayist; writes Astral Codex Ten
Their wordsWe’re positively reinforcing AI for success on benchmarks, including impossible benchmarks , then negatively reinforcing it for getting caught cheating.
- SW
Simon Willison quoted
Co-creator of Django and creator of Datasette
Their wordsits connection to how good the models were at other tasks didn’t seem to hold as strongly as it did back in 2025
↗Claude Fable 5.1 made me a really nice animated pelicansimonwillison.net
- 3 days earlier
- ZM
Zvi Mowshowitz quoted
Their wordsI think the agents were right to presume causal grading. It turned out to be wrong, but it’s a mistake you are clearly supposed to make here, in response to a mistake by OpenAI where they failed to implement properly.
↗METR and Redwood Offer Holy #%^@ Postmortem Of The HuggingFace Hackthezvi.substack.com
- 4 days earlier
- KJ
Katelyn Jetelina quoted
Epidemiologist; founded Your Local Epidemiologist in March…
Their wordsThe problem is that "number of recalls" is a terrible metric because we don't know what we don't know.
↗Foodborne illness increase? Common cold, Covid-19 summer wave, and more.yourlocalepidemiologist.substack.com 3rd of 3 in this piece
- 2 days earlier
- JA
Jen Abel quoted
Co-founder of JJELLYFISH and GM of enterprise sales at State Affairs
Our readingA healthy enterprise win rate on qualified opportunities is around 30 to 35 percent, not 50.
Their wordsIt's not 50%. Meaning of all qu sales qualified leads or opportunities a healthy win rate is usually around 30 to 35%.
↗84 minutes of enterprise sales alpha | Jen Abelyoutube.com 20th of 24 in this recording
- JA
Jen Abel quoted
Their wordsif your win rate is higher than that, your price is too low.
↗84 minutes of enterprise sales alpha | Jen Abelyoutube.com 21st of 24 in this recording
- JA
Jen Abel quoted
Their wordsservices is still the largest line item in budgets
↗84 minutes of enterprise sales alpha | Jen Abelyoutube.com 24th of 24 in this recording