benchmarks 8
- Every lab knows the benchmark grid is the wrong way to present a model, and publishes it anyway because everybody else does.
- Evaluating a model properly would mean delaying its release, and competitive pressure means no lab will.
Noam Brown
Research scientist at OpenAI, who built the poker AIs Libratus and Pluribus and helped pioneer test-time compute and reasoning in language models.
Noam Brown did not write this page.
We collected these quotes from things they published elsewhere, and every quote links to where it was said. They have no account here and have not endorsed this site. Quotes are word for word; the short line under each one is our own restatement, not their wording. Their own site. Is this you? Claim it or ask us to remove it. Or tell us what is wrong here.
What Noam Brown says most about
scaling laws 8
- Running a model five times and keeping the best answer buys a higher benchmark score without buying a better model.
- Running a model until its performance plateaus is no longer a usable evaluation rule, because a well-scaffolded model keeps improving for weeks.
LLMs 6
- Running a model five times and keeping the best answer buys a higher benchmark score without buying a better model.
- On everyday high-stakes questions, a model's answer can now be trusted more than an expert human's.