LLMs 5
- The main advantage of an LLM evaluator over human annotators is scalability, not higher accuracy.
- The right bar for an LLM evaluator is human-level performance, not perfect accuracy.
Eugene Yan
Member of technical staff at Anthropic. He has led ML/AI teams at Amazon, Alibaba and Lazada, and writes about LLMs, recommender systems and engineering at eugeneyan.com.
Eugene Yan did not write this page.
We collected these quotes from things they published elsewhere, and every quote links to where it was said. They have no account here and have not endorsed this site. Quotes are word for word; the short line under each one is our own restatement, not their wording. Their own site. Is this you? Claim it or ask us to remove it. Or tell us what is wrong here.
What Eugene Yan says most about
AI agents 4
- The surrounding system framework and scaffolding matter more for AI agent capability than the underlying model.
- Delegating a task to an AI agent only works if the output can be verified, which requires defining success criteria first.
AI alignment 3
- Delegating a task to an AI agent only works if the output can be verified, which requires defining success criteria first.
- Traditional recommender systems predict user behavior well but cannot be steered via natural language or explain their own reasoning.
benchmarks 2
- Long-context evaluation benchmarks are likely contaminated by model training data, making them unreliable as a sole evaluation method
- The right bar for an LLM evaluator is human-level performance, not perfect accuracy.