Hamel Husain
Machine learning engineer and independent AI consultant; writes about LLM evaluation and tooling at hamel.dev.
Hamel Husain did not write this page.
We collected these quotes from things they published elsewhere, and every quote links to where it was said. They have no account here and have not endorsed this site. Quotes are word for word; the short line under each one is our own restatement, not their wording. Their own site. Is this you? Claim it or ask us to remove it. Or tell us what is wrong here.
Filter All topics · Newest first
-
Our reading
Evaluation and error analysis will remain necessary practices even if AI models become near-perfect.
Their wordsEven with perfect models, you still need to verify they're solving the right problem. The need for systematic error analysis, domain-specific testing, and monitoring will still be important.
↗AI Evals: Everything You Need to Knowhamel.dev 1st of 2 in this piece
-
Our reading
People cannot fully specify what they want from an AI system without first observing how it behaves.
Their wordsAdditionally, a LLM cannot read your mind, and research shows that people need to observe the LLM's behavior in order to properly externalize their requirements.
↗AI Evals: Everything You Need to Knowhamel.dev 2nd of 2 in this piece
- 3 months earlier
-
Our reading
An artifact that is hard for its builder to verify is usually also hard for its users to verify.
Their wordsArtifacts that are hard for you to verify are often hard for users too.
↗“It’s Hard to Eval” Is a Product Smellhamel.dev 1st of 3 in this piece
-
Their wordsMore importantly, designing your product for ease of verification should come before building evals.
↗“It’s Hard to Eval” Is a Product Smellhamel.dev 2nd of 3 in this piece
-
Their wordsBefore AI, verification often happened incidentally during the process of creating work product. With AI, verification is the bottleneck.
↗“It’s Hard to Eval” Is a Product Smellhamel.dev 3rd of 3 in this piece
- 5 months earlier
-
Their wordsNotebooks don't force you to write clean code. AI tools don't force you to think carefully. Discipline comes from the developer, not the environment.
-
Our reading
Good documentation is produced by effort, not by tooling that keeps code and docs together
Their wordsThis reinforced my belief that good documentation comes from effort, not tooling.