korrents

korrents · piece

Product Evals in Three Simple Steps

Eugene Yan · 23 Nov 2025 · eugeneyan.com

3 korrents from this piece

Eugene Yan did not write this page.

Every claim below was made in this piece, quoted word for word and numbered in the order the piece makes them, so you can read it there rather than take our word for it. The sentence above each quote is our reading of the claim, not their wording. Each quote was checked against a stored copy of the page at build time; where the two differ, the quote is the fact.

  1. The benchmark is human performance, not perfection. We sometimes get requirements for 90%+ accuracy.
  2. And human annotators can miss as many as 50% of the defects due to fatigue after looking at hundreds of samples.
  3. In my opinion, the true benefit isn't higher accuracy than human annotators-it's scalability.