Generation configurations: temperature, top-k, top-p, and test time compute

Chip Huyen · 16 Jan 2024 · huyenchip.com ↗

3 korrents from this piece

Chip Huyen did not write this page.

Every claim below was made in this piece, quoted word for word and numbered in the order the piece makes them, so you can read it there rather than take our word for it. The sentence above each quote is our reading of the claim, not their wording. Each quote was checked against a stored copy of the page at build time; where the two differ, the quote is the fact.

Our reading
The probabilistic nature of AI models is what makes them well suited to creative tasks

Their wordsThis probabilistic nature makes AI great for creative tasks.
Our reading
When a model behaves unreliably, the better fix is to switch models rather than repeatedly sample it to work around the inconsistency

Their wordsThe optimal thing to do with a fickle model, however, is to swap it out for another.
Our reading
As models get better at following instructions, techniques like finetuning and constrained sampling for structured outputs will become less necessary

Their wordsI suspect that in the future, it'll be easier to get models to output exactly what we need with minimal prompting, and these techniques will become less important.