LLMs 12
- Spending most of the compute memorising the world in pre-training is brute force; what we want is a model that reasons the way a student sits an open-book exam.
- Every announcement of a longer context window hides the cost, which is that the model gets worse at following instructions.