Every announcement of a longer context window hides the cost, which is that the model gets worse at following instructions.

So most people, when they advertise new context window increase, they talk a lot about finding the needle in the haystack sort of evaluation metrics and less about whether there’s any degradation in the instruction following performance. So I think that’s where you need to make sure that throwing more information at a model doesn’t actually make it more confused.