Inference, not training, is now the constraint, and specialized low-energy hardware will beat general-purpose GPUs and TPUs on latency.

Chief Scientist at Google DeepMind and Google Research

Yeah, I mean it's a little different, but I think uh you're going to see more and more uh uh high performance and um low energy uh inference hardware systems because I think everyone is now realizing that inference is the key to making you know these agent-based systems be available to more and more people and that latency is really important and that specialization of the hardware is a really key way you can make uh things that are more energy efficient and lower latency than more general purpose uh computational devices like say GPUs or TPUs