How less is becoming the new more in AI: Sparsity driving the evolution of LLMs & compute

Once upon a time, in a magical world of computers, there were these amazing things called Large Language Models, or LLMs for short. They were like wizards of words, helping us understand and talk with computers better.

For many years, these LLMs loved to get more and more complex, like building big castles with lots of fancy decorations. And it was wonderful because they could do incredible things. But then, some clever researchers discovered that sometimes, having fewer things in the castle could make it even more powerful. This is sparsity.

Imagine you have a treasure box, and instead of filling it with many tiny toys, you put a few special magical toys inside. These special toys could do incredible tricks, and they didn’t need a lot of space. That’s what these sparse models are like — they have fewer parts, but they can do amazing things.

To make these LLMs even better, smart people started teaching them to be more organized and remember things more efficiently. It’s like they gave the LLMs bigger backpacks to carry more words, so they could understand even longer stories.

But here’s the tricky part: as they made the LLMs smarter and gave them bigger backpacks, they also had to make sure they didn’t become too heavy for the computers to carry. It’s like trying to carry a huge pile of books home from the library — it can be hard if it’s too heavy. Little did they know that there…