foundations · 8 minute lesson

Mixture of experts

Moonshot's Kimi K3 carries 2.8 trillion parameters, but any single word you feed it wakes up only about 104 billion of them. That trick has a name, and it explains why model sizes and running costs have stopped rising together.

The concept classroom is free to join. Sign in with an email link to read up to three lessons every month.