The name mixture of experts sounds like a team of AIs. You might picture one expert for maths, another for writing and a third for science, with a manager deciding which one should answer your question.
That is not how a typical mixture-of-experts language model works. It is one model. Its “experts” are internal neural-network blocks: groups of learned numbers that transform the information flowing through the model. They do not read your whole prompt, form opinions or produce separate answers.
What happens inside one layer
A language model processes text as tokens, which are small pieces of text. As a token moves through the model, its current representation passes through many layers. In an MoE layer, a small learned component called a router scores the available expert blocks and selects a few of them.
Here is a simplified example for the token water at one Mixtral-style MoE layer. The choice of experts 2 and 6 is hypothetical; it shows the path, not a measured routing decision.
The router does not ask an expert to answer. Each selected block performs a mathematical transformation on the token’s current representation. The router gives the selected outputs weights, the model combines them, and the updated representation continues through the network. Shared parts of the model, including attention, still operate outside these expert blocks.
Four tempting pictures—and what is really happening
The word expert invites several mental pictures. This table replaces each one with the mechanism inside the model.
| Tempting picture | What happens in a token-routed MoE |
|---|---|
| Several AIs discuss the answer | One model runs selected internal blocks. |
| Each expert owns a named subject | Specialisation is learned and may not match human subjects. |
| The router sends away the whole question | It makes routing choices for tokens inside each MoE layer. |
| One expert writes the final response | Selected outputs are combined; generation continues through the model. |
Do the experts specialise?
They can learn different patterns because different tokens reach them during training. But human labels such as “coding expert” can be misleading. In the Mixtral paper’s routing analysis, the authors found no obvious assignment by topic. At some layers, routing appeared more connected to syntax and token type.
This also means the expert used for one token does not have to be used for the next. The word water in the diagram could take a different route in another sentence because the router sees its current representation, not just the printed word.
Why build a model this way?
Sparse routing lets a model contain more learned capacity without running every expert for every token. The inactive expert blocks still belong to the model, but their calculations are skipped for that token. This is called conditional computation.
You meet the idea in models such as Mixtral 8x7B. Its “8” refers to the available experts in each MoE layer, while two are selected per token. The useful takeaway is simpler than the name: mixture of experts means one model choosing among its own internal blocks as it processes text.