Mixture of Experts (MoE) is a neural network architecture that divides the model into multiple specialized sub-networks, called experts. A component called a router (or gating network) decides, for each input, which experts to activate. Instead of passing every token through all the model's parameters, only a subset is used, which is known as sparse activation.
Its importance lies in efficiency: it allows for scaling the total number of parameters without proportionally skyrocketing the computational cost during inference. An MoE model can have hundreds of billions of parameters but activate only a fraction per token, offering great capacity at a reasonable cost.
Some practical nuances to consider:
- Load balancing between experts is critical; if the router always favors the same ones, others remain underutilized.
- It requires more memory, as all experts must be loaded even if they are not used.
Models like Mixtral 8x7B or various versions of GPT employ this approach to combine power and efficiency.