What is a Mixture of Experts (MoE)?

A mixture of experts is a model built as many smaller specialist sections, with only a few of them switched on for any one question. The model can be enormous in total while doing the work of a much smaller one every time it answers. That is how several of the biggest models stay affordable to run.

The point of a mixture of experts is that total size and work per answer stop being the same thing. In an ordinary model every question wakes the entire network, so doubling the model doubles the work done on every single answer. A mixture of experts is instead built as separate sections, the experts, with a small router in front that picks which few to consult for each piece of the input. Ask a coding question and it might wake two sections out of sixty-four, so the answer is computed by a slice of the model rather than all of it. Every section still has to sit in memory waiting to be called, so the saving is in work done per answer, not in what it costs to keep the model available. Training takes extra care too, because a router that keeps picking the same favorites leaves much of the model barely trained. The payoff explains why the design keeps appearing in frontier models: capacity can grow without the work on each answer growing with it.

A mechanic's toolbox holding two hundred tools. Any one repair needs four or five of them, and he reaches straight for those while the rest stay in the drawer, so the job is no heavier than if the box were small. He still bought all two hundred and still hauls the whole box to every call, because the point of owning the set is that whatever rolls in is already covered. Light work on each repair, permanent cost to keep that range on hand.