A transformer’s dense feed-forward layer treats every token the same way: each one is shoved through the same pile of weights, whether it is a comma or a rare Sanskrit verb. By 2021 it was clear that this was a strange way to spend compute. The comma did not need the Sanskrit machinery. The Sanskrit verb did not benefit from being averaged into the comma’s path. And yet the dense network paid for both, on every token, on every layer.
The idea of routing — letting each token pick which subnetwork would process it — was older than transformers. Robert Jacobs and Michael Jordan had written about adaptive mixtures of local experts in 1991, when the experts were tiny MLPs and the gating network was a softmax over a handful of options. What changed in 2021, with Google’s Switch Transformer, was scale and surgical placement. Replace the feed-forward block in each transformer layer with a bank of, say, sixty-four expert MLPs. Add a small router — a single linear layer plus a softmax. For each token, send the top-one or top-two experts a copy of that token’s residual stream. Sum the experts’ outputs, weighted by the router’s scores. Discard the rest.
The economics are striking. A model with sixty-four experts per layer has roughly sixty-four times the parameters of a dense model with one such block, but each token still only touches one or two experts. Memory grows; FLOPs barely move. You get a much larger function class for the same forward-pass cost. The trick, in spirit, is hardware arbitrage: GPUs love parallel matrix multiplies, and parameters are cheaper than activations when you have enough HBM. Mixtral, DeepSeek-MoE, and the open-weights wave that followed lived inside this gap.
The pathologies show up in the router. The router is trained end-to-end, but its decisions are discrete in the limit — top-k means most experts see no gradient on most tokens. Two failure modes recur. The first is collapse: a few experts grab most of the traffic, the rest atrophy, and the model behaves like a dense network with extra dead weight. The second is mode-specific routing that the router can’t undo: once an expert specializes in, say, code, the router keeps feeding it code, and any structural shift in the data hits a wall. Practitioners spent years inventing auxiliary losses — load-balancing penalties, importance regularizers, expert capacity caps that drop tokens past a budget — to keep the router honest. None of them are principled. All of them are necessary.
What makes MoE interesting beyond the cost story is what the experts learn. The early reading was that experts would carve up the data by domain: one for math, one for code, one for English prose. Probing studies have found that, mostly, they don’t. Experts cluster on shallower features — token frequency, punctuation, surface morphology — and the work of meaning still happens in the routed mixture, not in any single expert. Dmitry Lepikhin, one of the GShard authors, described it as the router learning a load-balancing protocol more than a taxonomy.
That observation matters because it pushes against a tidy story the field would like to tell. We would like to believe that scale produces modularity, that bigger MoE models will give us interpretable specialists we can read like a CV. The evidence so far is that scale produces something stranger: a population of mostly-similar experts whose differentiation is functional rather than semantic. The router is not picking the right tool. It is picking a sufficient one, fast.
The interesting frontier is hybrid sparsity — MoE in some layers, dense in others, sometimes per-head rather than per-FFN. And the routing question is alive: differentiable approximations like Soft MoE and Expert Choice avoid the discrete top-k and have moved capacity allocation in the right direction.