CyberFreezeDevRouter
A small layer scores every expert per token and picks the top few.
Read more about Router
A small layer scores every expert for the current token and selects the top few, often two.
- Home
- AI glossary
- Mixture of experts
What is mixture of experts (MoE)?
Many expert sub-networks in one model, with only a few running for each token.
Definition
A router sends each token to a few experts while the rest sit idle, so these models list total and active parameters.
Read more about Definition
In a standard ("dense") large language model, every parameter takes part in processing every token. In a mixture of experts (MoE) model, some layers are split into many parallel sub-networks called experts, and a small router (or gating network) picks a few experts for each token. The other experts sit idle for that token.
So MoE models are described with two numbers: total parameters (everything stored) and active parameters (what runs per token). Mistral's Mixtral 8x7B, for example, has eight experts per layer, routes each token to two of them, and has roughly 47 billion total but about 13 billion active parameters, according to Mistral's announcement.
How it works
Experts
Usually feed-forward blocks whose learned specialities rarely match human topics.
Read more about Experts
Each expert is usually a feed-forward block. Experts do not neatly map to human topics like "maths" or "law"; their specialisation is learned and often hard to interpret.
Load balancing
Training stops the router from sending everything to a few favourite experts.
Read more about Load balancing
Training includes measures to stop the router from sending everything to a few favourite experts.
Why it matters for business deployments
Speed
Compute follows active parameters, so it can run faster than a dense model of equal size.
Read more about Speed
Compute per token tracks active parameters, so an MoE model can generate faster than a dense model of the same total size.
Memory
Every expert must be loaded, so VRAM needs follow total parameters.
Read more about Memory
All experts must still be loaded, so VRAM needs track total parameters. A model with a small active count can still need a large GPU or several.
Common in large open models
DeepSeek-V3, some Qwen3 models and OpenAI's gpt-oss use MoE designs.
Read more about Common in large open models
Several open-weight models, including DeepSeek-V3, some Qwen3 models and OpenAI's gpt-oss, use MoE designs.
A practical example
An MoE model with 3 billion active parameters generates faster than a dense 14 billion model but needs roughly twice the memory to load.
Read more about A practical example
A team choosing a model for an on-premise server compares a dense 14 billion parameter model with an MoE model of 30 billion total and 3 billion active parameters. The MoE model generates text faster, because only 3 billion parameters run per token, but needs roughly twice the memory to load. If the server has the memory, MoE wins on speed; if memory is tight, the dense model is the only one that fits.
Common mistakes
Sizing hardware from active parameters
Active parameters describe compute, not memory. Plan memory from total parameters.
Read more about Sizing hardware from active parameters
Active parameters describe compute, not memory. Use total parameters for memory planning.
Assuming it is cheaper everywhere
With many users, tokens hit different experts; efficient serving needs a suitable engine.
Read more about Assuming it is cheaper everywhere
With many concurrent users, different tokens hit different experts, and serving MoE efficiently needs a suitable inference engine.
Picking it for a tiny device
On small edge hardware, a compact dense or distilled model usually fits better.
Read more about Picking it for a tiny device
On small edge hardware a compact dense model, or a distilled one, is usually the better fit.
Related terms and pages
Questions
Is a mixture of experts model the same as a multi-agent system?
No. MoE happens inside one model, token by token, and the experts are not separate assistants. A multi-agent system is several separate AI agents, each with its own instructions, working together.
Are MoE models better than dense models?
They trade memory for speed. For the same compute budget per token they can hold more knowledge, but they need more memory. Which is better depends on your hardware and workload.
What does a name like 30B-A3B mean?
It usually means 30 billion total parameters with about 3 billion active per token. Check the model card to be sure.
Last updated