Note (Must Read) Scaled up idea of MoEs for LLMs, scaling the idea to a 137B LSTM for translation. This introduced sparsity and very fast inference at high scale. They faced challenges due to training stabilities and communication costs. Google

Note (Optional) This is among first sparse NN papers University of Washington

Note (Optional) Not really about MoEs, but discussed quite a bit about scaling to 1 trillion parameters and improving training speed with existing hardware. Microsoft

Note (Optional) This paper explores using MoE mechanism for multi-head attention. University of Washington

Note (Must Read) Introduces Sparsely-Gated MoEs for transformers. This is among the first uses of MoEs in transformers, going beyond 600 billion parameters and with quite interesting hardware discussion. Google

Note (Must Read) The first open-source MoE with a 1.6 trillion parameter of 2048 experts released. It does a bunch of changes, such as simplifying the sparse routing, reducing to a single expert, and more. They also perform extensive studies and share interesting insights. Google

google/switch-c-2048

Updated Jan 11, 2024 • 96 • 295

Note This is one of the models released for Switch Transformers.

Note (Optional) Distributed MoE training system based on PyTorch. Most previous implementations were dependant on TPUs and Mesh TensorFlow, so the authors aimed to decouple. Code is open https://github.com/laekov/fastmoe Tsinghua

Note (Optional) A routing formulation formulated as a linear assignment problem leads to optimal assignment so all experts receive the same number of tokens. It removes the need of additional hyperparameters or auxiliary losses. FAIR

Note (Optional) MoEs in the speech domain. Not much to use from here, but quite interesting to see this usage! Tencent

Note (Optional) This introduces a new gate mechanism for the MoEs that is differentiable and hence can be nicely trained with the rest of the network + explicitly control the number of experts to select. Google, MIT

Note (Optional) This paper explores Vision MoE, which is really a sparse version of ViT. It ends up requiring half the compute for the same quality. This introduces batch prioritized routing which was later used for LLMs. Google

Note (Optional) Different routing strategy based on hashing the input tokens and requires no additional parameters, which makes it robust. FAIR

Note (Optional) DEMix conditions LMs on the domain of the input. A DEMix layer focuses on having domain-specific experts, which makes LMs modular - experts can easily be mixed, added or removed. University of Washington, Allen AI, FAIR

Note (Optional) PAQ8s are similar to mixture of experts University of British Columbia

Note (Optional) A detailed empirical study comparing MoEs with dense models in a bunch of settings (zero vs few-shot, etc) Meta

Note (Optional) This paper studies the behaviors of routing networks and derive scaling laws. Google

Note (Must Read) This is an excellent guide on all the issues around MoEs, how to face the training instabilities and more. It discussed fine-tuning issues, hyperparameters, and trade-offs. It also dives into what the experts learn. Google

Note (Optional) This paper introduces a new routing technique. Rather than have the tokens select the top-k experts, the experts select the top-k tokens, switching things around. Hence, a single token might be routed to the number of appropriate tokens. Google

Note (Optional) Being the first multimodal MoE, this paper explores how to handle different types of tokens. Google

Note (Optional) This paper looks at experts as domain-experts as well. This allows to easily train parameters for new domain and merge them back into the model. University of Washington, Allen AI, Meta

Note (Optional) This is more of a survey of sparse expert models. Nice to get an overview of things up to 2022. Google

Note (Optional) Routing techniques using optimal transport. University of Basel, Google

Note (Optional) While most MoEs focus on applying sparsity for the feed forward layers, MoA explores MoE mechanism for multi-head attention Beihang University, Mila, Tencent

Note (Optional) The authors reformulate MoEs computation in terms of block-sparse computations, which lead to 40% end-to-end speedup vs state-of-the-art. This framework was open-sourced and used for many follow-up works. Their proposal never drops tokens and maps efficiently to hardware. https://github.com/stanford-futuredata/megablocks Stanford, Microsoft, Google

Note (Optional) Sparse upcycling means reusing a dense checkpoint to initialize a sparse MoE. Google

Note (Optional) This is the first paper that does instruct-tuning with MoEs. Previous MoE fine-tunes did not have great quality. Surprisingly, it turned out that MoEs can do quite well for Instruct tuning. Google, Berkeley, MIT, University of Massachusetts Amherst, UT Austin

Note (Optional) Soft MoEs introduce a fully-differentiable sparse Transformer architecture. Rather than having hard assignment between tokens and experts, they insert a soft assignment by mixing tokens by computing weighted averages of all tokens. Google

Note (Optional) σ-MoE uses a bunch of changes, such as sigmoid rather than softmax, a special initialization, and regularization. Swiss AI Lab, Harvard, KAUST

Note (Optional) Sub-1-Bit Compression. Enough said. ISTA

Note (Optional) SwitchHead reduces compute and memory requirements while maintaining performance with the same parameter budget. Swiss AI Lab, KAUST, Harvard

Note (Optional) MoCLE is a MoE architecture to activate task-customized parameters based on the instruction clusters. It uses an additional universal expert to improve generalization. Southern University of Science and Technology, Hong Kong University of Science and Technology, Huawei Noah's Ark Lab, Peng Cheng Lab.

Note (Optional) The authors use parameter offloading with a MoE-specific strategy to be able to run Mixtral-8x7B with mixed quantization on free Google Colab Yandex

Note (Optional) The paper is a great example of a MoE which is open-sourced with great license. The paper does not provide many new things in terms of MoE understanding, but still nice to read given it's open source! Mistral

Note (Optional) Combine state space models like Mamba with MoE. IDEAS NCBR, Polish Academy of Sciences, University of Warsaw.

Note (Optional) This paper introduces architectural changes such as expert segmentation for specialization as well as expert isolation and sharing. DeepSeek-AI, Peking University, Tsinghua University, Nanjing University