Mixture of a Million Experts

The paper titled "Mixture of A Million Experts" by Xu Owen He addresses the computational challenges associated with feedforward layers in standard transformer architectures, which increase linearly with the width of hidden layers. To mitigate this issue, the author introduces a sparse mixture-of-experts (MoE) architecture that separates model size from computational cost. The study highlights the benefits of a fine-grained MoE scaling law, which indicates that increased granularity enhances performance. However, existing MoE models are constrained by a limited number of experts due to computational and optimization difficulties. The proposed solution, named PEER (parameter efficient expert retrieval), employs a product key technique for efficient retrieval from a large pool of over a million small experts. Experimental results in language modeling tasks show that PEER layers surpass traditional dense feedforward layers and coarse-grained MoEs in terms of performance-compute trade-off. This advancement allows for the effective use of a vast number of experts, paving the way for further scaling of transformer models while ensuring computational efficiency. The findings suggest that PEER could significantly enhance the capabilities of machine learning models by leveraging a larger expert pool without incurring prohibitive computational costs.