This research presents a comprehensive analysis of compression techniques for Mixture of Experts (MoE) models, addressing the critical challenge of deploying large language models efficiently.

• Introduces novel compression methods including Layer Drop, Block Drop, and Expert Slimming
• Evaluates compression strategies across various model sizes and configurations
• Demonstrates significant reduction in computational costs while preserving performance
• Provides practical insights for engineering teams implementing MoE architectures

For engineering teams, this research offers valuable guidance on optimizing MoE models for deployment scenarios where computational resources are limited but high performance is still required.

Towards Efficient Mixture of Experts: A Holistic Study of Compression Techniques