fMoE introduces a fine-grained expert offloading system that significantly improves memory efficiency for large Mixture-of-Experts (MoE) language models in production environments.

• Achieves 4.6-7.3× memory reduction with minimal latency impact
• Implements predictive prefetching to optimize expert loading times
• Uses fine-grained token batching to maximize GPU utilization
• Enables serving of much larger models on limited hardware

This research addresses a critical engineering challenge in deploying massive AI models cost-effectively, allowing organizations to deploy more powerful language models with existing infrastructure.

fMoE: Fine-Grained Expert Offloading for Large Mixture-of-Experts Serving