This research systematically redesigns LLM optimizers by leveraging Fisher Information Matrix approximations, achieving both memory efficiency and faster convergence.

• Introduces a unified framework showing how various state-of-the-art optimizers can be understood through structured FIM approximation
• Develops two novel algorithms (RACS and Alice) with low-rank structural assumptions
• Achieves comparable convergence to Adam while requiring significantly less memory
• Provides mathematical foundations for future optimizer designs with provable guarantees

This engineering breakthrough directly addresses the crucial challenge of efficiently training massive language models with limited computational resources, making advanced AI development more accessible and cost-effective.

Towards Efficient Optimizer Design for LLM via Structured Fisher Approximation with a Low-Rank Extension