This research introduces Gradient Multi-Normalization, a novel framework for training large language models without storing optimizer states, dramatically reducing memory requirements while maintaining performance.

• Eliminates memory overhead of traditional adaptive optimizers like Adam
• Achieves comparable or better performance through multi-normalization of gradients
• Enables more efficient scaling of LLM training with reduced computational resources
• Provides a stateless optimization approach that's particularly valuable for resource-constrained environments

This engineering advancement allows organizations to train larger models with existing infrastructure or reduce costs for current model sizes, potentially democratizing access to LLM development capabilities.

Gradient Multi-Normalization for Stateless and Scalable LLM Training