ZO2 is a novel framework that enables fine-tuning of extremely large language models (LLMs) with limited GPU memory by eliminating the need to store activations and gradients during training.

• Reduces memory requirements by computing gradients using only forward operations
• Leverages CPU memory to handle parameters that don't fit in GPU memory
• Achieves comparable performance to traditional methods while using significantly less GPU memory
• Enables fine-tuning of models that would otherwise be impossible on standard hardware

This engineering breakthrough democratizes access to large model training, allowing researchers and smaller organizations to work with state-of-the-art LLMs without requiring specialized infrastructure.

ZO2: Scalable Zeroth-Order Fine-Tuning for Extremely Large Language Models with Limited GPU Memory