MaZO introduces a memory-efficient optimization technique for fine-tuning large language models across multiple tasks simultaneously without backpropagation.

• Eliminates memory-intensive backpropagation by using zeroth-order optimization
• Introduces masked parameter selection to reduce gradient variance
• Enables multi-task fine-tuning on resource-constrained systems
• Achieves comparable performance to standard methods with significantly less memory

This research breakthrough makes LLM fine-tuning accessible to organizations with limited computational resources, potentially democratizing access to advanced AI capabilities in engineering applications.

MaZO: Masked Zeroth-Order Optimization for Multi-Task Fine-Tuning of Large Language Models