This research introduces TeZO, a novel approach that significantly improves the efficiency of large language model fine-tuning by exploiting low-rankness on the temporal dimension.

• Expands beyond traditional per-gradient low-rank methods to capture shared properties across all gradients during training
• Achieves superior parameter efficiency while maintaining high performance
• Reduces memory consumption through innovative tensor decomposition techniques
• Demonstrates practical application for resource-constrained LLM fine-tuning scenarios

For engineering teams, TeZO offers a practical solution to optimize LLM deployment in memory-limited environments, potentially enabling more efficient model adaptation on edge devices and in production systems.

TeZO: Empowering the Low-Rankness on the Temporal Dimension in the Zeroth-Order Optimization for Fine-tuning LLMs