This research introduces a novel adaptive batch size scheduling approach for distributed training of large language models that balances computational efficiency with model performance.

• Resolves the trade-off between large batches (efficient training) and small batches (better generalization)
• Combines data and model parallelism strategies for optimized distributed training
• Demonstrates improved convergence speed and generalization capabilities
• Provides practical guidelines for implementing adaptive scheduling in production environments

For AI engineering teams, this research offers a concrete framework to reduce training costs while maintaining or improving model quality—particularly valuable as language models continue to grow in size and complexity.

Adaptive Batch Size Schedules for Distributed Training of Language Models with Data and Model Parallelism