AxoNN is a novel open-source framework that makes training large language models more accessible through advanced parallelization techniques and performance optimizations.

• Implements a four-dimensional hybrid parallel algorithm for highly efficient training
• Enables training of models with hundreds of billions to trillions of parameters
• Improves matrix multiplication performance and overlaps communication with computation
• Addresses critical security concerns including privacy risks, copyright issues, and prevention of catastrophic memorization of sensitive data

This research democratizes AI by providing the tools needed for organizations to develop their own LLMs with greater control over security, privacy, and data governance rather than relying solely on proprietary models.

Democratizing AI: Open-source Scalable LLM Training on GPU-based Supercomputers