ATTNChecker is a novel fault tolerance technique that stabilizes large language model training by identifying and handling computational failures in the attention mechanism with minimal overhead.

• Systematically addresses INF, NaN, and near-INF values that can derail training
• Achieves near-zero overhead (only 0.02-0.15%) while providing robust fault protection
• Enables reliable training without performance degradation even in fault-prone environments
• Successfully tested with practical LLM architectures including GPT and LLaMA models

This engineering breakthrough matters because it significantly reduces training failures and resource waste in expensive LLM development, making the process more efficient and cost-effective.

ATTNChecker: Highly-Optimized Fault Tolerant Attention for Large Language Model Training