This research provides a comprehensive analysis of 8-bit floating-point (FP8) computation for LLM inference across different AI accelerator hardware.

• Reveals important differences in FP8 scaling factor methodologies between NVIDIA H100 and Intel Gaudi 2
• Demonstrates how implementation variations impact inference accuracy and performance
• Identifies optimal configurations for different LLM workloads
• Provides engineering guidelines for hardware-specific FP8 implementation

For engineering teams, this research offers critical insights for optimizing LLM deployment across different hardware platforms, enabling more efficient inference without sacrificing model quality.

An Investigation of FP8 Across Accelerators for LLM Inference