This research introduces a novel Mixed-precision Graph Neural Post-Training Quantization approach that significantly improves large language model deployment on resource-constrained devices.

• Achieves superior performance at extremely low bit widths (< 3 bits)
• Leverages a graph neural network to capture weight dependencies during quantization
• Employs an adaptive mixed-precision strategy to balance performance and efficiency
• Demonstrates practical viability for real-world deployment of compressed LLMs

This engineering breakthrough enables the deployment of powerful language models on edge devices without sacrificing critical performance, opening new possibilities for on-device AI applications and reduced infrastructure costs.

Mixed-Precision Graph Neural Quantization for Low Bit Large Language Models