This research provides the most extensive evaluation to date of quantization methods for language models ranging from 1B to 405B parameters.

• Smaller models (1-8B) are more resilient to aggressive quantization than larger ones
• Performance impact varies dramatically by task difficulty and domain
• Specialized quantization methods (AWQ, GPTQ) consistently outperform standard techniques
• Model performance on complex reasoning tasks degrades more rapidly under quantization

For engineers and ML practitioners, this research delivers practical insights for deploying efficient LLMs across different hardware constraints while maintaining performance on targeted tasks.

Exploring the Trade-Offs: Quantization Methods, Task Difficulty, and Model Size in Large Language Models From Edge to Giant