QUAD introduces a novel framework that enables efficient 4-bit quantization of medium-sized LLMs by addressing the activation outlier problem.

• Uses Singular Value Decomposition (SVD) to suppress activation outliers
• Achieves effective quantization for medium-sized models (e.g., Llama-3-8B)
• Maintains model performance while significantly reducing computational costs
• Combines quantization with parameter-efficient tuning for optimal results

This research matters for engineering teams deploying LLMs in resource-constrained environments, enabling wider adoption of powerful language models without sacrificing performance.

QUAD: Quantization and Parameter-Efficient Tuning of LLM with Activation Decomposition