SVDq introduces a breakthrough approach to compress key-value caches in Large Language Models, dramatically reducing memory requirements while maintaining performance.

• Combines Singular Value Decomposition (SVD) with mixed-precision quantization
• Achieves 410x compression of the key cache with minimal quality loss
• Uses only 1.25 bits per value compared to traditional 16-bit storage
• Enables more efficient inference for memory-constrained LLM deployments

This research addresses one of the most significant engineering challenges in LLM deployment: memory bottlenecks during inference. By drastically reducing memory requirements, SVDq makes larger models accessible on resource-limited hardware.

SVDq: 1.25-bit and 410x Key Cache Compression for LLM Attention