xKV introduces a novel technique for compressing Key-Value caches in Large Language Models, enabling longer context windows with lower memory requirements.

• Uses Singular Value Decomposition (SVD) to identify and leverage shared patterns across model layers
• Achieves up to 63% reduction in KV-cache memory footprint with minimal performance impact
• Requires no retraining and can be applied to existing pre-trained models
• Works effectively even when token similarity across layers is low

This engineering breakthrough helps make LLMs with long context windows more practical for deployment in memory-constrained environments, potentially enabling more widespread adoption of large context models in production systems.

xKV: Cross-Layer SVD for KV-Cache Compression