HACK introduces a novel compression technique for the Key-Value cache in disaggregated LLM inference, dramatically reducing data transfer bottlenecks and accelerating processing times.

• Addresses the critical challenge of KV data transmission between prefill and decode stages
• Employs homomorphic compression to operate directly on compressed data without decompression
• Achieves significant reductions in both network transfer time and compute overhead
• Optimizes resource utilization while maintaining inference quality

This engineering breakthrough matters because it enables more efficient scaling of LLM systems in production environments, potentially reducing costs and latency for AI applications that rely on large language models.

HACK: Homomorphic Acceleration via Compression of the Key-Value Cache for Disaggregated LLM Inference