BitDecoding offers a novel approach to overcome memory bottlenecks in long-context LLMs by optimizing low-bit KV cache with tensor cores.

• Enables up to 2.5x faster decoding while maintaining model accuracy
• Achieves significant speedup through GPU tensor core optimizations
• Implements bit manipulations to unlock hardware acceleration for low-bit quantized caches
• Delivers performance gains across different LLM architectures and context lengths

This research advances LLM deployment efficiency by reducing both memory requirements and computational overhead, making long-context models more practical for production environments.

BitDecoding: Unlocking Tensor Cores for Long-Context LLMs Decoding with Low-Bit KV Cache