BaKlaVa introduces a novel method for optimally allocating memory to Key-Value caches in LLMs, significantly reducing GPU memory requirements while maintaining performance for long-context inference.

• Addresses the critical problem of linear memory growth with context length
• Allocates memory selectively across different attention heads instead of uniformly
• Achieves up to 34% reduction in KV-cache memory requirements
• Maintains model quality while enabling longer context processing

This optimization is particularly valuable for engineering applications requiring efficient deployment of LLMs with long contexts, from chatbots to document processing systems, where memory constraints often limit practical applications.

BaKlaVa -- Budgeted Allocation of KV cache for Long-context Inference