This research introduces a novel perspective for identifying critical entries in the KV cache of large language models, addressing a major bottleneck in inference efficiency.

• Develops a formal framework based on output perturbation to identify which KV cache entries most impact output quality
• Provides theoretical grounding for KV cache pruning strategies that previous approaches lacked
• Demonstrates significant memory optimization while maintaining output quality
• Enables more efficient deployment of LLMs in resource-constrained environments

This engineering advancement tackles a fundamental challenge in LLM deployment, potentially making these powerful models more accessible and cost-effective for real-world applications.

Identify Critical KV Cache in LLM Inference from an Output Perturbation Perspective