This research introduces specialized algorithms for optimizing large language model inference when facing memory constraints from Key-Value (KV) caches.

• Addresses the critical challenge of managing KV cache memory during LLM text generation
• Proposes innovative batching techniques to improve throughput while maintaining performance
• Develops a theoretical framework for modeling LLM inference under memory constraints
• Offers practical solutions for reducing latency and improving resource utilization

For engineering teams, these advancements enable more efficient deployment of LLMs in production environments with limited computational resources, allowing organizations to scale AI capabilities with existing infrastructure.

Online Scheduling for LLM Inference with KV Cache Constraints