FlowKV introduces a novel framework that dramatically reduces latency in disaggregated LLM inference by optimizing KV cache transfer between prefill and decode nodes.

• Optimized Data Transfer: Reduces KV cache transfer latency by up to 62% through memory optimization techniques
• Load-Aware Scheduling: Dynamically assigns prefill and decode roles based on node workloads for better resource utilization
• Unified Memory Management: Implements continuous memory allocation to minimize data transfer overhead
• Superior Performance: Achieves 37% lower end-to-end latency compared to existing solutions

This research directly addresses critical engineering challenges in LLM deployment, enabling more efficient and cost-effective AI infrastructure for production environments.

FlowKV: A Disaggregated Inference Framework with Low-Latency KV Cache Transfer and Load-Aware Scheduling