Echo is a novel scheduling system that intelligently manages both interactive (online) and batch (offline) LLM tasks on the same infrastructure, significantly improving resource utilization without sacrificing performance.

• Addresses the common challenge of over-provisioned resources for handling bursty online traffic
• Introduces a preemption mechanism that efficiently switches between task types as demand fluctuates
• Leverages KV cache management to optimize memory allocation during task switching
• Demonstrates improved throughput and latency compared to traditional scheduling approaches

This research matters for Engineering teams because it offers a practical solution to reduce LLM serving costs while maintaining performance SLAs, potentially transforming how organizations deploy and scale their language model infrastructure.

Echo: Efficient Co-Scheduling of Hybrid Online-Offline Tasks for Large Language Model Serving