QLM introduces an intelligent queue management system that efficiently handles both interactive and batch LLM requests, improving overall resource utilization while meeting service level objectives.

• Addresses the challenge of poor multiplexing when serving both time-sensitive interactive requests and relaxed-SLO batch requests
• Optimizes resource allocation through dynamic queue management
• Improves system throughput without compromising latency requirements for interactive workloads
• Enables more efficient cloud infrastructure utilization for LLM serving systems

This research is particularly valuable for engineering teams building production LLM infrastructure, as it provides a practical approach to maximize hardware resources while maintaining service quality guarantees.

Queue management for SLO-oriented large language model serving