This research addresses the critical challenge of efficiently managing hardware resources when serving Large Language Models at scale.

• Dynamic priority adjustment based on request progress delivers 1.2-3.8x improvement in user experience
• Strategic preemption policies enhance throughput while maintaining fairness
• Hybrid scheduling approaches outperform traditional FIFO or round-robin methods
• Practical implementation demonstrated with real-world workloads

For engineering teams, these techniques provide immediately applicable solutions to increase throughput and response times when deploying LLMs in production environments, helping maximize return on expensive GPU infrastructure.

Is the GPU Half-Empty or Half-Full? Practical Scheduling Techniques for LLMs