This research introduces a unified management system for handling both fast (latency-sensitive) and slow (throughput-oriented) LLM inference workloads in cloud environments.

• Adaptive control mechanisms that dynamically allocate resources based on workload characteristics
• Cost-effective resource utilization by intelligently managing mixed workloads rather than siloing them
• Improved SLA compliance while maximizing overall system throughput
• Practical cloud architecture designed to scale across multiple hardware configurations and geographic regions

For engineering teams, this approach offers a pathway to significantly reduce infrastructure costs while maintaining performance guarantees for critical applications—potentially transforming how cloud providers optimize their LLM inference services.

Serving Models, Fast and Slow: Optimizing Heterogeneous LLM Inferencing Workloads at Scale