Apt-Serve introduces a novel approach to scaling LLM inference systems with significantly improved throughput while meeting latency requirements.

• Implements hybrid caching architecture that balances GPU and CPU resources
• Uses adaptive request scheduling to optimize performance under varying workloads
• Achieves up to 3.65× higher effective throughput compared to existing systems
• Maintains Time To First Token (TTFT) service level objectives even under high demand

This engineering breakthrough addresses critical scalability issues for LLM-serving infrastructure, enabling organizations to efficiently deploy LLM services at scale with better resource utilization.

Apt-Serve: Adaptive Request Scheduling on Hybrid Cache for Scalable LLM Inference Serving