Jenga introduces an innovative memory allocation framework specifically designed to address the challenges of serving modern LLMs efficiently.

• Tackles the increasing heterogeneity in embeddings dimensions and attention patterns in modern LLMs
• Improves upon PagedAttention to deliver more efficient GPU memory utilization
• Enables larger batch sizes for inference, directly reducing operational costs
• Demonstrates significant performance gains through intelligent memory management

For engineering teams, Jenga offers a practical solution to the growing challenge of managing computational resources while scaling LLM services, potentially reducing infrastructure costs without sacrificing performance.

Jenga: Effective Memory Management for Serving LLM with Heterogeneity