This research introduces Shared RAG-DCache, a novel approach that drastically improves LLM inference efficiency when handling multiple RAG-enhanced requests simultaneously.

• Reduces memory usage by up to 4.5× through shared disk-based KV cache management
• Decreases inference latency by 16.2% in multi-instance environments
• Maintains high performance even with increased context lengths from RAG
• Enables cost-effective scaling for production LLM services

Engineering teams can implement this solution to overcome memory bottlenecks in RAG deployments without compromising performance, allowing more efficient resource utilization for production LLM services.

Shared Disk KV Cache Management for Efficient Multi-Instance Inference in RAG-Powered LLMs