MeanCache is a novel semantic caching solution that significantly reduces computational costs for Large Language Model web services while preserving user privacy.

• Addresses the 31% of repeated queries that burden LLM systems
• Uses semantic matching to identify similar queries, not just exact matches
• Implements federated architecture to maintain user privacy while optimizing performance
• Achieves substantial cost savings for LLM inference without compromising quality

This innovation matters for engineering teams deploying LLMs at scale, offering a practical path to reduce infrastructure costs while maintaining service quality and respecting user privacy concerns.

MeanCache: User-Centric Semantic Caching for LLM Web Services