PQCache introduces a novel memory optimization technique that enables longer context windows for LLMs without requiring expensive hardware upgrades.

• Reduces KVCache memory footprint by 75% with minimal impact on output quality
• Achieves 12.2x speedup on batch processing compared to existing approaches
• Maintains over 97% of the original model quality across benchmarks
• Implements efficient quantization that works across diverse model architectures

This research addresses the critical memory bottleneck that currently limits LLM context lengths, making longer context models more accessible and affordable for production deployments.

PQCache: Product Quantization-based KVCache for Long Context LLM Inference