This research introduces Evaluator Head Prompt Compression (EHPC), a novel method to significantly reduce computational costs when processing long texts in large language models.

• Identifies specific evaluator heads within transformers that can effectively select the most important tokens
• Achieves training-free compression while maintaining key information and model performance
• Reduces computational resources needed for processing lengthy inputs
• Demonstrates practical applications for improving efficiency in commercial API contexts

This engineering breakthrough enables more efficient deployment of LLMs in long-context scenarios, potentially reducing costs and improving performance for real-world applications.

Efficient Prompt Compression with Evaluator Heads for Long-Context Transformer Inference