PII-Bench introduces a novel framework for evaluating how well systems protect personal information in LLM prompts.

• First comprehensive evaluation framework with 2,842 test samples across 55 PII categories
• Proposes innovative query-unrelated PII masking strategy
• Evaluates both effectiveness of PII protection and preservation of query intent
• Reveals significant gaps in current privacy protection approaches

This research is crucial for organizations deploying LLMs in security-sensitive environments, providing a standardized way to assess privacy risks and protection mechanisms before deployment.

PII-Bench: Evaluating Query-Aware Privacy Protection Systems