PII-Bench introduces the first comprehensive evaluation framework for assessing how well systems protect personally identifiable information in LLM prompts.

• Proposes a query-unrelated PII masking strategy to preserve functionality while protecting privacy
• Contains 2,842 test samples spanning 55 fine-grained PII categories
• Evaluates systems across diverse real-world scenarios with varying complexity
• Establishes benchmarks for balancing privacy protection with maintaining query intent

This research is critical for security professionals implementing LLM solutions in sensitive environments where data privacy regulations must be enforced while maintaining utility.

PII-Bench: Evaluating Query-Aware Privacy Protection Systems