SafeCLIP offers a lightweight, zero-shot method to protect Large Vision-Language Models (LVLMs) from harmful visual inputs without compromising performance or requiring expensive fine-tuning.

• Leverages inherent multimodal alignment in LVLMs to detect and filter harmful images
• Maintains model utility while enhancing security against toxic visual content
• Provides protection without the computational costs of traditional pre-filtering approaches
• Demonstrates effective defense without requiring access to harmful content during training

Security Impact: This research addresses a critical vulnerability in multimodal AI systems, enabling safer deployment of vision-language models in public-facing applications while maintaining their functionality.

Zero-Shot Defense Against Toxic Images via Inherent Multimodal Alignment in LVLMs