This research benchmarks large multimodal models' ability to detect and safely respond to harmful memes, revealing critical security vulnerabilities.

• Introduces GOAT-Bench, a framework specifically designed to test LMMs against meme-based social abuse
• Evaluates 10 leading LMMs including GPT-4V, Claude, and Gemini against various types of harmful meme content
• Reveals significant safety gaps in current models' ability to recognize subtle harmful content
• Demonstrates how models often fail to detect harmful content when meaning is implicit

For security professionals, this research highlights the urgent need for better safeguards against multimodal threats as visual-language AI becomes more mainstream in business applications.

GOAT-Bench: Safety Insights to Large Multimodal Models through Meme-Based Social Abuse