This research introduces CACP (Counterfactual Analysis for Programming Concept Predicates) to evaluate whether large language models genuinely understand programming logic or merely predict syntax.

• Models show limited understanding of critical programming concepts
• LLMs can struggle with data flow and control flow relationships
• Performance varies significantly across different programming concepts
• Models often rely on superficial patterns rather than true semantic understanding

Security Implications: The research reveals important gaps in code models' reasoning capabilities that could impact their reliability for security-critical applications. Understanding these limitations is crucial for developing safeguards when deploying LLMs in code analysis, vulnerability detection, and secure programming tasks.

Do Large Code Models Understand Programming Concepts? Counterfactual Analysis for Code Predicates