PseudoEval introduces a novel approach to evaluate LLMs' programming abilities by isolating problem-solving logic from language-specific syntax knowledge.

• Distinguishes between an LLM's ability to solve problems conceptually versus its familiarity with programming language syntax
• Enables precise identification of where LLMs struggle in code generation tasks
• Demonstrates how different models perform across the problem-solving-to-code-writing pipeline
• Provides a targeted evaluation framework for improving code generation capabilities

This research is critical for engineering teams developing or using code-generating AI, as it helps identify specific areas to improve model training and provides a clearer understanding of model limitations in real-world programming tasks.

Isolating Language-Coding from Problem-Solving: Benchmarking LLMs with PseudoEval