Programming with Pixels (PwP) introduces a revolutionary approach that enables AI agents to interact directly with visual interfaces rather than being limited to predefined tool APIs.

• Overcomes limitations of traditional tool-based AI programming approaches
• Creates more generalizable agents that can work across programming languages and domains
• Eliminates need for specialized tool development for each new programming task
• Mimics how humans naturally interact with software interfaces

This research represents a significant advancement for Engineering by creating more flexible, adaptable AI programming assistants that can potentially work across any IDE or development environment without requiring specialized tool integration.

Programming with Pixels: Computer-Use Meets Software Engineering