Inner Thinking Transformer (ITT) reimagines Transformer architecture by dynamically allocating computational resources where needed most, especially for complex reasoning tokens.

• Identifies and addresses gradient spikes across layers that occur during critical reasoning steps
• Implements dynamic depth scaling to allocate more processing power to challenging tokens
• Achieves improved performance while maintaining efficient computational footprint
• Provides a framework for models to adaptively engage in deeper processing when faced with complex reasoning tasks

This architectural innovation helps overcome performance bottlenecks in standard Transformers, allowing more efficient allocation of computational resources precisely where they deliver the most impact on reasoning capabilities.

Inner Thinking Transformer: Leveraging Dynamic Depth Scaling to Foster Adaptive Internal Thinking