DuoDecoding dramatically improves LLM inference speed by introducing a hardware-optimized approach to speculative decoding, addressing key bottlenecks in traditional methods.

• Achieves 3.6x average speedup over existing speculative decoding techniques
• Implements dynamic multi-sequence drafting to generate multiple draft candidates efficiently
• Utilizes hardware-aware deployment strategies to optimize CPU/GPU resource allocation
• Maintains output quality while reducing time to first token (TTFT)

This innovation matters for engineering teams seeking to deploy efficient LLM systems with minimal latency in real-world applications, enabling faster response times without requiring additional computing resources.

DuoDecoding: Hardware-aware Heterogeneous Speculative Decoding with Dynamic Multi-Sequence Drafting