CoSine introduces a collaborative speculative inference framework that significantly improves LLM serving efficiency while maintaining output quality.

• Employs multiple specialized small speculative models (SSMs) as drafters working in parallel
• Implements an innovative token fusion mechanism to combine draft tokens from different SSMs
• Achieves up to 2.6× speedup over traditional inference and outperforms existing speculative methods
• Optimizes resource utilization through dynamic workload allocation and parallel verification

This innovation addresses critical engineering challenges in LLM deployment, offering practical solutions for reducing inference latency and computational costs in production environments.

Collaborative Speculative Inference for Efficient LLM Inference Serving