A novel scheduling framework called ECCOS that intelligently routes queries to appropriate LLMs based on query complexity and computational costs.

• Routes simple queries to smaller, faster, cheaper LLMs
• Directs complex queries to more capable but costly models
• Optimizes overall system performance while reducing computational waste
• Demonstrates effective cost-capability balancing in multi-LLM deployments

Why It Matters: This approach enables organizations to build more efficient AI systems that maximize computational resources and reduce operational costs while maintaining high-quality responses across varying query types.

Smart Routing: Cost-Effective Multi-LLM Serving for Multi-Core AIOS