This research introduces DGD-LB, a novel distributed load balancing algorithm that optimizes routing of LLM inference requests across global networks with minimal communication overhead.

• Addresses the challenge of network latencies in distributed LLM serving systems
• Uses a fluid model approach with continuous flows of requests between frontends and backends
• Implements distributed gradient descent to dynamically adjust routing decisions
• Achieves near-optimal performance despite delayed feedback from backends

For engineering teams, this work provides a practical framework for scaling LLM infrastructure globally while maintaining response time quality, particularly valuable as AI inference workloads continue to grow exponentially.

Load Balancing with Network Latencies via Distributed Gradient Descent