The orchestration layer is critical to overall system efficiency in large-scale LLM serving, as how effectively it distributes heterogeneous request streams across LLM engine replicas impacts TTFT, TPOT, and throughput. When operating a cluster of vLLM replicas, the most critical decision the router makes is who gets the next request.