[Feature] Support service discovery on Kubernetes in router

[gaocegege]

Checklist

• 1. If the issue you raised is not a feature but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
• 2. Please use English, otherwise it will be closed.

Motivation

This feature proposes adding Kubernetes service discovery support to the router component. Service discovery will enable the router to dynamically identify and connect to backend services running in a Kubernetes cluster. This is particularly useful for distributed systems where backend instances may scale up or down dynamically.

UI/UX

# New approach
python -m sglang_router.launch_router --worker-service-on-k8s default/sglang-svc
# Static approach
python -m sglang_router.launch_router --worker-urls http://worker_url_1 http://worker_url_2

Pseudo code

# Load Kubernetes configuration (e.g., from kubeconfig or in-cluster config)
load_kube_config()

# Initialize Kubernetes API client
api_client = CoreV1Api()

# Define the service name and namespace
service_name = "my-service"
namespace = "default"

# Step 1: Get the service's selector
try:
    service = api_client.read_namespaced_service(service_name, namespace)
    selector = service.spec.selector  # e.g., {"app": "my-app"}
except ApiException as e:
    print(f"Error fetching service: {e}")
    exit(1)

# Step 2: List pods matching the selector
try:
    label_selector = ",".join([f"{k}={v}" for k, v in selector.items()])  # e.g., "app=my-app"
    pods = api_client.list_namespaced_pod(namespace, label_selector=label_selector)
except ApiException as e:
    print(f"Error listing pods: {e}")
    exit(1)

# Step 3: Extract pod IPs
pod_ips = []
for pod in pods.items:
    pod_name = pod.metadata.name
    pod_ip = pod.status.pod_ip
    if pod_ip:
        pod_ips.append((pod_name, pod_ip))
    else:
        print(f"Pod {pod_name} does not have an IP assigned yet.")

# Step 4: Output the results
print("Pods and their IPs:")
for pod_name, pod_ip in pod_ips:
    print(f"- {pod_name}: {pod_ip}")

Related resources

Maybe related to #2932

[zhaochenyang20]

cc @ByronHsu

[ByronHsu]

Each k8s svc can host multiple workers and scale up/down dynamically? Happy to chat offline on this. Feel free to ping me on sglang slack.