replacement for https://github.com/ray-project/kuberay/pull/1312

[blublinsky]

Why are these changes needed?

This is a replacement for #1312. Cleaned up

Related issue number

Checks

• I've made sure the tests are passing.
• Testing Strategy
  • [ x] Unit tests
  • Manual tests
  • This PR is not tested :(

[kevin85421]

cc @tedhtchang @z103cb could you please review and approve this PR again? Ignoring some details in #1312 is easy because it changes 45 files. Thanks!

[tedhtchang]

LGTM. I manually verified dashboard ingress was accessible and the ephemeral volume was created. Steps verify:
Create cluster:

cat <<EOF | kind create cluster --name ray-test --config -
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - role: control-plane
    kubeadmConfigPatches:
      - |
        kind: InitConfiguration
        nodeRegistration:
          kubeletExtraArgs:
            node-labels: "ingress-ready=true"
    extraPortMappings:
      - containerPort: 80
        hostPort: 80
        listenAddress: "0.0.0.0"
        protocol: tcp
EOF

Deploy Nginx ingress controller

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/kind/deploy.yaml

Deploy Kuberay

helm repo add kuberay https://ray-project.github.io/kuberay-helm/
helm install kuberay-operator kuberay/kuberay-operator --version 0.6.0 --set image.repository=quay.io/kuberay/operator

Apiserver run on separate terminal

make build && make run

Compute template

curl -X POST 'localhost:8888/apis/v1alpha2/namespaces/default/compute_templates' \
--data '{
  "name": "default-template",
  "namespace": "default",
  "cpu": 1,
  "memory": 1
}'

Create RayCluster with ingress and an ephemeral volume

curl -X POST 'http://localhost:8888/apis/v1alpha2/namespaces/default/clusters' --data '{
  "name": "myraycluster",
  "namespace": "default",
  "user": "tedchang",
  "version": "2.5.0",
  "annotations": {
    "kubernetes.io/ingress.class": "nginx",
    "nginx.ingress.kubernetes.io/rewrite-target": "/$1"
  },
  "clusterSpec": {
    "headGroupSpec": {
      "enableIngress": true,
      "computeTemplate": "default-template",
      "image": "quay.io/project-codeflare/ray:2.5.0-py38-cu116",
      "rayStartParams": {"dashboard-host": "0.0.0.0"},
      "volumes": [{
        "mountPath": "/data",
        "name": "test-vol",
        "volumeType": "EPHEMERAL",
        "storage": "1Gi"
        }
      ]
    },
    "workerGroupSpec": [
      {
        "groupName": "small-wg",
        "computeTemplate": "default-template",
        "image": "quay.io/project-codeflare/ray:2.5.0-py38-cu116",
        "replicas": 1,
        "minReplicas": 0,
        "maxReplicas": 1,
        "rayStartParams": {"metrics-export-port": "8080"}
      }
    ]
  }
}'

Verified Ray 1 head node and 1worker pod were created
Verify /data directory is writable in the headnode kubectl exec myraycluster-head-ql6xm -- touch /data/test1.txt
Verified dashboard accessible http://localhsot/myraycluster/#/overview in browser.

[kevin85421]

I have not reviewed this PR. I approved this PR because this PR only changes KubeRay API Server and @tedhtchang approves it.

[kevin85421]

The RayService e2e tests are known to be flaky. They are not related to this PR. Merge this PR.