[kuberay][autoscaler] Use new autoscaling fields from the KubeRay operator

[DmitriGekhtman]

Why are these changes needed?

This PR incorporates changes from KubeRay PRs ray-project/kuberay#163 ray-project/kuberay#274 ray-project/kuberay#278

• adds logic to process the new fields upscalingMode and idleTimeoutSeconds in the KubeRay CR
• refactors the example config to use kuberay's ability to autogenerate an autoscaler container
• uses the new autoscalerOptions.image field to set the autoscaler image in the test
• uses a small idleTimeoutSeconds to test scale-down in the autoscaler e2e test

Checks

• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Integration tests

[DmitriGekhtman]

Hmm...getting some logs claiming the operator doesn't have permissions to list service accounts, but the RBAC looks as expected...need to debug more.

[DmitriGekhtman]

Ok, having the issue with the operator not being able to list service accounts in upstream KubeRay as well.
Also reproduced with some dummy objects in my K8s cluster. Maybe my K8s cluster is messed up somehow?

[DmitriGekhtman]

Ok...no, definitely not my K8s cluster...going to keep debugging.

[DmitriGekhtman]

...lol...
https://github.com/ray-project/kuberay/pull/285/files
I really wish K8s would validate that.

[DmitriGekhtman]

There's some incorrect downscaling behavior which is causing the test to fail. Going to debug.

[DmitriGekhtman]

Welcome, codeowner SWAT team.

[DmitriGekhtman]

There's some incorrect downscaling behavior which is causing the test to fail. Going to debug.

For once, autoscaler was correct, I (test) was wrong. Test fixed. Let's wait for CI to run.

[DmitriGekhtman]

Head pod failed to start in the CI, very strange...will probably have to debug the CI environment.

[DmitriGekhtman]

It's an issue with ImagePull policy for the autoscaler container. This will require another quick fix to the KubeRay repo.

[DmitriGekhtman]

...crossing fingers that it works and I don't have to debug it in the CI environment again....

[DmitriGekhtman]

Ah. There's the green check mark.
Will let the CI run to completion before merging.

[DmitriGekhtman]

100% green, except for the forsaken MacOS tests, which never run.
Merging.