[k8s] SkyPilot overrides GIT_SSH_COMMAND environment variable

[romilbhardwaj]

Repro

Specify dockerfile:

# Dockerfile
FROM continuumio/miniconda3:23.3.1-0

ENV GIT_SSH_COMMAND="ssh -o HostKeyAlgorithms=+ssh-rsa -o PubkeyAcceptedKeyTypes=+ssh-rsa"

Build image:

docker buildx build --push --platform linux/amd64,linux/arm64 -t myrepo/customenv:latest -f Dockerfile_test .

Launch cluster using this image:

sky launch -c customenv --cloud kubernetes --image-id docker:berkeleyskypilot/customenv -- 'echo ${GIT_SSH_COMMAND};printenv'

Output:

(sky-cmd, pid=2714) ssh -o HostKeyAlgorithms
...
(sky-cmd, pid=2714) GIT_SSH_COMMAND=ssh -o HostKeyAlgorithms
...
# Should have been ssh -o HostKeyAlgorithms=+ssh-rsa -o PubkeyAcceptedKeyTypes=+ssh-rsa

Interestingly, kubectl exec -it customenv-2ea4-head /bin/bash has the right value. Looks like SkyPilot runtime is overriding it somewhere.

(base) ➜  ~ kubectl exec -it customenv-2ea4-head /bin/bash
(base) root@customenv-2ea4-head:/# printenv | grep GIT
GIT_SSH_COMMAND=ssh -o HostKeyAlgorithms=+ssh-rsa -o PubkeyAcceptedKeyTypes=+ssh-rsa

[concretevitamin]

Perhaps this is related to Ray? The job submission codepath does something like:

1. Generate a script job.sh

#!/bin/bash
source ~/.bashrc
# .. actual commands

2. Write a python script that essentially does subprocess.run('/bin/bash -i job.sh') (in reality Popen)

3. Invoke the above via ray job submit

Maybe we can check if kubectl exec -it plus one or more steps above can repro the problem.

[romilbhardwaj]

Was related to handling of spaces and equal signs in our k8s env var parsing, will be fixed in #3322.