[Core] Use original Python path for SkyPilot runtime

[Michaelvll]

Fixes #3324

master (823999a)
for i in {1..5}; do time sky launch -c test-gcp-$i --cloud gcp --cpus 2 -y; done

1m59.195s
2m0.108s
1m56.507s
1m54.553s
1m54.242s

mean: 1m56.921s

This PR (39daf35):
for i in {1..5}; do time sky launch -c test-gcp-$i --cloud gcp --cpus 2 -y; done

1m53.456s
2m7.647s
1m55.891s
1m59.496s
2m7.193s

mean: 2m0.7366s (3.8156s overhead)

Tested (run the relevant ones):

• Code formatting: bash format.sh
• Any manual or new tests for this PR (please specify below)
  • sky launch --cloud azure -c test-azure --cpus 2+ echo hi
  • sky launch -c test-env task.yaml --cloud gcp

    resources:
      cpus: 2+
    
    setup: |
      conda activate myenv
      if [ $? -ne 0 ]; then
        conda create -n myenv python=3.10 -y
        conda activate myenv
      fi
    
      grep -qxF 'conda activate myenv' ~/.bashrc || echo "conda activate myenv" >> ~/.bashrc
    
    run: |
      echo hi
      echo bye

• All smoke tests: pytest tests/test_smoke.py
• pytest tests/test_smoke.py --aws
• Relevant individual smoke tests: pytest tests/test_smoke.py::test_fill_in_the_name
• Backward compatibility tests: bash tests/backward_comaptibility_tests.sh

[concretevitamin]

Thanks @Michaelvll!

Wondering if we should test against the following. This change invokes <fixed path to ray/sky> under arbitrary user-created conda env. Let's say

• Ray/Sky depend on some library X requiring certain version range
• User conda env removed this library (or changed its version to be outside the range)

Then, does this change still work?

We may need to run all smoke tests too.

[Michaelvll]

This change invokes <fixed path to ray/sky> under arbitrary user-created conda env

We are invoking the ray/sky in the base or the default env when the VM is just created, instead of the user-created conda env.

Just tested with the following yaml and it works well:

resources:
  cpus: 2+

setup: |
  conda activate myenv
  if [ $? -ne 0 ]; then
    conda create -n myenv python=3.7 -y
    conda activate myenv
  fi

  grep -qxF 'conda activate myenv' ~/.bashrc || echo "conda activate myenv" >> ~/.bashrc
  pip install ray==2.6.0

run: |
  conda env list
  echo hi
  echo bye

[concretevitamin]

Thanks @Michaelvll! Do we need to run some back compat tests too?

[concretevitamin]

Why not grep this too on L16?

[Michaelvll]

L16 is for backward compatibility, as we need to make sure that the skylet started before this PR (will not have the full path to the python executable) will be correctly killed.

[concretevitamin]

LGTM, thanks @Michaelvll!