runpod docker credentials not working when using image_id from private repository

[alita-moore]

Because runpod doesn't support docker I need to use my docker image as the base image_id but setting the SKYPILOT_DOCKER_USERNAME and SKYPILOT_DOCKER_PASSWORD doesn't allow the created runpod to have access to the private repo. However, if you manually change the dockerhub config on the runpod web UI (which I setup beforehand) it works fine. Is there a way to authenticate the docker image for the created pod automatically?

For reference here's my config:

resources:
  image_id: docker:teamwoven/convert:sultan-1.0
  cloud: runpod
  ports: 8000
  accelerators: RTX4090:1

service:
  readiness_probe: /status

envs:
  SKYPILOT_DOCKER_USERNAME:
  SKYPILOT_DOCKER_PASSWORD:
  SKYPILOT_DOCKER_SERVER: docker.io

setup: |
  #

run: |
  # Run the FastAPI server with two workers
  /home/woven/.local/bin/uvicorn extractor_inference.app:app --host "0.0.0.0" --port 8000 --workers 2

and then an image of the runpod UI that I'm referring to:

Version & Commit info:

• sky -v: skypilot, version 0.7.0
• sky -c: 3f62588

[alita-moore]

also unrelated but it would be nice to be able to specify multiple cloud providers in the yaml

[concretevitamin]

also unrelated but it would be nice to be able to specify multiple cloud providers in the yaml

This is supported; check out https://skypilot.readthedocs.io/en/latest/examples/auto-failover.html#multiple-candidate-resources.

[alita-moore]

Got it, thanks. Any idea about the docker credentials?

[Michaelvll]

It can be an issue with our pumping for the docker credentials for RunPod specifically. cc'ing @cblmemo for a look.

[cblmemo]

Yes, this is due to we directly setting the image id to create pod instead of using the DockerInitializer. A solution would be manually calling the runpod credential API if env variables is detected. Will submit a PR to fix this.

[alita-moore]

@cblmemo how can I help get this resolved? I'm happy to contribute however needed :)

[cblmemo]

@cblmemo how can I help get this resolved? I'm happy to contribute however needed :)

Thanks for pinging! I'll take a look today. Could you share which docker registry are you using right now? This can help us debug :))

[alita-moore]

thank you! I'm using dockerhub right now for the ML images.

[cblmemo]

thank you! I'm using dockerhub right now for the ML images.

Thanks for the information! #4287 should work now. I tested with AWS ECR and it mostly works well except for some runpod api issue. Are you interested to test it out?

[alita-moore]

Sure thing, is it in the latest version of sky or do I need to do something different?

[cblmemo]

Sure thing, is it in the latest version of sky or do I need to do something different?

Try installing from source and checkout this branch:

git clone https://github.com/skypilot-org/skypilot.git
cd skypilot
git switch runpod-docker-login
pip install -e ".[all]"

[alita-moore]

sorry about that! I've been very busy recently, do you still need me to test it out?

[cblmemo]

sorry about that! I've been very busy recently, do you still need me to test it out?

We've already tested and merged it! Now it should be available on the nightly build. Feel free to try it out and see if that resolves your requirement!