[Core]Read working_dir zip from private s3

[loleek]

Description

Currently, boto3 cannot inject endpoint_url from ~/.aws/config or env_vars.
So I cannot submit a job which located at my private s3-like storage using s3-prefix working_dir.
I see the packing.py create boto3.client very simple:

 if protocol == Protocol.S3:
       try:
           import boto3
           from smart_open import open as open_file
       except ImportError:
           raise ImportError(
               "You must `pip install smart_open` and "
               "`pip install boto3` to fetch URIs in s3 "
               "bucket. " + install_warning
           )
       tp = {"client": boto3.client("s3")}

Can ray expose kwags here so I can inject endpoint_url, access_key_id, access_secret or other configurations?

Use case

No response

[rkooo567]

Hmm this seems to be a pretty reasonable request. cc @architkulkarni for thoughts?

[architkulkarni]

I agree, it seems reasonable. It may take some care to get the API right.

[rkooo567]

This is as similar issue as #31122.

[ketangangal]

Hello @rkooo567 & @architkulkarni

I can work on this issue !

[ketangangal]

Requirements :

1. if From Kwargs endpoint_url, access_key_id, access_secret or other configurations?
2. elif From ENV endpoint_url, access_key_id, access_secret or other configurations?
3. else. default boto3

[rkooo567]

that sounds great! @rynewang can shepherd the contribution!

[rynewang]

The same mechanism is used by java_jars, py_modules and working_dir. We can find a common way to model these.

We can either type the working_dir value to (pseudocode) str | Dict["uri" | **kwargs, str], accepting

• runtime_env={"working_dir":{"uri":"s3://my_private_bucket/dir", "aws_access_key_id":"my_key", "aws_secret_access_key":"my_secret"}}
• runtime_env={"working_dir":"s3://public_bucket/dir"}

Or we can add a special field in runtime_env _runtime_env_context like

• runtime_env={"working_dir":"s3://my_private_bucket/dir", "_runtime_env_context":{"aws_access_key_id":"my_key", "aws_secret_access_key":"my_secret"}}

and let different runtime env plugins to read them.

I personally prefer the former one, but I'd like to hear from you. cc @jjyao @rkooo567

[architkulkarni]

I believe boto3 now supports specifying the endpoint URL via environment variable (which was the initial reason for creating this issue). See boto/boto3#2099 (comment)

If we can verify that this workflow can succeed by setting the relevant variables in the runtime_env "env_vars" field, then perhaps we don't even need to add a new API here. Instead we can just add a quick section for this workflow in the docs.