[k8s_cloud] Add minio object store support.

[aviweit]

We have an on-prem installation of minio object store, which is S3 compatible. We had been able to add minio support to SkyPilot and test it against our local installation with Kubernetes.

We added example tasks under examples/storage/minio and smoke tests that can be invoked via pytest -v tests/test_smoke.py --generic-cloud kubernetes -m minio --minio.

We would like to contribute this to SkyPilot.

Thanks.

[aviweit]

Hi @romilbhardwaj ,
Can you please review and let us know what do you think?
Thanks.

[aviweit]

Please find below a setup example:

1. Start minio container:

docker run -p 9000:9000 -d -p 9001:9001 -e "MINIO_ROOT_USER=minio99" -e "MINIO_ROOT_PASSWORD=minio123" quay.io/minio/minio server /data --console-address ":9001"

2. Loginto minio web ui (localhost port 9001) with above minio root user/password. Select Access Key at left pane and generate it.

3. Create ~/.sky/config.yaml with minio endpoint:

    minio:
        endpoint: "http://<your minio host name>:9000"

4. Invoke AWS CLI to set access and secretes keys (generated in the step above). For rest prompts - enter defaults.

AWS_SHARED_CREDENTIALS_FILE=~/.minio/minio.credentials aws configure --profile minio

5. Invoke sky check. Kubernetes and minio should pass.

[aviweit]

I can see that one test failed when running pytest -v tests/test_smoke.py --generic-cloud kubernetes -m minio --minio
But perhaps it is ok as it attempted to use an invalid name?

weit@esxi3-vm-12:~/skypilot$ pytest -v tests/test_smoke.py  --generic-cloud kubernetes -m minio --minio
============================================================================ test session starts ============================================================================
platform linux -- Python 3.8.10, pytest-7.4.0, pluggy-1.2.0
rootdir: /home/weit/skypilot
configfile: pyproject.toml
plugins: xdist-3.3.1, env-0.8.2
16 workers [9 items]
F.[minio_storage_mounts] Test started. Log: less /tmp/minio_storage_mounts-1scxpwuw.log
......[minio_storage_mounts] Passed.
[minio_storage_mounts] Log: less /tmp/minio_storage_mounts-1scxpwuw.log
[minio_storage_mounts]
.
================================================================================= FAILURES ==================================================================================
_____________________________________________ TestStorageWithCredentials.test_invalid_names[invalid_name_list3-StoreType.MINIO] _____________________________________________
[gw8] linux -- Python 3.8.10 /usr/bin/python3
../.local/lib/python3.8/site-packages/sky/data/storage.py:2372: in _create_minio_bucket
    minio_client.create_bucket(Bucket=bucket_name)
../.local/lib/python3.8/site-packages/botocore/client.py:530: in _api_call
    return self._make_api_call(operation_name, kwargs)
../.local/lib/python3.8/site-packages/botocore/client.py:964: in _make_api_call
    raise error_class(parsed_response, operation_name)
E   botocore.exceptions.ClientError: An error occurred (InvalidBucketName) when calling the CreateBucket operation: The specified bucket is not valid.

The above exception was the direct cause of the following exception:
../.local/lib/python3.8/site-packages/sky/data/storage.py:733: in add_store
    store = store_cls(
../.local/lib/python3.8/site-packages/sky/data/storage.py:2102: in __init__
    super().__init__(name, source, region, is_sky_managed,
../.local/lib/python3.8/site-packages/sky/data/storage.py:225: in __init__
    self.initialize()
../.local/lib/python3.8/site-packages/sky/data/storage.py:2145: in initialize
    self.bucket, is_new_bucket = self._get_bucket()
../.local/lib/python3.8/site-packages/sky/data/storage.py:2320: in _get_bucket
    bucket = self._create_minio_bucket(self.name)
../.local/lib/python3.8/site-packages/sky/data/storage.py:2375: in _create_minio_bucket
    raise exceptions.StorageBucketCreateError(
E   sky.exceptions.StorageBucketCreateError: Attempted to create a bucket ab but failed.

During handling of the above exception, another exception occurred:
tests/test_smoke.py:3038: in test_invalid_names
    storage_obj.add_store(store_type)
../.local/lib/python3.8/site-packages/sky/data/storage.py:742: in add_store
    global_user_state.set_storage_status(self.name,
../.local/lib/python3.8/site-packages/sky/global_user_state.py:686: in set_storage_status
    raise ValueError(f'Storage {storage_name} not found.')
E   ValueError: Storage ab not found.
========================================================================== short test summary info ==========================================================================
FAILED tests/test_smoke.py::TestStorageWithCredentials::test_invalid_names[invalid_name_list3-StoreType.MINIO] - ValueError: Storage ab not found.
========================================================== 1 failed, 8 passed, 1537 warnings in 120.47s (0:02:00) ===========================================================

[romilbhardwaj]

Thanks @aviweit! Having minio support is great for enabling onprem storage. Will take a closer look soon.

cc @landscapepainter

[tammam1998]

Should this be num_objects > 1?

Thanks for the PR, I'm looking at it to learn how to add new storage options.

[romilbhardwaj]

Hey @aviweit - this PR got closed when we merged #2096 into master.

For our next sprint, we'll be switching to k8s_cloud_beta1 branch. Can you re-open this PR on that branch?