Using S3 storage with alternative S3-compliant services (e.g. Minio)

[oleksandr]

Use Case

There're non-production and production setups, where instead of AWS S3 alternatives are used (e.g. Minio or S3 Gateway on Azure).

Current S3 storage relies on Boto3, which doesn't support environment variable to switch endpoints:
boto/boto3#2099

Solution

Extend S3 storage configuration arguments to allow overriding endpoint.

Alternatives

Less likely to happen fast enough, but is an alternative: wait until boto3 adds this feature and rely on its environment variable.

[joshmeek]

@oleksandr The S3 storage currently has a client_options kwarg for passing additional options to the boto3 client. Does that satisfy your use case?

- client_options (dict, optional): Additional options for the `boto3` client.

[oleksandr]

@joshmeek thanks for the prompt response! checking now...

[oleksandr]

@joshmeek that helped to register the flow (the flow appeared in the Minio server), although I've noticed execution failed (result handler couldn't PUT files to the custom server).

So the working solution requires specifying Storage and ResultHandler in an explicit way:

    flow.storage = S3(
        bucket="flows",
        aws_access_key_id="admin",
        aws_secret_access_key="password",
        client_options=dict(endpoint_url=os.getenv("AWS_S3_ENDPOINT")),
    )

    flow.result_handler = S3ResultHandler(
        "flows",
        boto3_kwargs=dict(endpoint_url=os.getenv("AWS_S3_ENDPOINT"))
    )

That solves my problem and I assume the issue can be closed...

Thanks for the hint!

[FeryET]

Is it possible to handle this by the ui in prefect UI? On premise Minio is used a lot in our pipelines.