feat(backend): preserve querystring in pipeline root (fixes #10318)

[TobiasGoerke]

The kfp-driver now recognizes query params and assembles bucket names correctly.

See #10318

[google-cla]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

[TobiasGoerke]

Using custom S3 endpoints now works for me.

Ready for review!

[Tomcli]

/lgtm

[Tomcli]

/cc @chensun

[TobiasGoerke]

Thanks @Tomcli.
I pushed another minor change, see here.

This causes filenames to be stored without the query string and the artifacts in the pipeline yaml don't contain the connection string.
It's more a cosmetic change but also lets the pipelines ui correctly download the artifacts.

[Tomcli]

Since now you new function is similar to the old generateOutputURI function, it make sense to bring these comments to the changes you have at line 1065 or just update the generateOutputURI function.

[google-oss-prow]

@TobiasGoerke: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
kubeflow-pipelines-samples-v2	e2c34bf	link	false	/test kubeflow-pipelines-samples-v2

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[Tomcli]

thanks @TobiasGoerke
/lgtm

[chensun]

/approve

[google-oss-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: chensun

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• backend/OWNERS [chensun]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[thesuperzapper]

@chensun @Tomcli @connor-mccarthy is there any chance we can cut a 2.0.6 patch release that has this PR included?

It's pretty important, because external object stores can't be used with KFP v2 without query parameters (in the pipeline root URI).

This will be less important if we ever get #9689 resolved, but that's a longer term fix.

[RonaldFletcher]

Thanks @Tomcli. I pushed another minor change, see here.

This causes filenames to be stored without the query string and the artifacts in the pipeline yaml don't contain the connection string. It's more a cosmetic change but also lets the pipelines ui correctly download the artifacts.

How to use custom S3 endpoints ？Can you give an example? @TobiasGoerke

[TobiasGoerke]

Thanks @Tomcli. I pushed another minor change, see here.
This causes filenames to be stored without the query string and the artifacts in the pipeline yaml don't contain the connection string. It's more a cosmetic change but also lets the pipelines ui correctly download the artifacts.

How to use custom S3 endpoints ？Can you give an example? @TobiasGoerke

For v2 pipelines, you'll need to set the kfp-launcher ConfigMap's data.defaultPipelineRoot to your desired endpoint. Depending on your endpoint, you may now also need to set some env vars for authentication in your pipeline pods and artifacts proxies etc.