kfp-api from channel 2.0/stable isn't properly applying the object-store-bucket-name value

[jpnorenam]

Bug Description

After deploying Charmed Kubeflow from the 1.8/stable bundle, pipelines fail to upload artifacts to any non-default MinIO bucket "mlpipeline". For example, running the pipeline for [Tutorial] Data passing in python components results in:

F0102 13:04:00.790460 34 main.go:49\] failed to execute component: failed to upload output artifact "output\_dataset\_two\_path" to remote storage URI "minio://mlpipeline/v2/artifacts/tutorial-data-passing/3753b6d2-58aa-4246-9f53-cbc6dde56437/preprocess/output\_dataset\_two\_path": uploadFile(): unable to complete copying "/minio/mlpipeline/v2/artifacts/tutorial-data-passing/3753b6d2-58aa-4246-9f53-cbc6dde56437/preprocess/output\_dataset\_two\_path" to remote storage "preprocess/output\_dataset\_two\_path": failed to close Writer for bucket: blob (key "preprocess/output\_dataset\_two\_path") (code=Unknown): InternalError: We encountered an internal error, please try again.: cause(Put "[https://mlpipeline.s3.dualstack.us-east-1.amazonaws.com/v2/artifacts/tutorial-data-passing/3753b6d2-58aa-4246-9f53-cbc6dde56437/preprocess/output\_dataset\_two\_path](https://mlpipeline.s3.dualstack.us-east-1.amazonaws.com/v2/artifacts/tutorial-data-passing/3753b6d2-58aa-4246-9f53-cbc6dde56437/preprocess/output_dataset_two_path)": 301 response missing Location header)

Modifying the object-store-bucket-name config in kfp-api seems to have issues from 2.0/stable.

To Reproduce

1. Deploy Charmed Kubeflow from channel 1.8/stable with an overlay for minio, kfp-api and argo-controllers:

  minio:
    options:
      mode: gateway
      gateway-storage-service: s3
      access-key: AKIA*****************
      secret-key: ****************************
  kfp-api:
    options:
      object-store-bucket-name: my-non-default-ckf-bucket
  argo-controller:
    options:
      bucket: my-non-default-ckf-bucket

2. Run [Tutorial] Data passing in python components

Environment

microk8s v1.28.3
juju v3.1.6
kubeflow 1.8/stable
kfp-api 2.0/stable rev 1035

Relevant Log Output

.

Additional Context

No response

[syncronize-issues-to-jira]

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/KF-5161.

This message was autogenerated

[ca-scribner]

We pass the config here and it appears to be valid.

@jpnorenam noted that this worked for CKF 1.7 with kfp 2.0-alpha.7, but not with CKF 1.8 using kfp 2.0.3, so it might be kfp 2.0.3 has mlpipeline hard coded somewhere? We've seen similar issues with kfp before