Pipeline UI unable to access artifacts stored in external cloud storage

[pavankumarbo]

Hello,
I am using light weight components as shown in this example:

lightweight_component

But my source for confusion_matrix is coming from s3 location.
I can specify use_aws_secret from kfp.aws to access data at s3://xxx inside the componenets, but Pipeline UI is not able to read data from s3.
How do I provide the aws secrets to pipeline UI?

[Ark-kun]

@eterna2 Does that part work for you?

[pavankumarbo]

This is the metadata I am returning:

{'outputs': [{'storage': 'inline', 'source': '# Inline Markdown\n[A link](https://www.kubeflow.org/)', 'type': 'markdown'}, {'type': 'confusion_matrix', 'format': 'csv', 'schema': [{'name': 'target', 'type': 'CATEGORY'}, {'name': 'predicted', 'type': 'CATEGORY'}, {'name': 'count', 'type': 'NUMBER'}], 'source': 's3://<bucket>/<prefix>/confusion_matrix.csv', 'labels': ['a', 'b']}]}
metadata {"outputs": [{"storage": "inline", "source": "# Inline Markdown\n[A link](https://www.kubeflow.org/)", "type": "markdown"}, {"type": "confusion_matrix", "format": "csv", "schema": [{"name": "target", "type": "CATEGORY"}, {"name": "predicted", "type": "CATEGORY"}, {"name": "count", "type": "NUMBER"}], "source": "s3://slb-plug/ml_resources/kubeflow/metricsconfusion_matrix.csv", "labels": ["a", "b"]}]}

here is the code I am using for the light weight component.
s3_cm_source_light_weight_comp.txt

I am able to access the confusion matrix data from s3 inside the component, but the UI cannot.

[Bobgy]

I cannot test on AWS, but UI pod already uses standard env vars to get credentials from AWS: https://github.com/kubeflow/pipelines/blob/master/frontend/server/server.ts#L45.

What's your use case? Is it enough everyone shares one credentials?

You can

kubectl edit deployment ml-pipeline-ui

and add extra envs needed for authentication.

We don't yet have a plan to provide a first party solution.

[eterna2]

Yes. It works for me. But u need to either pass the AWS API keys to the pipeline UI pod (as bobgy mentioned) or in my case, assign an IAM role to the pod.

For tensorboard, u need to create a configmap with a modified pod template (to either set the AWS API keys or IAM role), mount it to the UI pod and point to it with the env var VIEWER_TENSORBOARD_POD_TEMPLATE_SPEC_PATH

[eterna2]

I will try to do a kustomize overlay for both AWS API key variant and IAM variant.

[pavankumarbo]

@Bobgy As of now, passing AWS credentials into the ml-pipeline-UI container specification is good enough for me. Thanks for the information.

@eterna2 Is there an example of configmap that needs to be created for tensorboard?

[eterna2]

@pavankumarboinapalli

NOTE that this is assuming u are using kube2iam or some equivalent. Alternatively, u can just add the AWS access key into the env var straight (or from secret). This is a standard k8s podTemplateSpec.

{
    "metadata": {
        "annotations": {
            "iam.amazonaws.com/role": "some-iam-role-name"
        }
    },
    "spec": {
        "containers": [
            {
                "env": [
                    {
                        "name": "AWS_REGION",
                        "value": "ap-southeast-1"
                    }
                ]
            }
        ]
    }
}

[eterna2]

sorry I was using kustomize.
Here is the how it looks like as k8s configmap yaml.

apiVersion: v1
data:
  template.json: |-
    {
        "metadata": {
            "annotations": {
                "iam.amazonaws.com/role": "some-iam-role-name"
            }
        },
        "spec": {
            "containers": [
                {
                    "env": [
                        {
                            "name": "AWS_REGION",
                            "value": "ap-southeast-1"
                        }
                    ]
                }
            ]
        }
    }
kind: ConfigMap
metadata:
  name: tensorboard-ui-template-gfmg77c7hd

[eterna2]

You can look at #2633 on how to configure the tensorboard podTemplateSpec, for both access key and IAM access.

[Bobgy]

it seems the questions are already answered
/close

[k8s-ci-robot]

@Bobgy: Closing this issue.

In response to this:

it seems the questions are already answered
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[Shasvat0601]

@pavankumarboinapalli @eterna2
what do you mean by ml-pipeline UI pod here?
how do I pass my aws credentials into UI pod

I am facing similar issue. Pipeline UI gets 500 error: access denied when it tries to read data for confusion matrix from s3
any help is appreciated
Thank you!

[Shasvat0601]

@eterna2 @pavankumarboinapalli @Ark-kun
I am attaching the AWS credentials ENV in pipeline UI by mounting the secret as a volume

Still I get 500 error: access denied when the pipeline UI tries to fetch my source for confusion_matrix from s3 location.

What can be causing this?

Containers:

ml-pipeline-ui:

Container ID: docker://415762af72b61d51503cf923f10904ad9ec9d5dd1e84d2b13d9ca812f664dfee
Image:          gcr.io/ml-pipeline/frontend:0.1.31
Image ID:       docker-pullable://gcr.io/ml-pipeline/frontend@sha256:7022552d3481edab0e683f9fcdad2ee29f2cabcfd54536d63dfaf20c101a6ed6
Port:           3000/TCP
Host Port:      0/TCP
State:          Running
  Started:      Mon, 09 Dec 2019 18:01:29 -0500
Ready:          True
Restart Count:  0
Environment:    <none>
Mounts:
  /var/run/secrets/kubernetes.io/serviceaccount from ml-pipeline-ui-token-265pv (ro)
Volumes:
  ml-pipeline-ui-token-265pv:
  Type:        Secret (a volume populated by a Secret)
  SecretName:  ml-pipeline-ui-token-265pv
  Optional:    false

[eterna2]

hi @Shasvat0601

You can look at https://github.com/e2fyi/kubeflow-aws/tree/master/pipelines for a manifest that support using S3 as the backend instead of minio.

Specifically, you shld look at https://github.com/e2fyi/kubeflow-aws/blob/3220922250ccb9b6207c8fb4fe60db3669ff0508/pipelines/overlay/accesskey/aws-configurations-patch.yaml#L40

on how to mount an AWS credential secret. The link above demonstrate how to mount the credential as env variable.

Volume mounting the secret will not work, as we are not using AWS official client (we are using minio). In any case, even if you are volume mounting, you shld ensure the secret is locate in the right place and have the same layout as the AWS credential file.

[Shasvat0601]

hi @eterna2
Thank you for your reply. Appreciate it :)
Could you tell me what does this line mean?
Volume mounting the secret will not work, as we are not using AWS official client (we are using minio).