[manifest] Added manifest for deploying on aws using s3 #2633

eterna2 · 2019-11-20T18:21:23Z

There are a number of requests both in the issues (#2627 #1610 #1131) as well as in the slack channel on how to configure kubeflow pipeline on AWS (namely interacting with S3 buckets, particularly for pipeline-ui).

This PR adds 2 kustomize overlays:

accesskey: overlay that reference a secret with aws credentials
iam: overlay that annotate the services with the iam role (assuming kube2iam or similar are already deployed in the cluster)

The more common configurations (e.g. bucket, folder) are also exposed inside params.env. archiveLogs for argo is also set to true so that the logs are persisted and can be retrieved by the ui.

A README is also provided in the root dir of the overlays.

This change is

k8s-ci-robot · 2019-11-20T18:21:38Z

Hi @eterna2. Thanks for your PR.

I'm waiting for a kubeflow member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Ark-kun · 2019-11-21T19:06:35Z

manifests/kustomize/env/aws/accesskey/aws-configurations-patch.yaml

+      containers:
+        - name: ml-pipeline-ui
+          env:
+            - name: AWS_ACCESS_KEY_ID


@IronPan In future we should probably switch to getting the keys from ConfigMap.

access_key_id and access_key sounds like they belong to a secret. I think this is appropriate.

access_key_id and access_key sounds like they belong to a secret.

AFAIK, they're not the secret itself, more like secret name.

P.S. If the Frontend currently gets those configurations from the ENV, then I guess this PR just reflects that. We should move them to configMap when we have a chance.

Ark-kun · 2019-11-21T19:07:16Z

manifests/kustomize/env/aws/accesskey/aws-configurations-patch.yaml

+                  key: $(awsSecretKeySecretKey)
+            - name: ARGO_ARCHIVE_LOGS
+              value: "true"
+            - name: ARGO_ARCHIVE_ARTIFACTORY


@Bobgy @IronPan I wonder why UI is not getting these values from ConfigMap. Probably the configmap did not exist at the time.

Bobgy

Thanks for the contribution!

I can see your efforts in this and I believe this could be helpful to many aws users.

My concerns are:
This repo doesn't have test infra to verify these configurations keep working. Therefore, no one can maintain/guarantee these are still up-to-date after future changes.

I suggest

make a repo of your own for these manifests and pin a pipeline release version as base, so that it doesn't break
add a readme page here pointing to your repo mentioning it's a variant maintained by you/community, it may not always be up-to-date with latest pipelines, but it is a good reference + any one can contribute to it after they figured out how to patch a later pipelines release.

What do you think?
@eterna2 @Ark-kun @IronPan

Bobgy · 2019-11-22T07:31:00Z

manifests/kustomize/env/aws/accesskey/aws-configurations-patch.yaml

+      containers:
+        - name: ml-pipeline-ui
+          env:
+            - name: AWS_ACCESS_KEY_ID


access_key_id and access_key sounds like they belong to a secret. I think this is appropriate.

Bobgy · 2019-11-22T07:35:02Z

manifests/kustomize/env/aws/accesskey/viewer-tensorboard-template-configmap.yaml

+                        },
+                        {
+                            "name": "AWS_REGION",
+                            "value": "ap-southeast-1"


Should this be configurable?

Yeah. Missed this.

eterna2 · 2019-11-22T11:49:52Z

Good idea. I will do that and revert this commit and update the readme instead.

Jeffwan

Overall looks good to me, my suggestion is to provide flexibility to support k2iam, EKS IAM for pod feature or other iam solutions

Jeffwan · 2019-11-22T23:34:57Z

manifests/kustomize/env/aws/iam/aws-configurations-patch.yaml

+  template:
+    metadata:
+      annotations:
+        iam.amazonaws.com/role: $(awsIAMRole)


Let's make the key configurable? iam.amazonaws.com/role is for k2iam users. As EKS release feature https://aws.amazon.com/blogs/opensource/introducing-fine-grained-iam-roles-service-accounts/, we have lots of users use eks.amazonaws.com/role-arn as well.

kustomize don't allow templating for keys yet - i.e. I cannot do $(key): $(value) for annotation.

a possible workaround is to use commonAnnotations (actually this is what I used for my deployment, cuz I was lazy. I gave all kfp deployments the same iam role)

This reverts commit 6a9c498.

…peline on AWS

eterna2 · 2019-11-23T18:52:24Z

Hey all,

Created a new repo: https://github.com/e2fyi/kubeflow-aws to hold the manifest. Streamlined the manifest too.

most of the configurations are done in base
variant:accesskey just patches the various deployment with the access key secret
variant:iam only patches the tensorboard configmap, and annotations are added using commonAnnotations

I also added a short write-up on the various components of kubeflow pipelines in my repo. Probably useful for pple to have some sense where and what to look at if they want to do something. Maybe I shld put this somewhere too - probably under developer notes?

service	description
`argo`	main workhorse to run the pipelines
`ml-pipeline-apiserver`	primary endpoint to interact with kubeflow pipelines APIs (e.g. get runs, experiments, etc)
`ml-pipeline-persistentagent`	multiple workers that performs various misc tasks - e.g. collect metrics, save resources to db, etc
`ml-pipeline-scheduledworkflow`	controller to schedule workflows
`ml-pipeline-ui`	serves the React frontend and nodejs backend - i.e. webapp for users to interact with pipelines
`ml-pipeline-viewer-crd`	controller to manage viewers - i.e. pods serving tensorboard
`ml-pipeline-visualizationserver`	endpoints to generate and serve visualizations for pipelines
`metadata-server`	endpoints to record and retrieve metadata associated with the workflows
`mysql`	primary database to store most of the pipeline resources and information
`minio`	`minio` can either be deployed as a standalone object store service, or as a gateway to object store services like S3. This manifest will not deploy `minio` service as we will be connecting to `s3` directly
`s3`	`s3` will be used as the primary storage for workflow templates and artifacts generated by the pipelines (i.e. logs, metrics, ui-metrics, intermediate data, etc)

For this PR:

i reverted the previous commit (remove all the manifest)
Added folder env/aws and a README inside (easier for pple to spot if they are looking at the repo)
Update the manifest README with a small section under customization

Bobgy · 2019-12-05T00:48:56Z

manifests/kustomize/README.md

+provides a community-maintained manifest for deploying kubeflow pipelines on AWS
+(with S3 as artifact store instead of minio).
+
+TL;DR


Shall we also move these instructions into the new repo?
I think that makes it more self contained, and instruction + manifests are versioned together makes more sense

We should probably still have at least a small section about AWS and a pointer to the other repo. @IronPan WDYT?

Agree, I was thinking the section only introduces the repo.

eterna2 · 2019-12-11T16:15:17Z

Ok. I have removed the unnecessary deployment instructions in the readme.

Bobgy · 2019-12-12T01:13:35Z

/lgtm
/approve

Thanks! this looks good

k8s-ci-robot · 2019-12-12T01:13:42Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Bobgy

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~manifests/kustomize/OWNERS~~ [Bobgy]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Bobgy · 2019-12-12T01:13:53Z

/ok-to-test

Added manifest for deploying on aws using s3

6a9c498

k8s-ci-robot added size/XL needs-ok-to-test labels Nov 20, 2019

k8s-ci-robot requested review from Bobgy and IronPan November 20, 2019 18:21

eterna2 mentioned this pull request Nov 21, 2019

Pipeline UI unable to access artifacts stored in external cloud storage #2627

Closed

Ark-kun reviewed Nov 21, 2019

View reviewed changes

Ark-kun assigned IronPan, rmgogogo and Bobgy Nov 21, 2019

Ark-kun added the area/deployment label Nov 21, 2019

jlewi added area/kfctl and removed area/deployment labels Nov 22, 2019

Bobgy reviewed Nov 22, 2019

View reviewed changes

Jeffwan reviewed Nov 22, 2019

View reviewed changes

eterna2 added 2 commits November 24, 2019 02:30

Revert "Added manifest for deploying on aws using s3"

89944cb

This reverts commit 6a9c498.

Added readme and link to kubeflow-aws on how to deploy lightweight pi…

1631d4a

…peline on AWS

k8s-ci-robot added size/S and removed size/XL labels Nov 23, 2019

updated readme on how to deploy on aws

66789ee

k8s-ci-robot added size/M and removed size/S labels Nov 23, 2019

Bobgy reviewed Dec 5, 2019

View reviewed changes

eterna2 added 2 commits December 12, 2019 00:12

Update README.md

218bad2

Update README.md

bdcc158

k8s-ci-robot removed the size/M label Dec 11, 2019

k8s-ci-robot added the size/S label Dec 11, 2019

k8s-ci-robot added the lgtm label Dec 12, 2019

k8s-ci-robot added the approved label Dec 12, 2019

k8s-ci-robot added ok-to-test and removed needs-ok-to-test labels Dec 12, 2019

k8s-ci-robot merged commit 5a0c2f4 into kubeflow:master Dec 12, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[manifest] Added manifest for deploying on aws using s3 #2633

[manifest] Added manifest for deploying on aws using s3 #2633

eterna2 commented Nov 20, 2019 •

edited by jlewi

Loading

k8s-ci-robot commented Nov 20, 2019

Ark-kun Nov 21, 2019

Bobgy Nov 22, 2019

Ark-kun Nov 22, 2019

Ark-kun Nov 21, 2019

Bobgy left a comment

Bobgy Nov 22, 2019

Bobgy Nov 22, 2019

eterna2 Nov 22, 2019

eterna2 commented Nov 22, 2019

Jeffwan left a comment •

edited

Loading

Jeffwan Nov 22, 2019

eterna2 Nov 23, 2019

eterna2 commented Nov 23, 2019 •

edited

Loading

Bobgy Dec 5, 2019

Ark-kun Dec 10, 2019

Bobgy Dec 10, 2019

eterna2 commented Dec 11, 2019

Bobgy commented Dec 12, 2019

k8s-ci-robot commented Dec 12, 2019

Bobgy commented Dec 12, 2019

[manifest] Added manifest for deploying on aws using s3 #2633

[manifest] Added manifest for deploying on aws using s3 #2633

Conversation

eterna2 commented Nov 20, 2019 • edited by jlewi Loading

k8s-ci-robot commented Nov 20, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Bobgy left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eterna2 commented Nov 22, 2019

Jeffwan left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eterna2 commented Nov 23, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eterna2 commented Dec 11, 2019

Bobgy commented Dec 12, 2019

k8s-ci-robot commented Dec 12, 2019

Bobgy commented Dec 12, 2019

eterna2 commented Nov 20, 2019 •

edited by jlewi

Loading

Jeffwan left a comment •

edited

Loading

eterna2 commented Nov 23, 2019 •

edited

Loading