Adding cosi specific changes to prow.sh #9

brahmaroutu · 2020-11-22T05:03:16Z

prow.sh is modifies to use COSI specific config variables. It installs crds, controller and has hooks to run serial and parallel e2e tests.

brahmaroutu · 2020-11-22T05:21:00Z

/assign @xing-yang

xing-yang · 2020-11-22T22:22:37Z

release-tools/prow.sh

-configvar CSI_PROW_E2E_FOCUS_1_16 '^' "non-alpha, feature-tagged tests for Kubernetes = 1.16" # no tests to run, match nothing
-configvar CSI_PROW_E2E_FOCUS_LATEST '\[Feature:VolumeSnapshotDataSource\]' "non-alpha, feature-tagged tests for Kubernetes >= 1.17"
-configvar CSI_PROW_E2E_FOCUS "$(get_versioned_variable CSI_PROW_E2E_FOCUS "${csi_prow_kubernetes_version_suffix}")" "non-alpha, feature-tagged tests"
+configvar COSI_PROW_E2E_FOCUS "$(get_versioned_variable COSI_PROW_E2E_FOCUS "${cosi_prow_kubernetes_version_suffix}")" "non-alpha, feature-tagged tests"


I'm not sure if the tag is relevant for COSI. For CSI tests, tests tagged with feature gates are only run in clusters that support them. We can keep them for now and remove them if we are sure they are not needed.

xing-yang · 2020-11-22T22:27:02Z

release-tools/prow.sh

+# Installs all nessesary CRDs  
+install_crds() {
+  # Wait until cosi CRDs are in place.
+#  CRD_BASE_DIR="https://github.com/kubernetes-sigs/container-object-storage-interface-api/crds"


Remove this

xing-yang · 2020-11-22T22:30:44Z

release-tools/prow.sh

+install_crds() {
+  # Wait until cosi CRDs are in place.
+#  CRD_BASE_DIR="https://github.com/kubernetes-sigs/container-object-storage-interface-api/crds"
+  CRD_BASE_DIR="https://raw.githubusercontent.com/brahmaroutu/api/proper_crds/crds"


This should be from an official repo?
If you are still testing, can you mark your PR as "WIP"?

Fixed it to use the official site (now available). But I need to work with a different error,
"Required value: protected groups must have approval annotation "api-approved.kubernetes.io", see kubernetes/enhancements#1111"
I am implementing a workaround using sed to insert the annotation until we move these api group into k8s build.

I think you can use this PR "kubernetes-retired/container-object-storage-interface-api#2" for now and replace it with the approved PR in the future when it is officially approved by the API reviewer.

xing-yang · 2020-11-22T22:34:13Z

release-tools/prow.sh

+  # Wait until cosi CRDs are in place.
+#  CRD_BASE_DIR="https://github.com/kubernetes-sigs/container-object-storage-interface-api/crds"
+  CRD_BASE_DIR="https://raw.githubusercontent.com/brahmaroutu/api/proper_crds/crds"
+  kubectl apply -f "${CRD_BASE_DIR}/objectstorage.k8s.io_bucketclasses.yaml" --validate=false


The install location should be dynamic. If this is testing a PR at the API repo "https://github.com/kubernetes-sigs/container-object-storage-interface-api", it should read from the API repo. In other cases, it should either install from master or a stable branch of the API repo.

See this PR for example: kubernetes-csi/csi-release-tools#113

xing-yang · 2020-11-22T22:37:51Z

release-tools/prow.sh

+# Install controller and associated RBAC, retrying until the pod is running.
+install_controller() {
+  #kubectl apply -f "https://raw.githubusercontent.com/kubernetes-sigs/container-object-storage-interface-controller/${COSI_CONTROLLER_VERSION}/deploy/controller.yaml"
+  kubectl apply -f "https://raw.githubusercontent.com/container-object-storage-interface/cosi-controller-manager/master/samples/rbac.yaml"


See my comments earlier about install location should be dynamic.

xing-yang · 2020-11-22T22:38:10Z

release-tools/prow.sh

  done


-  kubectl apply -f "https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/${CSI_SNAPSHOTTER_VERSION}/deploy/kubernetes/snapshot-controller/setup-snapshot-controller.yaml"
+  #kubectl apply -f "https://raw.githubusercontent.com/kubernetes-sigs/container-object-storage-interface-controller/${COSI_CONTROLLER_VERSION}/deploy/rbac.yaml"
+  kubectl apply -f "https://raw.githubusercontent.com/brahmaroutu/cosi-controller-manager/test/samples/controller.yaml"


See my comments earlier about install location should be dynamic.

xing-yang · 2020-11-22T22:38:28Z

release-tools/prow.sh

-done
+#docker exec cosi-prow-control-plane docker image ls --format='{{.Repository}} {{.Tag}} {{.ID}}' | grep -e COSI -e hostpath | while read -r repo tag id; do
+#    echo "$repo" "$tag" "$(docker exec cosi-prow-control-plane docker image inspect --format='{{ index .Config.Labels "revision"}}' "$id")"
+#done


Remove if no longer needed.

xing-yang · 2020-11-22T22:39:31Z

release-tools/prow.sh

-                    fi
+            echo "Srini Collect Cluster Info"
+            collect_cluster_info 
+            echo "Srini Done Collect Cluster Info"


Looks like these are your temporary debug messages.
Can you mark this PR as WIP? Looks like you are still testing.

NicolasT · 2020-12-22T11:22:08Z

release-tools/prow.sh

+  CRD_BASE_DIR="https://raw.githubusercontent.com/kubernetes-sigs/container-object-storage-interface-api/master/crds"
+  kubectl_apply  "${CRD_BASE_DIR}/objectstorage.k8s.io_bucketclasses.yaml"
+  kubectl_apply  "${CRD_BASE_DIR}/objectstorage.k8s.io_bucketrequests.yaml"
+  kubectl_apply  "${CRD_BASE_DIR}/objectstorage.k8s.io_buckets.yaml"
+  kubectl_apply  "${CRD_BASE_DIR}/objectstorage.k8s.io_bucketaccessclasses.yaml"
+  kubectl_apply  "${CRD_BASE_DIR}/objectstorage.k8s.io_bucketaccessrequests.yaml"
+  kubectl_apply  "${CRD_BASE_DIR}/objectstorage.k8s.io_bucketaccesses.yaml"


Since kubernetes-retired/container-object-storage-interface-api#13 there's a kustomization.yaml in https://github.com/kubernetes-sigs/container-object-storage-interface-api/tree/master/crds wich allows to deploy the CRDs (including required labels) without exhaustively listing all of them here.

You could use, as an example, kustomize build "https://github.com/kubernetes-sigs/container-object-storage-interface-api/crds/?ref=master" | kubectl apply -f -.

I would make this effort in the subsequent PRs as the 'kustomize' require additional effort, kind clusters may not bu default have Kustomize.

brahmaroutu · 2021-01-05T21:28:15Z

@xing-yang please review, I have addressed your review comments. This runs fine but may require further changes as we introduce post submit jobs, e2e/integration testing and other cost components into the build

xing-yang · 2021-01-06T19:20:05Z

release-tools/prow.sh

            cmds="$(grep '^\s*CMDS\s*=' Makefile | sed -e 's/\s*CMDS\s*=//')"
            # Get the image that was just built (if any) from the
            # top-level Makefile CMDS variable and set the
            # deploy.sh env variables for it. We also need to
            # side-load those images into the cluster.
            for i in $cmds; do
                e=$(echo "$i" | tr '[:lower:]' '[:upper:]' | tr - _)
-                images="$images ${e}_REGISTRY=none ${e}_TAG=csiprow"
+                images="$images ${e}_REGISTRY=quay.io/containerobjectstorage ${e}_TAG=cosiprow"


Are you going to use quay.io to store images, not k8s.gcr.io?

Discussed at the review meeting. Will use quay.io for now and move to k8s.gcr.io in the future.

xing-yang · 2021-01-07T18:08:42Z

/lgtm
/approve

k8s-ci-robot · 2021-01-07T18:08:52Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: brahmaroutu, xing-yang

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [xing-yang]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Nov 22, 2020

k8s-ci-robot requested review from msau42 and saad-ali November 22, 2020 05:03

k8s-ci-robot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Nov 22, 2020

k8s-ci-robot assigned xing-yang Nov 22, 2020

xing-yang reviewed Nov 22, 2020

View reviewed changes

brahmaroutu force-pushed the update_prow branch from d6ba67a to 6457831 Compare November 23, 2020 03:54

brahmaroutu changed the title ~~Adding cosi specific changes to prow.sh~~ WIP. Adding cosi specific changes to prow.sh Nov 23, 2020

k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Nov 23, 2020

NicolasT reviewed Dec 22, 2020

View reviewed changes

Adding cosi specific changes to prow.sh

e4385e4

brahmaroutu force-pushed the update_prow branch from 6457831 to e4385e4 Compare January 5, 2021 21:24

brahmaroutu changed the title ~~WIP. Adding cosi specific changes to prow.sh~~ Adding cosi specific changes to prow.sh Jan 5, 2021

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jan 5, 2021

xing-yang reviewed Jan 6, 2021

View reviewed changes

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jan 7, 2021

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 7, 2021

k8s-ci-robot merged commit ef52359 into kubernetes-retired:master Jan 7, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding cosi specific changes to prow.sh #9

Adding cosi specific changes to prow.sh #9

brahmaroutu commented Nov 22, 2020

brahmaroutu commented Nov 22, 2020

xing-yang Nov 22, 2020

brahmaroutu Nov 23, 2020

xing-yang Nov 22, 2020

brahmaroutu Nov 23, 2020

xing-yang Nov 22, 2020

brahmaroutu Nov 23, 2020

xing-yang Nov 23, 2020

brahmaroutu Jan 5, 2021

xing-yang Nov 22, 2020

brahmaroutu Jan 5, 2021

xing-yang Nov 22, 2020

brahmaroutu Jan 5, 2021

xing-yang Nov 22, 2020

brahmaroutu Jan 5, 2021

xing-yang Nov 22, 2020

brahmaroutu Nov 23, 2020

xing-yang Nov 22, 2020

brahmaroutu Nov 23, 2020

NicolasT Dec 22, 2020

brahmaroutu Jan 5, 2021

brahmaroutu commented Jan 5, 2021

xing-yang Jan 6, 2021

xing-yang Jan 7, 2021

xing-yang commented Jan 7, 2021

k8s-ci-robot commented Jan 7, 2021

Adding cosi specific changes to prow.sh #9

Adding cosi specific changes to prow.sh #9

Conversation

brahmaroutu commented Nov 22, 2020

brahmaroutu commented Nov 22, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

brahmaroutu commented Jan 5, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xing-yang commented Jan 7, 2021

k8s-ci-robot commented Jan 7, 2021