Make an E2E test for manifests

[krishnadurai]

Since #1237 removes the erstwhile GCP cluster-based E2E test. We might need an E2E test to:

1. Ensure that dependencies within the manifest definitions are maintained and are deployed in order.
2. All required custom resources definitions are created before the CRs are applied
3. Ensure that common services defined in manifests remain functional

The bar for this E2E test would be:

1. The test is fast and reliable
2. That the test is catching breakages that would otherwise be missed

[issue-label-bot]

Issue-Label Bot is automatically applying the labels:

Label	Probability
kind/feature	0.92

Please mark this comment with 👍 or 👎 to give our bot feedback!
Links: app homepage, dashboard and code for this bot.

[issue-label-bot]

Issue-Label Bot is automatically applying the labels:

Label	Probability
area/engprod	0.79

Please mark this comment with 👍 or 👎 to give our bot feedback!
Links: app homepage, dashboard and code for this bot.

[issue-label-bot]

Issue-Label Bot is automatically applying the labels:

Label	Probability
area/engprod	0.79

Please mark this comment with 👍 or 👎 to give our bot feedback!
Links: app homepage, dashboard and code for this bot.

[jlewi]

Per my comment, I think the question is what tests should happen upstream vs. downstream?

For GCP our plan is to move the E2E tests downstream GoogleCloudPlatform/kubeflow-distribution#42. We are doing this because I think we need better separation of concerns.

The manifests themselves should largely be owned and maintained by the application OWNERs.

It seems like the areas you want to test are mostly about deployment which I think should be a downstream concern.

Deploying kubeflow seems like a two step process

kustomize build | kubectl apply 

Should testing reflect that? So tests for kubeflow/manifests should be focused on ensuring that kustomize build produces the correct output. Testing that the manifests get applied correctly seems like it should be the responsibility of platform/distribution owners. For example, ensuring manifests are applied in the correct order will depend on the deployment tooling used which will vary by platform.

Likewise ensuring that configuration is correct in terms of integrating with external systems seems like it should be the responsibility of platform owners. e.g. testing that on GCP pipelines is actually configured to talk to CloudSQL should happen downstream or at best the test should be tightly scoped so that unless you are directly changing the GCP configs the code doesn't get triggered.

Ensure that common services defined in manifests remain functional

What do you mean by common service?

[krishnadurai]

Most of my concerns are deployment concerns.

It may benefit us to have a conformance test with a neutral platform (like KinD) apart from the downstream tests with vendors. This will help us in having a deployment validity check in CI before the merge of a PR to master.

What do you mean by common service?

Services like cert-manager, Istio etc fall under this bracket.

[jtfogarty]

/priority p1

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in one week if no further activity occurs. Thank you for your contributions.

[PatrickXYS]

This is finished I think

/close

[k8s-ci-robot]

@PatrickXYS: Closing this issue.

In response to this:

This is finished I think

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.