Kubetest2 - terraform support

[rifelpet]

This adds support for --target terraform and applying/destroying the cluster using terraform.

The specified terraform version is downloaded and unzipped each run, ideally we could cache this or use tfenv or something.

[rifelpet]

/test pull-kops-e2e-kubernetes-aws-kubetest2

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: rifelpet

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [rifelpet]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[rifelpet]

/test pull-kops-e2e-kubernetes-aws-kubetest2

[rifelpet]

/test pull-kops-e2e-kubernetes-aws-kubetest2

[rifelpet]

/test pull-kops-e2e-kubernetes-aws-kubetest2

[rifelpet]

/test pull-kops-e2e-kubernetes-aws-kubetest2

[hakman]

/retest

[rifelpet]

I backed out the makefile update so this should be ready for review now.

I noticed an interesting behavior that I'm hoping is the source of some of our test job problems...

When I was porting over the kops commands from kubetest1 to kubetest2 I copied the validate cluster command as-is and didn't copy the node readiness loop that ensures nodes arent flapping their readiness:

https://github.com/kubernetes/test-infra/blob/dee9d4bff3ffb5130c88dd3c122ec3c3e4f95272/kubetest/kops.go#L497-L500

We built that logic into validate cluster but didnt update kubetest1 to use it instead of its own node readiness check.

The k8s e2e test framework has a --num-nodes flag that is used in some tests and defaults to the number of Ready and Schedulable nodes fetched at startup.

Kops not waiting for nodes to stop flapping their readiness caused the e2e framework to incorrectly detect the number of nodes, and some tests were being skipped as a result. See Line 7335 here. I'm hoping the --count 10 on kops validate cluster will fix this and hopefully some other test related issues.

[hakman]

/lgtm

[hakman]

/lgtm