AWS: Create ControlPlaneMachineSet CRDs

[rna-afk]

Creating an extra CRD asset for the ControlPlaneMachineSet which
is required for the machine api operator for more control over
the control-plane nodes that come up on AWS.

[patrickdillon]

/cc @JoelSpeed

[JoelSpeed]

The ControlPlaneMachineSet is different to the MachineSet. There should be a single ControlPlaneMachineSet, which has to be called cluster in the openshift-machine-api namespace.

Then the FailureDomains should contain information for all of the zones, not just a single zone. It is up to the ControlPlaneMachineSet operator to handle spreading the master machines across the failure domains

[JoelSpeed]

Looks good, can you please attach as a code block an example of a generated ControlPlaneMachineSet based on this PR? Then we can run that through our validation webhooks and double check that it is valid, looks like it should be good though

[rna-afk]

Fixed some issues and generated the CRD.

apiVersion: machine.openshift.io/v1
kind: ControlPlaneMachineSet
metadata:
  creationTimestamp: null
  labels:
    machine.openshift.io/cluster-api-cluster: anarayan-h5gwf
  name: cluster
  namespace: openshift-machine-api
spec:
  replicas: 3
  selector:
    matchLabels:
      machine.openshift.io/cluster-api-cluster: anarayan-h5gwf
      machine.openshift.io/cluster-api-machine-role: master
      machine.openshift.io/cluster-api-machine-type: master
  strategy: {}
  template:
    machineType: machines_v1beta1_machine_openshift_io
    machines_v1beta1_machine_openshift_io:
      failureDomains:
        aws:
        - placement:
            availabilityZone: us-east-1a
          subnet:
            filters:
            - name: tag:Name
              values:
              - anarayan-h5gwf-private-us-east-1a
            id: ""
            type: ID
        - placement:
            availabilityZone: us-east-1b
          subnet:
            filters:
            - name: tag:Name
              values:
              - anarayan-h5gwf-private-us-east-1b
            id: ""
            type: ID
        - placement:
            availabilityZone: us-east-1c
          subnet:
            filters:
            - name: tag:Name
              values:
              - anarayan-h5gwf-private-us-east-1c
            id: ""
            type: ID
        - placement:
            availabilityZone: us-east-1d
          subnet:
            filters:
            - name: tag:Name
              values:
              - anarayan-h5gwf-private-us-east-1d
            id: ""
            type: ID
        - placement:
            availabilityZone: us-east-1f
          subnet:
            filters:
            - name: tag:Name
              values:
              - anarayan-h5gwf-private-us-east-1f
            id: ""
            type: ID
        platform: AWS
      metadata:
        labels:
          machine.openshift.io/cluster-api-cluster: anarayan-h5gwf
          machine.openshift.io/cluster-api-machine-role: master
          machine.openshift.io/cluster-api-machine-type: master
      spec:
        lifecycleHooks: {}
        metadata: {}
        providerSpec:
          value:
            ami:
              id: ami-0b9c602d07818c309
            apiVersion: machine.openshift.io/v1beta1
            blockDevices:
            - ebs:
                encrypted: true
                iops: 0
                kmsKey:
                  arn: ""
                volumeSize: 120
                volumeType: gp3
            credentialsSecret:
              name: aws-cloud-credentials
            deviceIndex: 0
            iamInstanceProfile:
              id: anarayan-h5gwf-master-profile
            instanceType: m6i.xlarge
            kind: AWSMachineProviderConfig
            loadBalancers:
            - name: anarayan-h5gwf-int
              type: network
            - name: anarayan-h5gwf-ext
              type: network
            metadata:
              creationTimestamp: null
            metadataServiceOptions: {}
            placement:
              region: us-east-1
            securityGroups:
            - filters:
              - name: tag:Name
                values:
                - anarayan-h5gwf-master-sg
            subnet: {}
            tags:
            - name: kubernetes.io/cluster/anarayan-h5gwf
              value: owned
            userDataSecret:
              name: master-user-data
status: {}

[JoelSpeed]

- placement:
            availabilityZone: us-east-1a
          subnet:
            filters:
            - name: tag:Name
              values:
              - anarayan-h5gwf-private-us-east-1a
            id: ""
            type: ID

This is invalid, you can't have filters as a non nil entry when type is ID, you need to override the value to Filters for type if you are using the type, you must also nil the id field

[rna-afk]

Fixed it. PTAL new CRD.

apiVersion: machine.openshift.io/v1
kind: ControlPlaneMachineSet
metadata:
  creationTimestamp: null
  labels:
    machine.openshift.io/cluster-api-cluster: anarayan-fkc85
  name: cluster
  namespace: openshift-machine-api
spec:
  replicas: 3
  selector:
    matchLabels:
      machine.openshift.io/cluster-api-cluster: anarayan-fkc85
      machine.openshift.io/cluster-api-machine-role: master
      machine.openshift.io/cluster-api-machine-type: master
  strategy: {}
  template:
    machineType: machines_v1beta1_machine_openshift_io
    machines_v1beta1_machine_openshift_io:
      failureDomains:
        aws:
        - placement:
            availabilityZone: us-east-1a
          subnet:
            filters:
            - name: tag:Name
              values:
              - anarayan-fkc85-private-us-east-1a
            type: Filters
        - placement:
            availabilityZone: us-east-1b
          subnet:
            filters:
            - name: tag:Name
              values:
              - anarayan-fkc85-private-us-east-1b
            type: Filters
        - placement:
            availabilityZone: us-east-1c
          subnet:
            filters:
            - name: tag:Name
              values:
              - anarayan-fkc85-private-us-east-1c
            type: Filters
        - placement:
            availabilityZone: us-east-1d
          subnet:
            filters:
            - name: tag:Name
              values:
              - anarayan-fkc85-private-us-east-1d
            type: Filters
        - placement:
            availabilityZone: us-east-1f
          subnet:
            filters:
            - name: tag:Name
              values:
              - anarayan-fkc85-private-us-east-1f
            type: Filters
        platform: AWS
      metadata:
        labels:
          machine.openshift.io/cluster-api-cluster: anarayan-fkc85
          machine.openshift.io/cluster-api-machine-role: master
          machine.openshift.io/cluster-api-machine-type: master
      spec:
        lifecycleHooks: {}
        metadata: {}
        providerSpec:
          value:
            ami:
              id: ami-0b9c602d07818c309
            apiVersion: machine.openshift.io/v1beta1
            blockDevices:
            - ebs:
                encrypted: true
                iops: 0
                kmsKey:
                  arn: ""
                volumeSize: 120
                volumeType: gp3
            credentialsSecret:
              name: aws-cloud-credentials
            deviceIndex: 0
            iamInstanceProfile:
              id: anarayan-fkc85-master-profile
            instanceType: m6i.xlarge
            kind: AWSMachineProviderConfig
            loadBalancers:
            - name: anarayan-fkc85-int
              type: network
            - name: anarayan-fkc85-ext
              type: network
            metadata:
              creationTimestamp: null
            metadataServiceOptions: {}
            placement:
              region: us-east-1
            securityGroups:
            - filters:
              - name: tag:Name
                values:
                - anarayan-fkc85-master-sg
            subnet: {}
            tags:
            - name: kubernetes.io/cluster/anarayan-fkc85
              value: owned
            userDataSecret:
              name: master-user-data
status: {}

[JoelSpeed]

I think this looks ok but I'd like to quickly run this through our webhook validations to double check it passes, I'll have to create a cluster to do that so it may take some time

[jstuever]

/retest

[rna-afk]

/retest

[JoelSpeed]

So I gave this a test and hit highlighted an issue with one of the assumptions we made while developing the CPMS. This will work for any region with 3 failure domains but if there are more this won't work currently. I've created openshift/cluster-control-plane-machine-set-operator#73 to fix this, so we can merge this once this new PR merges

[JoelSpeed]

I'd like to hold off merging this until openshift/cluster-control-plane-machine-set-operator#80 merges, as we will then have the operator deployed in CI and be able to verify that this change doesn't break installations.

If we merge the other way round we may need to patch things up here later

[JoelSpeed]

/test e2e-aws
/test e2e-aws-imdsv2
/test e2e-aws-proxy
/test e2e-aws-single-node

I think everything I wanted to see merged before we merge this is done now so let's try a selection of AWS presubmits and see how badly they break 😉

[JoelSpeed]

@rna-afk I don't really understand how but all the AWS installs seem to be broken with some terraform issue right now, is that a result of this PR do you think?

[rna-afk]

/retest

[rna-afk]

/test e2e-aws
/test e2e-aws-imdsv2
/test e2e-aws-proxy
/test e2e-aws-single-node

[rna-afk]

Should work now. Was just a rogue pointer reference that I made.

[JoelSpeed]

/test e2e-aws
/test e2e-aws-proxy
/test e2e-aws-imdsv2

[JoelSpeed]

Something went wrong with the CI system there
/test e2e-aws
/test e2e-aws-proxy
/test e2e-aws-imdsv2

[rna-afk]

/test yaml-lint

[patrickdillon]

/approve
/skip

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: patrickdillon

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [patrickdillon]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[rna-afk]

/test gofmt
/test images
/test shellcheck
/test verify-vendor
/test tf-fmt
/test tf-lint
/test yaml-lint

[rna-afk]

/test shellcheck
/test tf-fmt
/test yaml-lint

[r4f4]

/lgtm

[r4f4]

/test okd-images

[rna-afk]

/test aro-unit
/test e2e-azure
/test e2e-gcp

[rna-afk]

/skip

[rna-afk]

/test e2e-azure
/test e2e-gcp

[rna-afk]

/test okd-unit

[openshift-ci]

@rna-afk: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-azurestack	14de63d815f2b9fb0d1dbf94f0e262f810daaeab	link	false	/test e2e-azurestack
ci/prow/okd-e2e-aws-ovn	779e0ed	link	false	/test okd-e2e-aws-ovn
ci/prow/e2e-metal-ipi	779e0ed	link	false	/test e2e-metal-ipi

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[openshift-ci-robot]

/retest-required

Remaining retests: 2 against base HEAD 0792ab3 and 8 for PR HEAD 779e0ed in total

[JoelSpeed]

Everything is in payload now so now reason from my side to hold this up anymore, let's try get this merged asap

[rna-afk]

/retest