Skip to content

Commit

Permalink
add details on scaling with machinesets
Browse files Browse the repository at this point in the history
  • Loading branch information
@rvanderp3
rvanderp3 committed Oct 21, 2022
1 parent e83daa4 commit 82ce751
Showing 1 changed file with 73 additions and 79 deletions.
152 changes: 73 additions & 79 deletions enhancements/network/static-ip-addresses-vsphere.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -7,12 +7,15 @@ reviewers:
- elmiko - machine API
- padillon - installer
- jcpowermac
- cybertron
- zaneb
approvers:
- jspeed
- joelspeed
- padillon
- cybertron
api-approvers:
- jspeed
creation-date: yyyy-mm-dd
creation-date: 2022-10-21
last-updated: yyyy-mm-dd
tracking-link: [OCPPLAN-9654](https://issues.redhat.com/browse/OCPPLAN-9654)
see-also:
Expand Down Expand Up @@ -61,6 +64,7 @@ As an OpenShift engineer, I want to know if a IPI installation was performed so
### Non-Goals

- OpenShift will not be responsible for managing IP addresses.
- Multiple interfaces, IP configurations, and/or routes support is out of scope.

## Proposal

Expand Down Expand Up @@ -151,6 +155,8 @@ spec:
- preCreate hook
- Modify [vSphere v1beta1 machine API](https://github.com/openshift/api/blob/master/machine/v1beta1/types_vsphereprovider.go) to support nmstate configuration.


###### IP Configuration in the Machine resource
The machine API `VSphereMachineProviderSpec.Network` will be extended to include a new type called `NetworkConfig`.

~~~
Expand Down Expand Up @@ -178,7 +184,6 @@ type DnsResolver struct {
Config DnsResolverConfig `json:"config"`
}
// IPV4Addresses defines a slice of IPv4 addresses
type IPV4Addresses struct {
// Address slice of IPv4 addresses
Expand Down Expand Up @@ -235,6 +240,20 @@ type NetworkDeviceSpec struct {
}
~~~

An additional lifecycle hook will be added to enable a controller to augment a machine resource before
it is rendered in the backing infrastructure. This hook, a `preCreate` hook, will allow a controller
to augment the machine resource with IP configuration.

~~~
// PreCreate hooks prevent the machine from being created in the backing infrastructure.
// +listType=map
// +listMapKey=name
// +optional
PreCreate []LifecycleHook `json:"preCreate,omitempty"`
~~~

The `machine.machine.openshift.io` and `machinesets.machine.openshift.io` CRDs will be updated to add the `preCreate` hook.

### Workflow Description

#### Installation
Expand All @@ -255,64 +274,80 @@ type NetworkDeviceSpec struct {
- create machine resources containing specified IP configuration
5. The machine API will render the nodes with the specified IP configuration.

#### Scaling new Nodes with `machinesets`
1. OpenShift administrator configures machineset with `preCreate` lifecycle hook
~~~
apiVersion: machine.openshift.io/v1beta1
kind: MachineSet
metadata:
name: static-machineset-worker
namespace: openshift-machine-api
labels:
machine.openshift.io/cluster-api-cluster: cluster
spec:
replicas: 0
selector:
matchLabels:
machine.openshift.io/cluster-api-cluster: cluster
machine.openshift.io/cluster-api-machineset: static-machineset-worker
template:
metadata:
labels:
machine.openshift.io/cluster-api-cluster: cluster
machine.openshift.io/cluster-api-machine-role: worker
machine.openshift.io/cluster-api-machine-type: worker
machine.openshift.io/cluster-api-machineset: static-machineset-worker
spec:
lifecycleHooks:
preCreate:
- name: ipamController
owner: network-admin
~~~
2. OpenShift administrator or machine autoscaler scales `n` machines
3. Controller watches machine resources created with the expected lifecycle hook
4. Controller updates machine providerSpec with IP configuration
5. Controller sets `preTerminate` lifecycle hook
6. Controller removes `preCreate` lifecycle hook

On scale down, the controller will recognize a machine is being deleted and check for a `preTerminate`
lifecycle hook. If the hook exists, the controller will retrieve the IP address of the node from
nmstate and release the IP. The `preTerminate` should be removed regardless of if the IP address was
successfully released to prevent blocking the machine's deletion.

In this workflow, the controller is responsible for managing, claiming, and releasing IP addresses. An example
controller that implements this logic is [here](https://github.com/rvanderp3/machine-ipam-controller).


#### Variation [optional]

### API Extensions

The CRDs `machines.machine.openshift.io` and `machinesets.machine.openshift.io` will be modified to add a new lifecycle
hook called `preCreate`. When defined, the `preCreate` lifecycle hook will block the rendering of a machine in it's
backing infrastructure.

### Implementation Details/Notes/Constraints [optional]

### Risks and Mitigations


### Drawbacks

- Scaling nodes will become more complex. This will require the OpenShift administrator to integrate IP configuration
management with the scaling of machine API machine resources.
management to enable scaling of machine API machine resources.

- If a `machineset` is configured to specify the `preCreate` lifecycle hook, a controller must remove the hook before
machine creation will continue.

- `install-config.yaml` will grow in complexity.

## Design Details

### Open Questions [optional]

How can we support machinesets? There has been a suggesstion raised relative to creating a [`preCreate` lifecycle hook](https://github.com/openshift/machine-api-operator/blob/f7f326323f37ab91ec79430769635fc859f15b96/install/0000_30_machine-api-operator_03_machineset.crd.yaml#L244)
that an controller would maintain once a new machine is created.

### Test Plan

### Graduation Criteria

**Note:** *Section not required until targeted at a release.*

Define graduation milestones.

These may be defined in terms of API maturity, or as something else. Initial proposal
should keep this high-level with a focus on what signals will be looked at to
determine graduation.

Consider the following in developing the graduation criteria for this
enhancement:

- Maturity levels
- [`alpha`, `beta`, `stable` in upstream Kubernetes][maturity-levels]
- `Dev Preview`, `Tech Preview`, `GA` in OpenShift
- [Deprecation policy][deprecation-policy]

Clearly define what graduation means by either linking to the [API doc definition](https://kubernetes.io/docs/concepts/overview/kubernetes-api/#api-versioning),
or by redefining what graduation means.

In general, we try to use the same stages (alpha, beta, GA), regardless how the functionality is accessed.

[maturity-levels]: https://git.k8s.io/community/contributors/devel/sig-architecture/api_changes.md#alpha-beta-and-stable-versions
[deprecation-policy]: https://kubernetes.io/docs/reference/using-api/deprecation-policy/

**If this is a user facing change requiring new or updated documentation in [openshift-docs](https://github.com/openshift/openshift-docs/),
please be sure to include in the graduation criteria.**

**Examples**: These are generalized examples to consider, in addition
to the aforementioned [maturity levels][maturity-levels].

#### Dev Preview -> Tech Preview

- Ability to utilize the enhancement end to end
Expand All @@ -337,59 +372,18 @@ end to end tests.**

#### Removing a deprecated feature

- Announce deprecation and support policy of the existing feature
- Deprecate the feature

### Upgrade / Downgrade Strategy


### Version Skew Strategy

### Operational Aspects of API Extensions

Describe the impact of API extensions (mentioned in the proposal section, i.e. CRDs,
admission and conversion webhooks, aggregated API servers, finalizers) here in detail,
especially how they impact the OCP system architecture and operational aspects.

- For conversion/admission webhooks and aggregated apiservers: what are the SLIs (Service Level
Indicators) an administrator or support can use to determine the health of the API extensions

Examples (metrics, alerts, operator conditions)
- authentication-operator condition `APIServerDegraded=False`
- authentication-operator condition `APIServerAvailable=True`
- openshift-authentication/oauth-apiserver deployment and pods health

- What impact do these API extensions have on existing SLIs (e.g. scalability, API throughput,
API availability)

Examples:
- Adds 1s to every pod update in the system, slowing down pod scheduling by 5s on average.
- Fails creation of ConfigMap in the system when the webhook is not available.
- Adds a dependency on the SDN service network for all resources, risking API availability in case
of SDN issues.
- Expected use-cases require less than 1000 instances of the CRD, not impacting
general API throughput.

- How is the impact on existing SLIs to be measured and when (e.g. every release by QE, or
automatically in CI) and by whom (e.g. perf team; name the responsible person and let them review
this enhancement)

#### Failure Modes

- Describe the possible failure modes of the API extensions.
- Describe how a failure or behaviour of the extension will impact the overall cluster health
(e.g. which kube-controller-manager functionality will stop working), especially regarding
stability, availability, performance and security.
- Describe which OCP teams are likely to be called upon in case of escalation with one of the failure modes
and add them as reviewers to this enhancement.

#### Support Procedures

## Implementation History

Major milestones in the life cycle of a proposal should be tracked in `Implementation
History`.

## Alternatives

## Infrastructure Needed [optional]

0 comments on commit 82ce751

Please sign in to comment.