Support updating VM networks and changing cluster network migration of vlanconfigs

[yaocw2020]

Related issue: harvester/harvester#3372
Solution:

• Add a mutating webhook to record the outdated cluster network and VLAN id by network.harvesterhci.io/last-clusternetwork and network.harvesterhci.io/last-vlan-id. Then controllers are able to know the outdated VLAN id and cluster network to do the cleanup process including deleting the related bridge configuration and route job.
• All related running VMs need to be stopped before updating VM networks and vlanconfigs. The validating webhook is responsible for the VM checking.
• Indicate the ready status of VM networks by the label network.harvesterhci.io/ready. Once the cluster network is unready(no vlanconfig in effect) , the nad under the cluster network will be unready and can't be used by running VMs.

[w13915984028]

Few questions, please double check, thanks.

[w13915984028]

another question, thanks.

[chrisho]

LGTM and have verified the updated VM network and VlanConfig migration.

apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  creationTimestamp: "2023-02-10T07:13:09Z"
  finalizers:
  - wrangler.cattle.io/harvester-network-nad-controller
  - wrangler.cattle.io/harvester-network-manager-nad-controller
  generation: 2
  labels:
    network.harvesterhci.io/clusternetwork: enp7s0
    network.harvesterhci.io/last-type: L2VlanNetwork
    network.harvesterhci.io/last-vlan-id: "1"
    network.harvesterhci.io/ready: "true"
    network.harvesterhci.io/type: UntaggedNetwork
  name: vlan1
  namespace: default
  resourceVersion: "1506062"
  uid: e24a0755-c5d9-47f2-bb6c-5ad16fc82fc5
spec:
  config: '{"cniVersion":"0.3.1","name":"vlan1","type":"bridge","bridge":"enp7s0-br","promiscMode":true,"ipam":{}}'

[w13915984028]

Please double check the last question, thanks.

[guangbochen]

LGTM, thanks.

[w13915984028]

LGTM, thanks.

[w13915984028]

Hint:
The general practice is to return the updated object when it is successful, let other OnChange be invoked with the updated object instead of old one.