k8s 1.21: External LoadBalancer failures for Windows + Availability Set nodes

[jackfrancis]

Describe the bug

Our upgrade tests tell us that after running aks-engine upgrade against a cluster using availability set nodes (i.e., VMs, not VMSS), creating an external load balancer fails. From controller-manager logs:

E0205 20:32:49.145484       1 azure_loadbalancer.go:189] reconcileLoadBalancer(default/ingress-nginx-elb) failed: invalid ip config ID /subscriptions/aa3d3369-e814-4495-899d-d31e8d7d09ce/resourceGroups/kubernetes-westus2-80684/providers/Microsoft.Network/networkInterfaces/1598k8s01nic-1/ipConfigurations/ipconfig1
E0205 20:32:49.145704       1 controller.go:254] error processing service default/ingress-nginx-elb (will retry): failed to ensure load balancer: invalid ip config ID /subscriptions/aa3d3369-e814-4495-899d-d31e8d7d09ce/resourceGroups/kubernetes-westus2-80684/providers/Microsoft.Network/networkInterfaces/1598k8s01nic-1/ipConfigurations/ipconfig1
I0205 20:32:49.146435       1 event.go:291] "Event occurred" object="default/ingress-nginx-elb" kind="Service" apiVersion="v1" type="Warning" reason="SyncLoadBalancerFailed" message="Error syncing load balancer: failed to ensure load balancer: invalid ip config ID /subscriptions/aa3d3369-e814-4495-899d-d31e8d7d09ce/resourceGroups/kubernetes-westus2-80684/providers/Microsoft.Network/networkInterfaces/1598k8s01nic-1/ipConfigurations/ipconfig1"

Steps To Reproduce

Expected behavior

AKS Engine version

Kubernetes version

Additional context

[jackfrancis]

This is not an upgrade issue, this is an upstream k8s issue:

kubernetes-sigs/cloud-provider-azure#505