Zone redundant NAT Gateways

[kon-angelo]

How to categorize this PR?

/area control-plane
/kind enhancement
/platform azure

What this PR does / why we need it:
This PR allows the deployment of shoots with a new network setup for Azure. The new setup is akin to other extensions where it is possible to have finer-grained control over the networking of each availability zone. The main motivation is to have zone redundant NAT Gateways. Prior to this PR we used a single subnet and thus a single NAT Gateway instance. Using the new setup each subnet can potentially have its own NAT gateway instance and separate public IPs.

In addition, there is a migration path from our current zoned setup to this "enhanced" zone setup. For this to be possible, we impose the constraint that the first zone specified in the infrastructure must match the CIDR range of the existing workers subnet. The main reason is to preserve the subnet of an existing cluster and hence prevent having to teardown all the existing VMs to recreate the network setup. Instead, by preserving the original subnet we can have a controlled node migration (via MCM) to the new machineClasses and ensure minimal downtime during the migration.

Which issue(s) this PR fixes:
Fixes #https://github.com/orgs/gardener/projects/7#card-49971123

Special notes for your reviewer:
Currently missing:

• documentation
• a new terraform azurerm provider.

Unfortunately there is a major bug with with TF provider for Azure that we encountered (link) - when creating multiple "association" resources, the terraform provider is permanently stuck due to locking issues. With the current (at the time of writing) version we use it is happening consistently. The "custom" version of the terraformer used in this PR is build based on a newer version (2.60) and while the occurrence rate has fallen, it is still around 50% according to my tests and I consider it a showstopper until fixed. The custom terraform image is there to facilitate reviewers and testers.

We have a PR with a potential fix waiting for review (hashicorp/terraform-provider-azurerm#12267).

Release note:

Azure provider now supports a new network setup that allows for zone redundant NAT Gateways.

[dkistner]

/assign

[kon-angelo]

/test

[dkistner]

Now also done the validation part.

After an intensive pair review on that on functional and code level I have only minor comments lefts. After checking and addressing them where necessary I think we are finally good with this PR.

[dkistner]

minor hint: You could use your helper IsUsingSingleSubnetLayout() here ;)

[dkistner]

Can you move this before the if else block?
Then we could eliminate the if else by returning after the helper.IsUsingSingleSubnetLayout if block is completed.

[gardener-robot]

@kon-angelo You need rebase this pull request with latest master branch. Please check.

[kon-angelo]

/test

[testmachinery]

Testrun: e2e-rz54w
Workflow: e2e-rz54w-wf
Phase: Failed

+---------------------+---------------------+--------+----------+
|        NAME         |        STEP         | PHASE  | DURATION |
+---------------------+---------------------+--------+----------+
| infrastructure-test | infrastructure-test | Failed | 22m25s   |
+---------------------+---------------------+--------+----------+

[kon-angelo]

/test

[testmachinery]

Testrun: e2e-jzbw7
Workflow: e2e-jzbw7-wf
Phase: Succeeded

+---------------------+---------------------+-----------+----------+
|        NAME         |        STEP         |   PHASE   | DURATION |
+---------------------+---------------------+-----------+----------+
| infrastructure-test | infrastructure-test | Succeeded | 33m17s   |
+---------------------+---------------------+-----------+----------+

[dkistner]

/lgtm
/unassign