feat (infra): [cluster-stamp] migrate arm json to bicep and upgrade k8s version

docs/deploy/09-aks-cluster.md

@@ -45,21 +45,21 @@ Now that the [hub-spoke network is provisioned](./08-cluster-networking.md), the
 
    ```bash
    # [This takes about 20 minutes to run.]
-   az deployment group create -g rg-bu0001a0005 -f cluster-stamp.json -p targetVnetResourceId=${RESOURCEID_VNET_CLUSTERSPOKE} clusterAdminAadGroupObjectId=${AADOBJECTID_GROUP_CLUSTERADMIN} k8sControlPlaneAuthorizationTenantId=${TENANTID_K8SRBAC} appGatewayListenerCertificate=${APP_GATEWAY_LISTENER_CERTIFICATE_BASE64} aksIngressControllerCertificate=${INGRESS_CONTROLLER_CERTIFICATE_BASE64} jumpBoxImageResourceId=${RESOURCEID_IMAGE_JUMPBOX} jumpBoxCloudInitAsBase64=${CLOUDINIT_BASE64}
+   az deployment group create -g rg-bu0001a0005 -f cluster-stamp.bicep -p targetVnetResourceId=${RESOURCEID_VNET_CLUSTERSPOKE} clusterAdminAadGroupObjectId=${AADOBJECTID_GROUP_CLUSTERADMIN} k8sControlPlaneAuthorizationTenantId=${TENANTID_K8SRBAC} appGatewayListenerCertificate=${APP_GATEWAY_LISTENER_CERTIFICATE_BASE64} aksIngressControllerCertificate=${INGRESS_CONTROLLER_CERTIFICATE_BASE64} jumpBoxImageResourceId=${RESOURCEID_IMAGE_JUMPBOX} jumpBoxCloudInitAsBase64=${CLOUDINIT_BASE64}
 
    # Or if you updated and wish to use the parameters file …
-   #az deployment group create -g rg-bu0001a0005 -f cluster-stamp.json -p "@azuredeploy.parameters.prod.json"
+   #az deployment group create -g rg-bu0001a0005 -f cluster-stamp.bicep -p "@azuredeploy.parameters.prod.json"
    ```
 
 1. Update cluster deployment with managed identity assignments.
 
-   **cluster-stamp.v2.json** is a _tiny_ evolution of the **cluster-stamp.json** ARM template you literally just deployed in the step above. Because we are using Azure AD Pod Identity v1 as a Microsoft-managed add-on, the mechanism to associate identities with the cluster is via ARM template instead of via Kubernetes manifest deployments (as you would do with the vanilla open source solution). However, due to a current limitation of the add-on, managed identities for Pod Managed Identities CANNOT be associated to the cluster when the cluster is first being created, only as an update to an existing cluster. So this deployment will re-deploy with the Pod Managed Identity association as the _only change_. Pod Managed Identity v2 is in development and it will support assignment at cluster-creation time. This implementation will evolve to use Azure AD Pod Identity v2 when available and we'll remove this step and add the assignment directly in `cluster-stamp.json`.
+   **cluster-stamp.v2.json** is a _tiny_ evolution of the **cluster-stamp.bicep** ARM template you literally just deployed in the step above. Because we are using Azure AD Pod Identity v1 as a Microsoft-managed add-on, the mechanism to associate identities with the cluster is via ARM template instead of via Kubernetes manifest deployments (as you would do with the vanilla open source solution). However, due to a current limitation of the add-on, managed identities for Pod Managed Identities CANNOT be associated to the cluster when the cluster is first being created, only as an update to an existing cluster. So this deployment will re-deploy with the Pod Managed Identity association as the _only change_. Pod Managed Identity v2 is in development and it will support assignment at cluster-creation time. This implementation will evolve to use Azure AD Pod Identity v2 when available and we'll remove this step and add the assignment directly in `cluster-stamp.bicep`.
 
-   > :eyes: If you're curious to see what changed in the cluster stamp, [view the diff](https://diffviewer.azureedge.net/?l=https://raw.githubusercontent.com/mspnp/aks-baseline-regulated/main/cluster-stamp.json&r=https://raw.githubusercontent.com/mspnp/aks-baseline-regulated/main/cluster-stamp.v2.json).
+   > :eyes: If you're curious to see what changed in the cluster stamp, [view the diff](https://diffviewer.azureedge.net/?l=https://raw.githubusercontent.com/mspnp/aks-baseline-regulated/main/cluster-stamp.bicep&r=https://raw.githubusercontent.com/mspnp/aks-baseline-regulated/main/cluster-stamp.v2.bicep).
 
    ```bash
    # [This takes about five minutes to run.]
-   az deployment group create -g rg-bu0001a0005 -f cluster-stamp.v2.json -p targetVnetResourceId=${RESOURCEID_VNET_CLUSTERSPOKE} clusterAdminAadGroupObjectId=${AADOBJECTID_GROUP_CLUSTERADMIN} k8sControlPlaneAuthorizationTenantId=${TENANTID_K8SRBAC} appGatewayListenerCertificate=${APP_GATEWAY_LISTENER_CERTIFICATE_BASE64} aksIngressControllerCertificate=${AKS_INGRESS_CONTROLLER_CERTIFICATE_BASE64} jumpBoxImageResourceId=${RESOURCEID_IMAGE_JUMPBOX} jumpBoxCloudInitAsBase64=${CLOUDINIT_BASE64}
+   az deployment group create -g rg-bu0001a0005 -f cluster-stamp.v2.bicep -p targetVnetResourceId=${RESOURCEID_VNET_CLUSTERSPOKE} clusterAdminAadGroupObjectId=${AADOBJECTID_GROUP_CLUSTERADMIN} k8sControlPlaneAuthorizationTenantId=${TENANTID_K8SRBAC} appGatewayListenerCertificate=${APP_GATEWAY_LISTENER_CERTIFICATE_BASE64} aksIngressControllerCertificate=${AKS_INGRESS_CONTROLLER_CERTIFICATE_BASE64} jumpBoxImageResourceId=${RESOURCEID_IMAGE_JUMPBOX} jumpBoxCloudInitAsBase64=${CLOUDINIT_BASE64}
 
    # Or if you used the parameters file …
    #az deployment group create -g rg-bu0001a0005 -f cluster-stamp.v2.json -p "@azuredeploy.parameters.prod.json"

modules/ensureClusterIdentityHasRbacToSelfManagedResources.bicep

@@ -0,0 +1,144 @@
+targetScope = 'resourceGroup'
+
+/*** PARAMETERS ***/
+
+@description('The AKS Control Plane Principal Id to be given with Network Contributor Role in different spoke subnets, so it can join VMSS and load balancers resources to them.')
+@minLength(36)
+@maxLength(36)
+param miClusterControlPlanePrincipalId string
+
+@description('The AKS Control Plane Principal Name to be used to create unique role assignments names.')
+@minLength(3)
+@maxLength(128)
+param clusterControlPlaneIdentityName string
+
+@description('The regional network spoke VNet Resource name that the cluster is being joined to, so it can be used to discover subnets during role assignments.')
+@minLength(1)
+param vnetSpokeName string
+
+@allowed([
+  'australiaeast'
+  'canadacentral'
+  'centralus'
+  'eastus'
+  'eastus2'
+  'westus2'
+  'francecentral'
+  'germanywestcentral'
+  'northeurope'
+  'southafricanorth'
+  'southcentralus'
+  'uksouth'
+  'westeurope'
+  'japaneast'
+  'southeastasia'
+])
+@description('AKS Service, Node Pools, and supporting services (KeyVault, App Gateway, etc) region. This needs to be the same region as the vnet provided in these parameters.')
+@minLength(4)
+param location string
+
+/*** EXISTING SUBSCRIPTION RESOURCES ***/
+
+resource networkContributorRole 'Microsoft.Authorization/roleDefinitions@2022-04-01' existing = {
+  name: '4d97b98b-1d4f-4787-a291-c67834d212e7'
+  scope: subscription()
+}
+
+resource dnsZoneContributorRole 'Microsoft.Authorization/roleDefinitions@2022-04-01' existing = {
+  name: 'b12aa53e-6015-4669-85d0-8515ebb3ae7f'
+  scope: subscription()
+}
+
+/*** EXISTING SPOKE RESOURCES ***/
+
+resource pdzMc 'Microsoft.Network/privateDnsZones@2020-06-01' existing = {
+  name: 'privatelink.${location}.azmk8s.io'
+}
+
+resource vnetSpoke 'Microsoft.Network/virtualNetworks@2022-01-01' existing = {
+  name: vnetSpokeName
+
+  resource snetClusterSystemNodePools 'subnets' existing = {
+    name: 'snet-cluster-systemnodepool'
+  }
+
+  resource snetClusterInScopeNodePools 'subnets' existing = {
+    name: 'snet-cluster-inscopenodepools'
+  }
+
+  resource snetClusterOutofScopeNodePools 'subnets' existing = {
+    name: 'snet-cluster-outofscopenodepools'
+  }
+
+  resource snetClusterIngressServices 'subnets' existing = {
+    name: 'snet-cluster-ingressservices'
+  }
+}
+
+/*** RESOURCES ***/
+
+resource vnetMiClusterControlPlaneDnsZoneContributorRole_roleAssignment 'Microsoft.Authorization/roleAssignments@2020-04-01-preview' = {
+  scope: vnetSpoke
+  name: guid(vnetSpoke.id, dnsZoneContributorRole.id, clusterControlPlaneIdentityName)
+  properties: {
+    roleDefinitionId: dnsZoneContributorRole.id
+    description: 'Allows cluster identity to attach custom DNS zone with Private Link information to this virtual network.'
+    principalId: miClusterControlPlanePrincipalId
+    principalType: 'ServicePrincipal'
+  }
+}
+
+resource snetSystemNodePoolSubnetMiClusterControlPlaneNetworkContributorRole_roleAssignment 'Microsoft.Authorization/roleAssignments@2020-04-01-preview' = {
+  scope: vnetSpoke::snetClusterSystemNodePools
+  name: guid(vnetSpoke::snetClusterSystemNodePools.id, networkContributorRole.id, clusterControlPlaneIdentityName)
+  properties: {
+    roleDefinitionId: networkContributorRole.id
+    description: 'Allows cluster identity to join the nodepool vmss resources to this subnet.'
+    principalId: miClusterControlPlanePrincipalId
+    principalType: 'ServicePrincipal'
+  }
+}
+
+resource snetInScopeNodePoolSubnetsnetSystemNodePoolSubnetMiClusterControlPlaneNetworkContributorRole_roleAssignment 'Microsoft.Authorization/roleAssignments@2020-04-01-preview' = {
+  scope: vnetSpoke::snetClusterInScopeNodePools
+  name: guid(vnetSpoke::snetClusterInScopeNodePools.id, networkContributorRole.id, clusterControlPlaneIdentityName)
+  properties: {
+    roleDefinitionId: networkContributorRole.id
+    description: 'Allows cluster identity to join the nodepool vmss resources to this subnet.'
+    principalId: miClusterControlPlanePrincipalId
+    principalType: 'ServicePrincipal'
+  }
+}
+
+resource snetOutOfScopeNodePoolSubnetMiClusterControlPlaneNetworkContributorRole_roleAssignment 'Microsoft.Authorization/roleAssignments@2020-04-01-preview' = {
+  scope: vnetSpoke::snetClusterOutofScopeNodePools
+  name: guid(vnetSpoke::snetClusterOutofScopeNodePools.id, networkContributorRole.id, clusterControlPlaneIdentityName)
+  properties: {
+    roleDefinitionId: networkContributorRole.id
+    description: 'Allows cluster identity to join the nodepool vmss resources to this subnet.'
+    principalId: miClusterControlPlanePrincipalId
+    principalType: 'ServicePrincipal'
+  }
+}
+
+resource snetIngressServicesSubnetMiClusterControlPlaneNetworkContributorRole_roleAssignment 'Microsoft.Authorization/roleAssignments@2020-04-01-preview' = {
+  scope: vnetSpoke::snetClusterIngressServices
+  name: guid(vnetSpoke::snetClusterIngressServices.id, networkContributorRole.id, clusterControlPlaneIdentityName)
+  properties: {
+    roleDefinitionId: networkContributorRole.id
+    description: 'Allows cluster identity to join load balancers (ingress resources) to this subnet.'
+    principalId: miClusterControlPlanePrincipalId
+    principalType: 'ServicePrincipal'
+  }
+}
+
+resource pdzMcPrivatelinkAzmk8sIoMiClusterControlPlaneDnsZoneContributorRole_roleAssignment 'Microsoft.Authorization/roleAssignments@2020-04-01-preview' = {
+  scope: pdzMc
+  name: guid(pdzMc.id, dnsZoneContributorRole.id, clusterControlPlaneIdentityName)
+  properties: {
+    roleDefinitionId: dnsZoneContributorRole.id
+    description: 'Allows cluster identity to manage zone Entries for cluster\'s Private Link configuration.'
+    principalId: miClusterControlPlanePrincipalId
+    principalType: 'ServicePrincipal'
+  }
+}

networking/hub-region.v2.bicep

@@ -892,7 +892,7 @@ resource hubFirewall 'Microsoft.Network/azureFirewalls@2021-05-01' = {
               name: 'az-login'
               description: 'Allow jumpboxes to perform az login.'
               sourceIpGroups: [
-                aks_ipgroup.id
+                aksJumpbox_ipgroup.id
               ]
               protocols: [
                 {
@@ -909,7 +909,7 @@ resource hubFirewall 'Microsoft.Network/azureFirewalls@2021-05-01' = {
               name: 'az-management-api'
               description: 'Allow jumpboxes to communicate with Azure management APIs.'
               sourceIpGroups: [
-                aks_ipgroup.id
+                aksJumpbox_ipgroup.id
               ]
               protocols: [
                 {
@@ -926,7 +926,7 @@ resource hubFirewall 'Microsoft.Network/azureFirewalls@2021-05-01' = {
               name: 'az-cli-extensions'
               description: 'Allow jumpboxes query az cli status and download extensions'
               sourceIpGroups: [
-                aks_ipgroup.id
+                aksJumpbox_ipgroup.id
               ]
               protocols: [
                 {
@@ -946,7 +946,7 @@ resource hubFirewall 'Microsoft.Network/azureFirewalls@2021-05-01' = {
               name: 'github'
               description: 'Allow pulling things down from GitHub. [Only a requirement of this walkthrough because we deploy some manifests that you clone from your repo.]'
               sourceIpGroups: [
-                aks_ipgroup.id
+                aksJumpbox_ipgroup.id
               ]
               protocols: [
                 {
@@ -964,7 +964,7 @@ resource hubFirewall 'Microsoft.Network/azureFirewalls@2021-05-01' = {
               name: 'azure-monitor-addon'
               description: 'Required for Azure Monitor Extension on Jumpbox.'
               sourceIpGroups: [
-                aks_ipgroup.id
+                aksJumpbox_ipgroup.id
               ]
               protocols: [
                 {

networking/spoke-BU0001A0005-01.bicep

@@ -862,7 +862,7 @@ resource aksPrivateDnsZones_virtualNetworkLink_toClusterVNet 'Microsoft.Network/
     location: 'global'
     properties: {
         virtualNetwork: {
-            id: clusterVNet.id
+            id: hubVnetResourceId
         }
         registrationEnabled: false
     }