Correct wiring of the subscriptions-ci pipeline and prompt for NVA fi…

…rewall username & password (Azure#285)

.pipelines/README.md

@@ -18,12 +18,16 @@ The following top-level pipelines are present in the `.pipelines/` repository fo
 | 2 | Platform Logging | `platform-logging.yml` | platform-logging-ci
 | 3 | Policy | `policy.yml` | policy-ci
 | 4 | Roles | `roles.yml` | roles-ci
-| 5 | Networking | `platform-connectivity-hub-nva.yml` | platform-connectivity-hub-nva-ci
-| 6 | Subscription | `subscription.yml` | subscription-ci
+| 5a | Networking (NVA) | `platform-connectivity-hub-nva.yml` | platform-connectivity-hub-nva-ci
+| 5b | Networking (Azure Firewall) | `platform-connectivity-hub-azfw-policy.yml` | platform-connectivity-hub-azfw-policy-ci
+| 5b | Networking (Azure Firewall) | `platform-connectivity-hub-azfw.yml` | platform-connectivity-hub-azfw-ci
+| 6 | Subscriptions | `subscriptions.yml` | subscriptions-ci
 
 These pipelines need to be run in the order specified. For example, the `Policy` pipeline is dependent on resources deployed by the `Platform Logging` pipeline. Think of it as a layered approach; once the layer is deployed, it only requires re-running if some configuration at that layer changes.
 
-In the default implementation, the `Management Groups`, `Platform Logging`, `Policy`, and `Roles` pipelines are run automatically (trigger) whenever a related code change is detected on the `main` branch. The `Networking` and `Subscription` pipelines do not run automatically (no trigger). This behavior can be changed by modifying the corresponding YAML pipeline definition files.
+There are two distinct `Networking` pipelines, each deploys the hub side of a hub & spoke network topology. The `Networking (NVA)` option is intended for environments with a Network Virtual Appliance, and the `Networking (Azure Firewall)` option is intended for environments using Azure Firewall.
+
+In the default implementation, the `Management Groups`, `Platform Logging`, `Policy`, and `Roles` pipelines are run automatically (trigger) whenever a related code change is detected on the `main` branch. The `Networking` and `Subscriptions` pipelines do not run automatically (no trigger). This behavior can be changed by modifying the corresponding YAML pipeline definition files.
 
 In the default implementation, the `Roles` and `Platform Logging` pipelines are run automatically after a successful run of the `Management Groups` pipeline, and the `Policy` pipeline is run automatically after a successful run of the `Platform Logging` pipeline. Again, this behavior can be changed by modifying the corresponding YAML pipeline definition files.
 

docs/archetypes/authoring-guide.md

@@ -116,12 +116,12 @@ Each archetype is intended to be self-contained and provides all deployment temp
 
 6. Create a JSON Schema definition for the archetype.  Consider using a tool such as [JSON to Jsonschema](https://jsonformatter.org/json-to-jsonschema) to generate the initial schema definition that you customize.  For all common features, you must reference the existing definitions for the types. See example: [schemas/latest/landingzones/lz-generic-subscription.json](../../schemas/latest/landingzones/lz-generic-subscription.json)
 
-7. Verify archetype deployment through `subscription-ci` Azure DevOps Pipeline.  More information on the pipeline can be found in [Azure DevOps Onboarding Guide](../onboarding/ado.md#step-8--configure-subscription-archetypes).
+7. Verify archetype deployment through `subscriptions-ci` Azure DevOps Pipeline.  More information on the pipeline can be found in [Azure DevOps Onboarding Guide](../onboarding/ado.md#step-8--configure-subscription-archetypes).
 
       - Create a subscription JSON Parameters file per [deployment instructions](#deployment-instructions).
       - Run the pipeline by providing the subscription guid
 
-    `subscription-ci` pipeline will automatically identify the archetype, the subscription and region based on the file name.  The JSON Schema is located by the archetype name and used for pre-deployment verification.  
+    `subscriptions-ci` pipeline will automatically identify the archetype, the subscription and region based on the file name.  The JSON Schema is located by the archetype name and used for pre-deployment verification.  
 
     Once verifications are complete, the pipeline will move the subscription to the target management group (based on the folder structure) and execute `main.bicep`.
 
@@ -175,7 +175,7 @@ An archetype can deploy & configure any number of Azure services.  For consisten
 - **Subscription Tags** - configures subscription tags
 - **Resource Tags** - configures tags on resource groups
 
-> **Log Analytics Workspace integration**: `main.bicep` must accept an input parameter named `logAnalyticsWorkspaceResourceId`.  This parameter is automatically set by `subscription-ci` Pipeline based on the environment configuration.  This parameter is used to link Microsoft Defender for Cloud to Log Analytics Workspace.
+> **Log Analytics Workspace integration**: `main.bicep` must accept an input parameter named `logAnalyticsWorkspaceResourceId`.  This parameter is automatically set by `subscriptions-ci` Pipeline based on the environment configuration.  This parameter is used to link Microsoft Defender for Cloud to Log Analytics Workspace.
 
 Input parameters for common features are:
 
@@ -276,7 +276,7 @@ As a result, we could either
 
 - have Azure deploy the archetype and fail on invalid inputs.  An administrator would have to deploy multiple times to fix all errors; or
 
-- attempt to detect invalid inputs as a pre-check in our `subscription-ci` pipeline.
+- attempt to detect invalid inputs as a pre-check in our `subscriptions-ci` pipeline.
 
 We chose to check the input parameters prior to deployment to identify misconfigurations faster.  Validations are performed using JSON Schema definitions.  These definitions are located in [schemas/latest/landingzones](../../schemas/latest/landingzones) folder.
 
@@ -341,7 +341,7 @@ These parameter files are located in [config/subscription](../../config/subscrip
 
 Immediate subfolder defines the environment which is based on Azure DevOps Organization (i.e. `CanadaESLZ`) & Git branch name (i.e. `main`), for example the subfolder will be called `CanadaESLZ-main`.  You can have many environments based on Git branch names such as `CanadaESLZ-feature-1`, `CanadaESLZ-dev`, etc.
 
-ARM parameter files are used by `subscription-ci` Azure DevOps Pipeline when configuring subscriptions with Azure resources.  The pipeline will detect environment, management group, subscription, deployment location and deployment parameters using the folder hierarchy, file name and file content.
+ARM parameter files are used by `subscriptions-ci` Azure DevOps Pipeline when configuring subscriptions with Azure resources.  The pipeline will detect environment, management group, subscription, deployment location and deployment parameters using the folder hierarchy, file name and file content.
 
 For example when the file path is:
 
@@ -395,7 +395,7 @@ There are two approaches for achieving uniquness:
 
     In this approach, you must ensure all management group ids are unique yourself.
 
-The `subscription-ci` management group detection logic is built to accommodate both scenarios.
+The `subscriptions-ci` management group detection logic is built to accommodate both scenarios.
 
 **To support approach #1:**
 
@@ -408,7 +408,7 @@ The `subscription-ci` management group detection logic is built to accommodate b
                     - DevTest
     ```
 
-- `subscription-ci` will then take the folder structure and concatenate it to create the management group id.  In this example `DevTest` management group id will be `pubsecLandingZonesDevTest`.
+- `subscriptions-ci` will then take the folder structure and concatenate it to create the management group id.  In this example `DevTest` management group id will be `pubsecLandingZonesDevTest`.
 
 **To support approach #2:**
 
@@ -421,4 +421,4 @@ The `subscription-ci` management group detection logic is built to accommodate b
             - DevTest
     ```
 
-- `subscription-ci` will then take the folder name as the structure (since there aren't any sub folders).  In this example `DevTest` management group id will be `DevTest`.
+- `subscriptions-ci` will then take the folder name as the structure (since there aren't any sub folders).  In this example `DevTest` management group id will be `DevTest`.

docs/architecture.md

@@ -556,7 +556,7 @@ Use the [Azure DevOps Pipelines](onboarding/azure-devops-pipelines.md) onboardin
 | Platform – Hub Networking using NVAs | platform-connectivity-hub-nva.yml | platform-connectivity-hub-nva-ci | Configures Hub Networking with Fortigate Firewalls. | spn-azure-platform-ops | None |
 | Platform – Hub Networking with Azure Firewall - Firewall Policy | platform-connectivity-hub-azfw-policy.yml | platform-connectivity-hub-azfw-policy-ci | Configures Azure Firewall Policy.  A policy contains firewall rules and firewall configuration such as enabling DNS Proxy.  Firewall policies can be updated independently of Azure Firewall. | spn-azure-platform-ops | None |
 | Platform – Hub Networking with Azure Firewall | platform-connectivity-hub-azfw.yml | platform-connectivity-hub-azfw-ci | Configures Hub Networking with Azure Firewall. | spn-azure-platform-ops | None |
-| Subscriptions | subscription.yml | subscription-ci | Configures a new subscription based on the archetype defined in the configuration file name. | spn-azure-platform-ops | None |
+| Subscriptions | subscriptions.yml | subscriptions-ci | Configures a new subscription based on the archetype defined in the configuration file name. | spn-azure-platform-ops | None |
 | Pull Request Validation | pull-request-check.yml | pull-request-validation-ci | Checks for breaking changes to Bicep templates & parameter schemas prior to merging the change to main branch.  This pipeline must be configured as a check for the `main` branch. | spn-azure-platform-ops | None |
 
 ### 9.4 Release Process

docs/onboarding/azure-devops-pipelines.md

@@ -1500,7 +1500,7 @@ In order to configure audit stream for Azure Monitor, identify the following inf
     5. Select Existing Azure Pipeline YAML file
     6. Identify the pipeline in `.pipelines/subscriptions.yml`.
     7. Save the pipeline (don't run it yet)
-    8. Rename the pipeline to `subscription-ci`
+    8. Rename the pipeline to `subscriptions-ci`
 
 2. Create a subscription configuration file (JSON)
 
@@ -1525,9 +1525,9 @@ In order to configure audit stream for Azure Monitor, identify the following inf
 3. Run the subscription pipeline
 
     1. In Azure DevOps, go to Pipelines
-    2. Select the `subscription-ci` pipeline and run it.
+    2. Select the `subscriptions-ci` pipeline and run it.
 
-       > The `subscription-ci` pipeline YAML is configured, by default, to **not** run automatically; you can change this if desired.
+       > The `subscriptions-ci` pipeline YAML is configured, by default, to **not** run automatically; you can change this if desired.
 
     3. In the Run Pipelines dialog window, enter the first 4 digits of your new subscription configuration file name (4 is usually enough of the GUID to uniquely identify the subscription) between the square brackets in the `subscriptions` parameter field. For example: `[802e]`.
 

docs/onboarding/azure-devops-scripts.md

@@ -227,7 +227,6 @@ Next, edit the newly created file, using the guidance in the following table.
 | DEVOPS_SE_NAME | Azure DevOps service endpoint name. | spn-azure-platform-ops
 | DEVOPS_SE_TEMPLATE | File name for the generated Azure DevOps service endpoint template JSON file. | service-endpoint.AzDevOpsOrg.json
 | DEVOPS_VARIABLES_GROUP_NAME | Azure DevOps variable group name. Leave this set to `firewall-secrets` as the YAML pipeline for networking is hard-coded to use this value. | firewall-secrets
-| DEVOPS_VARIABLES_VALUES | Specify values for the NVA firewall username and password in format `key=value key=value`. Replace `YourUsername` and `YourPassword` in the example with your values. DO NOT commit changes that include username and password plaintext values to your repository. | var-hubnetwork-nva-fwUsername=YourUserName var-hubnetwork-nva-fwPassword=YourPassword
 | DEVOPS_VARIABLES_ARE_SECRET | Indicates whether variables in the variable group are marked as secret. Possible values are `true` or `false`. Recommend using `true` unless you plan to reconfigure your variable group to use another secure source such as KeyVault. | true
 | DEVOPS_OUTPUT_DIR | Name of temporary folder for generated files. | .\output
 
@@ -261,7 +260,7 @@ Run the `create-pipelines.bat` script to create the landing zone pipelines:
 - platform-connectivity-hub-nva-ci
 - platform-connectivity-hub-azfw-ci
 - platform-connectivity-hub-azfw-policy-ci
-- subscription-ci
+- subscriptions-ci
 
 If you would rather perform these steps manually, detailed guidance is available in the following sections of the [Azure DevOps Pipelines Onboarding Guide](./azure-devops-pipelines.md):
 
@@ -296,7 +295,7 @@ Detailed guidance on these configuration requirements is available in the [Azure
 
 ### Run pipelines
 
-Run the `run-pipelines.bat` script to interactively run individual landing zone pipelines. Note that at present time the `subscription-ci` pipeline is not included in the list of runnable pipelines as the script requires additional work to enable that capability.
+Run the `run-pipelines.bat` script to interactively run individual landing zone pipelines. Note that at present time the `subscriptions-ci` pipeline is not included in the list of runnable pipelines as the script requires additional work to enable that capability.
 
 ### Clear environment variables used by scripts
 

scripts/onboarding/create-pipelines.bat

@@ -22,7 +22,7 @@ choice /C YN /M "Do you want to proceed?"
 if errorlevel 2 exit /b 0
 
 REM Process all pipeline definitions
-for %%N in (management-groups roles platform-logging policy platform-connectivity-hub-nva platform-connectivity-hub-azfw platform-connectivity-hub-azfw-policy subscription) do (
+for %%N in (management-groups roles platform-logging policy platform-connectivity-hub-nva platform-connectivity-hub-azfw platform-connectivity-hub-azfw-policy subscriptions) do (
 
     REM Check for pipeline existence
     set FOUND=

scripts/onboarding/create-variable-group.bat

@@ -14,7 +14,6 @@ echo.
 echo   DevOps Organization:          %DEVOPS_ORG%
 echo   DevOps Project:               %DEVOPS_PROJECT_NAME%
 echo   DevOps Variable Group:        %DEVOPS_VARIABLES_GROUP_NAME%
-echo   DevOps Variables:             %DEVOPS_VARIABLES_VALUES%
 echo   DevOps Variables are Secret:  %DEVOPS_VARIABLES_ARE_SECRET%
 echo.
 choice /C YN /M "Do you want to proceed?"
@@ -33,8 +32,18 @@ if defined ID (
 )
 
 REM Create the variable group
-echo Creating variable group [%DEVOPS_VARIABLES_GROUP_NAME%] with variables: %DEVOPS_VARIABLES_VALUES%...
-call az pipelines variable-group create --name %DEVOPS_VARIABLES_GROUP_NAME% --authorize true --query "[?name=='%DEVOPS_VARIABLES_GROUP_NAME%'].id | [0]" -o tsv --org %DEVOPS_ORG% --project %DEVOPS_PROJECT_NAME% --variables %DEVOPS_VARIABLES_VALUES%
+echo Enter NVA username and password to set variables in DevOps variable group [%DEVOPS_VARIABLES_GROUP_NAME%]
+echo.
+echo **********************************************************************
+echo  CAUTION: your input is not masked, i.e. it will be visible on-screen
+echo **********************************************************************
+echo.
+set /P NVA_USERNAME=Enter the user name for the NVA firewall: 
+set /P NVA_PASSWORD=Enter the password for the NVA firewall: 
+echo.
+
+echo Creating variable group [%DEVOPS_VARIABLES_GROUP_NAME%]...
+call az pipelines variable-group create --name %DEVOPS_VARIABLES_GROUP_NAME% --authorize true --query "[?name=='%DEVOPS_VARIABLES_GROUP_NAME%'].id | [0]" -o tsv --org %DEVOPS_ORG% --project %DEVOPS_PROJECT_NAME% --variables var-hubnetwork-nva-fwUsername=%NVA_USERNAME% var-hubnetwork-nva-fwPassword=%NVA_PASSWORD%
 echo.
 echo Variable group [%DEVOPS_VARIABLES_GROUP_NAME%] has been created.
 echo.
@@ -43,9 +52,17 @@ echo.
 echo RECOMMENDED that you use the Azure DevOps portal to restrict access to this
 echo   variable group to only the `platform-connectivity-hub-nva` pipeline.
 echo.
-echo RECOMMENDED that you DO NOT commit to your repository any changes made
-echo   to this file that include a plaintext username or password.
-echo.
 
 REM Set variables as secret in Azure DevOps if requested
-if "%DEVOPS_VARIABLES_ARE_SECRET%" == "true" call update-variable-group.bat true
+if "%DEVOPS_VARIABLES_ARE_SECRET%" == "true" (
+    echo.
+    echo Setting variables in Azure DevOps variable group [%DEVOPS_VARIABLES_GROUP_NAME%] as secret...
+    echo.
+    call update-variable-group.bat true
+) else (
+    echo.
+    echo **************************************************************************
+    echo  WARNING: NVA firewall variables are not marked as secret in Azure DevOps
+    echo **************************************************************************
+    echo.
+)

scripts/onboarding/set-variables.DevOpsOrgName.bat

@@ -47,10 +47,6 @@ set DEVOPS_SE_TEMPLATE=service-endpoint.DEVOPS-ORG-NAME.json
 REM Do not change this value (hard-coded in YAML pipeline definition)
 set DEVOPS_VARIABLES_GROUP_NAME=firewall-secrets
 
-REM Variables is a space-delimited key=value string. Provide values for
-REM 'var-hubnetwork-nva-fwUsername' and 'var-hubnetwork-nva-fwPassword'.
-set DEVOPS_VARIABLES_VALUES=var-hubnetwork-nva-fwUsername=YourUserName var-hubnetwork-nva-fwPassword=YourPassword
-
 REM Are variables in the firewall-secrets group marked as secret? 'true' or 'false'.
 set DEVOPS_VARIABLES_ARE_SECRET=true
 

scripts/onboarding/set-variables.ocag148outlook.bat

@@ -47,10 +47,6 @@ set DEVOPS_SE_TEMPLATE=service-endpoint.ocag148outlook.json
 REM Do not change this value (hard-coded in YAML pipeline definition)
 set DEVOPS_VARIABLES_GROUP_NAME=firewall-secrets
 
-REM Variables is a space-delimited key=value string. Provide values for
-REM 'var-hubnetwork-nva-fwUsername' and 'var-hubnetwork-nva-fwPassword'.
-set DEVOPS_VARIABLES_VALUES=var-hubnetwork-nva-fwUsername=YourUserName var-hubnetwork-nva-fwPassword=YourPassword
-
 REM Are variables in the firewall-secrets group marked as secret? 'true' or 'false'.
 set DEVOPS_VARIABLES_ARE_SECRET=true