Merge pull request #124 from Azure/feature/v6.1.0

Multiple small improvements and bug fixes for v6.1.0
Azure · Jan 23, 2023 · 459a832 · 459a832
2 parents 8398485 + f1623f9
commit 459a832
Show file tree

Hide file tree

Showing 5 changed files with 134 additions and 73 deletions.
diff --git a/Docs/breaking-changes-v6.0.md b/Docs/breaking-changes-v6.0.md
@@ -2,63 +2,114 @@
 
 **On this page**
 
-* [Why and what](#why-and-what)
-* [Upgrade path](#upgrade-path)
+* [Changed az cli to Az PowerShell Modules](#changed-az-cli-to-az-powershell-modules)
+* [Recommendation to Simplify GitHub Flow](#recommendation-to-simplify-github-flow)
+* [Renamed `Definitions` Folders](#renamed-definitions-folders)
+* [Replaced `-SuppressDelete` Switch with Desired State Handling](#replaced--suppressdelete-switch-with-desired-state-handling)
+* [Replaced `-IncludeResourceGroups` Switch with Desired State Handling](#replaced--includeresourcegroups-switch-with-desired-state-handling)
+* [Multiple Changes in `global-settings.jsonc`](#multiple-changes-in-global-settingsjsonc)
+* [Centralized Documentation in Docs folder](#centralized-documentation-in-docs-folder)
+* [Microsoft Breaking Change in Implementation of  `Get-AzRoleAssignment`](#microsoft-breaking-change-in-implementation-of--get-azroleassignment)
 * [Reading List](#reading-list)
 
-## Why and what
-
-We had make breaking changes to accommodate new features and improve speed. We decided to break all the items we had envisioned in one update.
-
-| Reason | Change | Impact |
-| :----- | :----- | :----- |
-| Remove requirement to have a default subscription. | `"defaultSubscriptionId"` field removed from global-settings.jsonc | Modify global-settings.jsonc |
-| Simplify and clarify `"rootScope"` definition. | Replace `"rootScope": { "ManagementGroupName": "my-mg"}` with `"deploymentRootScope": "/providers/Microsoft.Management/managementGroups/my-mg"` | Modify global-settings.jsonc |
-| Support multiple EPAC (and other PaC) solutions to manage Policy. | Add `"pacOwnerId": "e6581a31-51a3-4dc6-806d-2541dc251d31"` | Modify global-settings.jsonc |
- Support "brownfield" scenarios with smarter more granular approach. | Remove the command line switch `-noDelete`. <br/><br/>Add (optional) element within each pacEnvironment `"desiredState": { "strategy": "ownedOnly" }`.  | Remove switch in CI/CD pipelines. <br/><br/>Modify global-settings.jsonc |
- | Simplify the command line and increase granularity for resource group handling. | Remove the command line switch `-includeResourceGroups`. <br/><br/>Set `includeResourceGroups` to `true` in `desiredState` | Remove switch in CI/CD pipelines. <br/><br/>Modify global-settings.jsonc |
- | Increase execution speed and pipeline uniformity. We replaced az cli usage with faster Resource Graph queries and AZ PowerShell Modules. Additionally we simplified the cmdlets  naming. | Pipeline task use `task: AzurePowerShell@5`. <br/><br/>Modify script names and parameters. | Modify pipeline definition |
- | Rename definition folders to match Microsoft's standard naming in our Policy repo on GitHub. | Rename folders within `Definitions` folder (see below) | Change folder names |
- | Microsoft has deprecated Azure AD Graph API. It has been replaced with Microsoft Graph API impacting service connection setup | Add `MS Graph` [permissions](ci-cd-pipeline.md#ms-graph-permissions) for the pipeline service connections | Service Principal Permissions |
- | Centralized documentation files into Docs folder. | Readme.md files in `Definitions` folders are no longer used or updated, | Remove deprecated files from `Definitions` folders. They have been moved to the docs folder. | Remove the legacy `readme.md` files to avoid confusion.
-
-## Upgrade path
-
-For details consult the above table and the newly updated samples in StarterKit.
-
-* Modify global-settings.jsonc (look at the example in the StarterKit)
-  * pacOwnerId
-  * Remove defaultSubscriptionId
-  * Change rootScope to deploymentRootScope
-  * If you used the switch parameters `SuppressDelete`, add desiredState strategy as ownedOnly
-* Rename the definition folders to
-  * policyDefinitions
-  * policySetDefinitions
-  * policyAssignments
-  * policyExemptions
-  * policyDocumentations
-* Remove `README.md` files from all `Definitions` folders.
-* Remove `README.md` files from `Pipeline` folders.
- Fix the pipeline for
-  * Changed command line arguments
-  * Change command names
-    * `Build-DeploymentPlans.ps1` to `Build-DeploymentPlans.ps1`
-    * `Deploy-PolicyPlan.ps1` to `Deploy-PolicyPlan.ps1`
-    * `Deploy-RolesPlan.ps1` to `Deploy-RolesPlan.ps1`
-  * Change usage of `task: AzureCLI@2` to `task: AzurePowerShell@5`
-    * Add `azurePowerShellVersion: LatestVersion` (or `OtherVersion` with your preferred version at 6.3.1 or greater)
-    * If `OtherVersion` is used, see the documentation below to see how to set `preferredAzurePowerShellVersion`
-    * [AzurePowerShell@5 Syntax](https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/reference/azure-powershell-v5?view=azure-pipelines)
-  * Fix the artifact up/downloads by replacing the `publish` and `artifact` line items with those listed below
-
-```yml
-- publish: "$(PAC_OUTPUT_FOLDER)/plans-$(pacEnvironmentSelector)"
-  artifact: "plans-$(pacEnvironmentSelector)"
-  condition: and(succeeded(), or(eq(variables['planStep.deployPolicyChanges'], 'yes'), eq(variables['planStep.deployRoleChanges'], 'yes')))
+## Changed az cli to Az PowerShell Modules
+
+To increase execution speed and pipeline uniformity:
+
+* Replaced az cli usage with faster Resource Graph queries and AZ PowerShell Modules.
+* Simplified the cmdlets naming.
+* Simplified parameters
+* Removed -SuppressDeletes flag
+* Changed handling of plan files
+* Support to write pipeline variables for GitLab
+
+Change pipeline definition:
+
+* Change usage of task: `AzureCLI@2` to task: `AzurePowerShell@5`. Use `-devOpsType "ado"` for Azure DevOps or `-devOpsType "gitlab"` for Gitlab pipelines.
+
+``` yaml
+    - task: AzurePowerShell@5
+      name: planStep
+      displayName: Plan
+      inputs:
+        azureSubscription: $(devServiceConnection)
+        pwsh: true
+        azurePowerShellVersion: LatestVersion
+        ScriptPath: "Scripts/Deploy/Build-DeploymentPlans.ps1"
+        ScriptArguments:
+          -pacEnvironmentSelector $(pacEnvironmentSelector) `
+          -devOpsType "ado" `
+          -InformationAction Continue
+```
+
+* Changed command line arguments as needed
+* Change command names in pipeline definition
+  * `Build-AzPoliciesInitiativesAssignmentsPlan.ps1` to `Build-DeploymentPlans.ps1`
+  * `Deploy-AzPoliciesInitiativesAssignmentsFromPlan.ps1` to `Deploy-PolicyPlan.ps1`
+  * `Set-AzPolicyRolesFromPlan.ps1` to `Deploy-RolesPlan.ps1`
+* Fix the artifact up/downloads occurrences by replacing the publish and artifact line items with:
+
+``` yaml
+    - publish: "$(PAC_OUTPUT_FOLDER)/plans-$(pacEnvironmentSelector)"
+      artifact: "plans-$(pacEnvironmentSelector)"
+      condition: and(succeeded(), or(eq(variables['planStep.deployPolicyChanges'], 'yes'), eq(variables['planStep.deployRoleChanges'], 'yes')))
 ```
 
-* If you're using Azure DevOps pipelines add parameter `-devOpsType "ado"` to `Build-DeploymentPlans.ps1`
-* Add required `MS Graph` [permissions](ci-cd-pipeline.md#ms-graph-permissions) for the pipeline service connections.
+## Recommendation to Simplify GitHub Flow
+
+We have found that the additional test environment after a Pull Request merge does not lead to finding problems; therefore, we removed that stage from the starter kit pipelines as seen in our [CI/CD Pipeline documentation](ci-cd-pipeline.md#simplified-github-flow-for-policy-as-code).
+
+## Renamed `Definitions` Folders
+
+Renamed definition folders to match Microsoft's standard naming in our Policy repo on GitHub. Rename the folders in your repo to:
+
+* policyDefinitions
+* policySetDefinitions
+* policyAssignments
+* policyExemptions
+* policyDocumentations
+
+## Replaced `-SuppressDelete` Switch with Desired State Handling
+
+As part of the support for [multiple EPAC (and other PaC) solutions to manage Policy in a tenant(s)](desired-state-strategy.md), we changed our approach to "brownfield" scenarios. The setting has moved to `global-settings.jsonc`.
+
+Remove the command line switch `-SuppressDelete` in the pipeline and the `brownfield` variable. The equivalent in `global-settings.jsonc` is:
+
+``` json
+"desiredState":
+{
+  "strategy": "ownedOnly"
+}
+```
+
+## Replaced `-IncludeResourceGroups` Switch with Desired State Handling
+
+As part of the support for [multiple EPAC (and other PaC) solutions to manage Policy in a tenant(s)](desired-state-strategy.md), we changed our approach to including resource groups in desired state. Without any modifications, Resource Group level assignments are not managed by EPAC to preserve previous behavior.
+
+Remove the command line switch `-IncludeResourceGroups` in the pipeline. The equivalent in `global-settings.jsonc` is:
+
+``` json
+"desiredState": {
+    "includeResourceGroups": true,
+}
+```
+
+## Multiple Changes in `global-settings.jsonc`
+
+* Simplify and clarify `"rootScope"` definition by replaceing `"rootScope": { "ManagementGroupName": "my-mg"}` with `"deploymentRootScope": "/providers/Microsoft.Management/managementGroups/my-mg"`.
+* Removed requirement to have a default subscription. Remove `"defaultSubscriptionId"` element from `global-settings.jsonc`.
+* Support for multiple EPAC (and other PaC) solutions to manage Policy. Add required `"pacOwnerId": "e6581a31-51a3-4dc6-806d-2541dc251d31"`.
+* Add element for [desired state handling](desired-state-strategy.md) as needed.
+
+## Centralized Documentation in Docs folder
+
+Instead of README.md files in multiple folders, move all content from `README.md` files not at the solution root to the `Docs` folder.
+
+Remove `README.md` files in folders (and subfolders) `Pipeline`, `Definitions`, and `Scripts`.
+
+## Microsoft Breaking Change in Implementation of  `Get-AzRoleAssignment`
+
+The implementation was changed from Azure AD to MS Graph API impacting the roles requirements for the cmdlet. This changed the implementation of `New-AzPolicyReaderRole.ps1`. Add required `MS Graph` [permissions for the pipeline service connections](ci-cd-pipeline.md#ms-graph-permissions).
 
 ## Reading List
 

diff --git a/Docs/operating-environment.md b/Docs/operating-environment.md
@@ -15,7 +15,7 @@ Your operating environment will include two repos, a runner, and at least one de
 
 * PowerShell 7.2 or later, 7.3.1 (latest) recommended
 * PowerShell Modules
-  * Az required 9.3.1 or later
+  * Az required 9.3.0 or later - **9.2.x has a bug which causes EPAC to fail**
   * ImportExcel (required only if using Excel functionality)
 * Git latest version
 

diff --git a/README.md b/README.md
@@ -14,6 +14,16 @@
 > ---
 > ---
 
+> ---
+> ---
+>
+> **Az PowerShell Module 9.2.x has a known issue (bug)**
+>
+> This bug causes multiple failures of EPAC and any other Policy as Code solution depending on Az Module. Az 9.3.0 fixed this issue. Azure DevOps Microsoft hosted agents updates with the fix started rolling out on 1/23/2023.
+>
+> ---
+> ---
+
 <br/><br/>
 
 **On this page**

diff --git a/Scripts/Helpers/Convert-PolicySetsToDetails.ps1 b/Scripts/Helpers/Convert-PolicySetsToDetails.ps1
@@ -141,7 +141,7 @@ function Convert-PolicySetsToDetails {
                                 }
                             }
                             else {
-                                Write-Error "Policy uses parameter '$effectParameterName' for the effect not defined in the parameters. This should not be possible!" -ErrorAction Stop
+                                Write-Error "Policy '$($policyId)', referenceId '$($policyInPolicySet.policyDefinitionReferenceId)' tries to pass an unknown Policy Set parameter '$policySetLevelEffectParameterName' to the Policy parameter '$effectParameterName'. Check the spelling of the parameters occurrences in the Policy Set definition." -ErrorAction Stop
                             }
                         }
                         else {

diff --git a/StarterKit/Pipelines/AzureDevOps/multi-tenant-pipeline.yml b/StarterKit/Pipelines/AzureDevOps/multi-tenant-pipeline.yml
@@ -90,7 +90,7 @@ stages:
                     ScriptArguments:
                       -pacEnvironmentSelector $(pacEnvironmentSelector) `
                       -InformationAction Continue
-  - stage: tenantPlanFeatureStage-1
+  - stage: tenant1PlanFeatureStage
     displayName: "Tenant 1 Plan - Feature Branch"
     dependsOn:
       - devStage
@@ -124,7 +124,7 @@ stages:
             #     artifactName: "plans-feature-branch-$(pacEnvironmentSelector)"
             #     condition: and(succeeded(), or(eq(variables['planStep.deployPolicyChanges'], 'yes'), eq(variables['planStep.deployRoleChanges'], 'yes')))
           # End Publish
-  - stage: tenantPlanFeatureStage-2
+  - stage: tenant2PlanFeatureStage
     displayName: "Tenant 2 Plan - Feature Branch"
     dependsOn:
       - devStage
@@ -161,8 +161,8 @@ stages:
   - stage: completedFeature
     displayName: "Completed Plans - Feature Branch"
     dependsOn:
-      - tenant1PlanFeatureStage-1
-      - tenant2PlanFeatureStage-2
+      - tenant1PlanFeatureStage
+      - tenant2PlanFeatureStage
     condition: and(not(failed()), not(canceled()), in(variables['Build.Reason'], 'Manual', 'IndividualCI', 'BatchedCI'), not(contains(variables['Build.SourceBranch'], 'refs/heads/main')))
     jobs:
       - job:
@@ -173,7 +173,7 @@ stages:
               targetType: "inline"
               script: |
                 Write-Host "Completed Plans - Feature Branch"
-  - stage: tenantPlanMainStage-1
+  - stage: tenant1PlanMainStage
     displayName: "Tenant 1 Plan - Main Branch"
     dependsOn:
       - completedFeature
@@ -207,11 +207,11 @@ stages:
             #     artifactName: "policy-plan-$(pacEnvironmentSelector)"
             #   condition: and(succeeded(), or(eq(variables['planStep.deployPolicyChanges'], 'yes'), eq(variables['planStep.deployRoleChanges'], 'yes')))
           # End Publish
-  - stage: tenantDeployPolicyStage-1
+  - stage: tenant1DeployPolicyStage
     displayName: "Tenant 1 Deploy Policy"
     dependsOn:
-      - tenantPlanMainStage-1
-    condition: and(not(failed()), not(canceled()), eq(dependencies.tenantPlanMainStage-1.outputs['planJob.planStep.deployPolicyChanges'], 'yes'), in(variables['Build.Reason'], 'Manual', 'IndividualCI', 'BatchedCI'), contains(variables['Build.SourceBranch'], 'refs/heads/main'))
+      - tenant1PlanMainStage
+    condition: and(not(failed()), not(canceled()), eq(dependencies.tenant1PlanMainStage.outputs['planJob.planStep.deployPolicyChanges'], 'yes'), in(variables['Build.Reason'], 'Manual', 'IndividualCI', 'BatchedCI'), contains(variables['Build.SourceBranch'], 'refs/heads/main'))
     variables:
       pacEnvironmentSelector: tenant1
       PAC_INPUT_FOLDER: "$(Pipeline.Workspace)"
@@ -234,11 +234,11 @@ stages:
                     ScriptArguments:
                       -pacEnvironmentSelector $(pacEnvironmentSelector) `
                       -InformationAction Continue
-  - stage: tenantRolesStage-1
+  - stage: tenant1RolesStage
     displayName: "Tenant 1 Deploy Role Assignments"
     dependsOn:
-      - tenantDeployPolicyStage-1
-    condition: and(not(failed()), not(canceled()), eq(dependencies.tenantPlanMainStage-1.outputs['planJob.planStep.deployRoleChanges'], 'yes'), in(variables['Build.Reason'], 'Manual', 'IndividualCI', 'BatchedCI'), contains(variables['Build.SourceBranch'], 'refs/heads/main'))
+      - tenant1DeployPolicyStage
+    condition: and(not(failed()), not(canceled()), eq(dependencies.tenant1PlanMainStage.outputs['planJob.planStep.deployRoleChanges'], 'yes'), in(variables['Build.Reason'], 'Manual', 'IndividualCI', 'BatchedCI'), contains(variables['Build.SourceBranch'], 'refs/heads/main'))
     variables:
       pacEnvironmentSelector: tenant1
       PAC_INPUT_FOLDER: "$(Pipeline.Workspace)"
@@ -261,7 +261,7 @@ stages:
                       -pacEnvironmentSelector $(pacEnvironmentSelector) `
                       -InformationAction Continue
                   condition: succeeded()
-  - stage: tenantPlanMainStage-2
+  - stage: tenant2PlanMainStage
     displayName: "Tenant 2 Plan - Main Branch"
     dependsOn:
       - completedFeature
@@ -295,11 +295,11 @@ stages:
             #     artifactName: "policy-plan-$(pacEnvironmentSelector)"
             #   condition: and(succeeded(), or(eq(variables['planStep.deployPolicyChanges'], 'yes'), eq(variables['planStep.deployRoleChanges'], 'yes')))
           # End Publish
-  - stage: tenantDeployPolicyStage-2
+  - stage: tenant2DeployPolicyStage
     displayName: "Tenant 2 Deploy Policy"
     dependsOn:
-      - tenantPlanMainStage-2
-    condition: and(not(failed()), not(canceled()), eq(dependencies.tenantPlanMainStage-2.outputs['planJob.planStep.deployPolicyChanges'], 'yes'), in(variables['Build.Reason'], 'Manual', 'IndividualCI', 'BatchedCI'), contains(variables['Build.SourceBranch'], 'refs/heads/main'))
+      - tenant2PlanMainStage
+    condition: and(not(failed()), not(canceled()), eq(dependencies.tenant2PlanMainStage.outputs['planJob.planStep.deployPolicyChanges'], 'yes'), in(variables['Build.Reason'], 'Manual', 'IndividualCI', 'BatchedCI'), contains(variables['Build.SourceBranch'], 'refs/heads/main'))
     variables:
       pacEnvironmentSelector: tenant2
       PAC_INPUT_FOLDER: "$(Pipeline.Workspace)"
@@ -322,11 +322,11 @@ stages:
                     ScriptArguments:
                       -pacEnvironmentSelector $(pacEnvironmentSelector) `
                       -InformationAction Continue
-  - stage: tenantRolesStage-2
+  - stage: tenant2RolesStage
     displayName: "Tenant Deploy Role Assignments"
     dependsOn:
-      - tenantDeployPolicyStage-2
-    condition: and(not(failed()), not(canceled()), eq(dependencies.tenantPlanMainStage-2.outputs['planJob.planStep.deployRoleChanges'], 'yes'), in(variables['Build.Reason'], 'Manual', 'IndividualCI', 'BatchedCI'), contains(variables['Build.SourceBranch'], 'refs/heads/main'))
+      - tenant2DeployPolicyStage
+    condition: and(not(failed()), not(canceled()), eq(dependencies.tenant2PlanMainStage.outputs['planJob.planStep.deployRoleChanges'], 'yes'), in(variables['Build.Reason'], 'Manual', 'IndividualCI', 'BatchedCI'), contains(variables['Build.SourceBranch'], 'refs/heads/main'))
     variables:
       pacEnvironmentSelector: tenant2
       PAC_INPUT_FOLDER: "$(Pipeline.Workspace)"
-Original file line number
+Diff line change
@@ Expand Up / @@ -141,7 +141,7 @@ function Convert-PolicySetsToDetails { @@
                                     }
                                 }
                                 else {
-                                    Write-Error "Policy uses parameter '$effectParameterName' for the effect not defined in the parameters. This should not be possible!" -ErrorAction Stop
+                                    Write-Error "Policy '$($policyId)', referenceId '$($policyInPolicySet.policyDefinitionReferenceId)' tries to pass an unknown Policy Set parameter '$policySetLevelEffectParameterName' to the Policy parameter '$effectParameterName'. Check the spelling of the parameters occurrences in the Policy Set definition." -ErrorAction Stop
                                 }
                             }
                             else {
@@ Expand Down @@