Proposal/Request for improvement for automated deployment testing/assertion

[ricmestre]

Given the multiple issues I've raised with resources not working in deployments would it be possible for you to create an environment where to test deployments of all resources of the project? And then call Assert-M365DSCBlueprint to check back if there any drifts, this would ensure that not only if the resource was deployed correctly but also if the drifts checking is working properly or if it needs to be corrected. Ideally each resource should be tested with all the authentication methods that it supports.

On my side I'm currently only testing deployments with resources that we will be using (customers may be using completely different resources and therefore not tested) and I'm doing all this manually which means it has a really narrow scope of which resources may or may not work on a real deployment scenario and it's also extremely time consuming.

I'm perfectly aware this is not an easy feat since you'd have to ensure that the target tenant(s) would need to return to a clean state at the end of each test so that new tests run correctly, maybe you'd need several tenants in order to run several tests in parallel, have these tests run on public or private repo? It also wouldn't be able to test every single value the resource supports and all their combinations between them, but at least would be a start to have 1 or 2 examples per resource that could be tested, maybe even use the ones already included in the Unit Testing?

At the end of the day this would help me immensely since at the moment I'm basically adding conditions such as the below to essentially bypass resources that I know will fail the deployment (noticed by manual testing) but of course this is a hack since I won't be able to rule out every single one of the failing resources because I won't have time to test them all. This also means that something might go utterly wrong in a deployment in a customer's tenant (not only might it fail to deploy but deploy resources partially) and I won't know prior that the resource(s) that caused it was/were not working because I didn't test it.

<#
    Workaround to bypass IntuneApplicationControlPolicyWindows10
    policies until issue
    https://github.com/microsoft/Microsoft365DSC/pull/3437 is not
    fixed
#>
if ($Resource.ResourceName -eq "IntuneApplicationControlPolicyWindows10") {
    continue
}

Really appreciate you take this in consideration!

[andikrueger]

This is a very fair point. Real-world tests of Microsoft 365 DSC may indeed pose several challenges. Some of these challenges could be:

General challenges

• Within Microsoft 365, there are settings that can only be set once. Basically we would only be able to test it one time within a tenant.
• Sadly, not all settings are immediately active. Some even need 72 hours to take effect - and in these instances, we would get back the wrong/old value for the setting.
• Some cmdLets for configuring the services are only available with a specific license.

M365DSC specific challenges

• Microsoft 365 supports five different authentication methods, which should be tests independently. Some resources will work with credentials, but not with service principals.
• Large configurations should be avoided to save runtime and allow for parallel processing.
• API Permissions need admin consent - if they would be updated

Overall test procedure could like a several step approach:

1. Export a full tenant configuration

2. Apply each and every sample of Microsoft 365 to change the settings. *

3. Asses the state of the tenant with the newly set settings

4. Create a drift report

5. Reset to the original values. *

6. Create a report of the original configuration

7. and 5. are going to be challenging parts. To make changes the key-parameters within the example must meet the original configuration. Same applies for the "reset" of the settings. This could be done with the help of DSCParser and a bit of magic ;)

Things to think about

• How are settings handled, that are not part of a default tenant? Is just create/delete for them? This would apply for all settings that are not single instances within a tenant.
• Is it required to Export, Set, Update, Delete settings?
• ...

[ricmestre]

@andikrueger Very well put! See below some added context to the conversation.

Resources that are known to take a considerable amount of time to apply in the tenant should be left out of the testing.

When I said that it should test all authentication methods of all resources I meant that it should test the ones each resource supports, of course some of them don't support Service Principals (unfortunately) so that shouldn't be tested.

I'd say that resources that can be only deployed once, shouldn't be tested, at the same time resources that have settings that can be only applied once then those settings should be left out of the tests but the resource should still be tested with remaining ones.

Additionally In order to test all code paths and have a proper testing then Export, Set, Update, Delete on those settings should be required. For resources that tweak the accesses to the tenant itself such as authentication methods, conditional accesses, etc, then they should be handled carefully in such a way that whatever policy is being applied uses the right setting values that doesn't restrict the access and subsequent tests stop working, if for specific cases of these resources the actual deployment will for sure restrict the access then those should be left out of the tests.

[andikrueger]

The tenants used to test M365DSC should have a certain configuration in place to enable all tests. e.g. it would be required to enable O365OrgCustomizationSetting once to allow customizations.

[ricmestre]

@andikrueger Well what do you know, digging for something else I came across these integration tests [0] which actually deploy over to a public cloud, but for the past few months have been failing [1] due to the change to require resources to have an ID key property defined which these resources in the tests lack.

It only seems to be testing the resources with regular user/pass credentials and not any other authentication methods, but it would be a great start to have this working again.

[0] https://github.com/microsoft/Microsoft365DSC/blob/Dev/Tests/Integration/M365DSCIntegration.Master.Tests.ps1

[1] https://github.com/microsoft/Microsoft365DSC/actions/runs/5601482737/job/15174036645

[ricmestre]

@andikrueger There's also this ResourceIntegration pipeline [0] which is exactly what I was asking for, it tests different types of authentication, it tests for the resource creation, its update and removal with Test-DSCConfiguration called between each os these steps.

Unfortunately this one is also failing due to a small envelope size, it just needs to have MaxEnvelopeSizekb bumped to an higher number, default is 512KB and is failing now with 600 something, but the biggest problem is that it only contains these tests for AADApplication and no other resources so it falls short but with this working properly it would allow me to sleep better at night so please consider taking a look into this 👍

[0] https://github.com/microsoft/Microsoft365DSC/actions/runs/5601482731/workflow

[ricmestre]

@andikrueger Is there any news about this?

At this moment I need to invest time in creating tests to test all resources by myself in order to not allow customers to use resources that are not working, so far I've only had time to test Intune ones but from those tests I can confirm there are about ~30 types of DSC resources that are currently not functional and cannot be used for different reasons, since they're so many I've only reported issues for a bunch of them.

[ricmestre]

FYI, this is the list of non-working resources that I'm currently excluding from being used for Intune.

$global:ExcludedResources = @(
    # If value(s) is/are encrypted they cannot be applied to other tenants
    # since deployments will fail.
    "IntuneDeviceConfigurationCustomPolicyWindows10",
    # https://github.com/microsoft/Microsoft365DSC/pull/3444
    # https://github.com/microsoft/Microsoft365DSC/issues/3851
    # PR https://github.com/microsoft/Microsoft365DSC/pull/3886
    "IntuneDeviceConfigurationEndpointProtectionPolicyWindows10",
    # https://github.com/microsoft/Microsoft365DSC/pull/3440 and
    # https://github.com/microsoft/Microsoft365DSC/issues/3456
    "IntuneDeviceConfigurationPolicyIOS",
    # https://github.com/microsoft/Microsoft365DSC/pull/3442
    # PR https://github.com/microsoft/Microsoft365DSC/pull/3878
    "IntuneDeviceEnrollmentStatusPageWindows10",
    # https://github.com/microsoft/DSCParser/issues/34
    "IntuneSettingCatalogCustomPolicyWindows10",
    # https://github.com/microsoft/Microsoft365DSC/pull/3438
    "IntuneWindowsUpdateForBusinessFeatureUpdateProfileWindows10",
    # https://github.com/microsoft/Microsoft365DSC/issues/3539
    "IntuneDeviceConfigurationPolicyMacOS",
    # https://github.com/microsoft/Microsoft365DSC/issues/3540
    "IntuneAntivirusPolicyWindows10SettingCatalog",
    # https://github.com/microsoft/Microsoft365DSC/issues/3542
    "IntuneAppConfigurationPolicy",
    # https://github.com/microsoft/Microsoft365DSC/issues/3582
    # PR https://github.com/microsoft/Microsoft365DSC/pull/3873
    "IntuneSettingCatalogCustomPolicyWindows10",
    # https://github.com/microsoft/Microsoft365DSC/issues/3730
    "IntuneDeviceEnrollmentPlatformRestriction",
    # https://github.com/microsoft/Microsoft365DSC/issues/3869
    # PR https://github.com/microsoft/Microsoft365DSC/pull/3874
    "IntuneDeviceConfigurationAdministrativeTemplatePolicyWindows10",
    # https://github.com/microsoft/Microsoft365DSC/issues/3888
    # PR https://github.com/microsoft/Microsoft365DSC/pull/3893
    "IntuneDeviceConfigurationDeliveryOptimizationPolicyWindows10",
    "IntuneDeviceConfigurationHealthMonitoringConfigurationPolicyWindows10",
    "IntuneDeviceConfigurationIdentityProtectionPolicyWindows10",
    # https://github.com/microsoft/Microsoft365DSC/issues/3892
    # These also have the problem reported on #3888, PR #3893
    "IntuneWindowsAutopilotDeploymentProfileAzureADHybridJoined",
    "IntuneWindowsAutopilotDeploymentProfileAzureADJoined",
    # https://github.com/microsoft/Microsoft365DSC/issues/3892
    "IntuneWindowsUpdateForBusinessRingUpdateProfileWindows10",
    "IntuneDeviceConfigurationPolicyWindows10",
    # https://github.com/microsoft/Microsoft365DSC/issues/3897
    "IntuneRoleDefinition",

    # These resources are not working but still don't have any issues raised in
    # GitHub
    "IntuneAttackSurfaceReductionRulesPolicyWindows10ConfigManager",
    "IntuneDeviceAndAppManagementAssignmentFilter",
    "IntuneDeviceConfigurationEmailProfilePolicyWindows10",
    "IntuneDeviceConfigurationNetworkBoundaryPolicyWindows10",
    "IntuneDeviceConfigurationPolicyAndroidDeviceAdministrator",
    "IntuneDeviceConfigurationPolicyAndroidOpenSourceProject",
    "IntuneDeviceConfigurationPolicyAndroidWorkProfile",
    "IntuneDeviceConfigurationScepCertificatePolicyWindows10",
    "IntuneDeviceConfigurationWiredNetworkPolicyWindows10",
    "IntuneExploitProtectionPolicyWindows10SettingCatalog",
    "IntuneSettingCatalogASRRulesPolicyWindows10",
    "IntuneWifiConfigurationPolicyAndroidDeviceAdministrator",
    "IntuneWifiConfigurationPolicyAndroidEnterpriseDeviceOwner",
    "IntuneWifiConfigurationPolicyAndroidEnterpriseWorkProfile",
    "IntuneWifiConfigurationPolicyAndroidForWork",
    "IntuneWifiConfigurationPolicyAndroidOpenSourceProject",
    "IntuneWifiConfigurationPolicyIOS",
    "IntuneWifiConfigurationPolicyMacOS",
    "IntuneWifiConfigurationPolicyWindows10"

[andikrueger]

@ricmestre Thanks for bringing this again to attention. AFAIK, this is still a pending topic. There are several issues in the issues list that would have been prevented by the testing procedure described.

[ricmestre]

Just to let you know that I've updated the list, I found a few more resources with issues, others as mentioned in my comments already have PRs that are either yet to be reviewed or already merged but not yet included in the next release, but it feels like almost all Intune resources are not working that great.