Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

New-AzResource -Properties JSON white space removed - Breaking Change #20386

Closed
duttong13 opened this issue Dec 7, 2022 · 6 comments
Closed

New-AzResource -Properties JSON white space removed - Breaking Change #20386

duttong13 opened this issue Dec 7, 2022 · 6 comments
Assignees
@RakeshMohanMSFT
Labels
ARM - Core bug This issue requires a change to an existing behavior in the product in order to be resolved. customer-reported issue-addressed
Milestone
January 2023 (202...

Comments

@duttong13
Copy link

duttong13 commented Dec 7, 2022 •
edited
Loading

Description

New Bug within Az-Resource 6.5 version
When we call the following command whilst passing in ADFv2 Pipeline code in -Properties as JSON, we see the whitespaces are removed, e.g. properties.activities.typeProperties changes from 'file name.py' to 'filename' which breaks the ADFv2 pipeline. This is new in 6.5 and wasn't present in 6.4.1.

New-AzResource -ResourceType "Microsoft.DataFactory/factories/pipelines" -ResourceGroupName $resourceGroupName -Name "$dataFactoryName/$pipelineName" -ApiVersion "2018-06-01" -Properties $pipelineFile -IsFullObject -Force

Where $pipelineFile is below:

{
	"name": "Pipeline1",
	"properties": {
		"activities": [
			{
				"name": "Activity1",
				"type": "DatabricksNotebook",
				"dependsOn": [],
				"policy": {
					"timeout": "03:00:00",
					"retry": 0,
					"retryIntervalInSeconds": 30,
					"secureOutput": false,
					"secureInput": false
				},
				"userProperties": [],
				"typeProperties": {
					"notebookPath": "/Notebooks/notebook Extract",
					"baseParameters": {
						"OutputPath": {
							"value": "@pipeline().parameters.OutputPath",
							"type": "Expression"
						},
						"ProcessName": {
							"value": "@pipeline().parameters.ProcessName",
							"type": "Expression"
						},
						"SliceDate": {
							"value": "@formatDateTime(pipeline().parameters.SliceDate)",
							"type": "Expression"
						},
						"OutputHeader": {
							"value": "@pipeline().parameters.OutputHeader",
							"type": "Expression"
						},
						"QuoteAllColumns": {
							"value": "@pipeline().parameters.QuoteAllColumns",
							"type": "Expression"
						},
						"QuoteCharacter": {
							"value": "@pipeline().parameters.QuoteCharacter",
							"type": "Expression"
						},
						"DataLakeArea": {
							"value": "@pipeline().parameters.DataLakeArea",
							"type": "Expression"
						},
						"Encoding": {
							"value": "@pipeline().parameters.Encoding",
							"type": "Expression"
						},
						"IgnoreLeadingWhitepace": {
							"value": "@pipeline().parameters.ignoreLeadingWhitepace",
							"type": "Expression"
						},
						"IgnoreTrailingWhiteSpace": {
							"value": "@pipeline().parameters.IgnoreTrailingWhiteSpace",
							"type": "Expression"
						},
						"FileSeparator": {
							"value": "@pipeline().parameters.FileSeparator",
							"type": "Expression"
						}
					}
				},
				"linkedServiceName": {
					"referenceName": "LS_Databricks_Dynamic",
					"type": "LinkedServiceReference"
				}
			}
		],
		"parameters": {
			"OutputPath": {
				"type": "string"
			},
			"ProcessName": {
				"type": "string"
			},
			"SliceDate": {
				"type": "string"
			},
			"OutputHeader": {
				"type": "string",
				"defaultValue": "true"
			},
			"QuoteAllColumns": {
				"type": "string",
				"defaultValue": "false"
			},
			"QuoteCharacter": {
				"type": "string",
				"defaultValue": "\""
			},
			"DataLakeArea": {
				"type": "string",
				"defaultValue": "clean"
			},
			"Encoding": {
				"type": "string",
				"defaultValue": "UTF-8"
			},
			"IgnoreLeadingWhitepace": {
				"type": "string",
				"defaultValue": "true"
			},
			"IgnoreTrailingWhiteSpace": {
				"type": "string",
				"defaultValue": "true"
			},
			"FileSeparator": {
				"type": "string",
				"defaultValue": ","
			}
		},
		"folder": {
			"name": "Common"
		},
		"annotations": [],
		"lastPublishTime": "2022-10-18T20:23:40Z"
	},
	"type": "Microsoft.DataFactory/factories/pipelines"
}

Issue script & Debug output

$Pipeline =  New-AzResource -ResourceType "Microsoft.DataFactory/factories/pipelines" -ResourceGroupName $resourceGroupName -Name "$dataFactoryName/$pipelineName" -ApiVersion "2018-06-01" -Properties $pipelineFile -IsFullObject -Force #| Format-List
$Pipeline.properties.activities

result:

name              : Activity1
type              : DatabricksNotebook
dependsOn         : {}
policy            : @{timeout=03:00:00; retry=0; retryIntervalInSeconds=30; secureOutput=False; secureInput=False}
userProperties    : {}
typeProperties    : @{notebookPath=/Notebooks/notebookExtract; baseParameters=}
linkedServiceName : @{referenceName=LS_Databricks_Dynamic; type=LinkedServiceReference}

Environment data

Name                           Value                                                                                                                                                                                    
----                           -----                                                                                                                                                                                    
PSVersion                      5.1.22000.832                                                                                                                                                                            
PSEdition                      Desktop                                                                                                                                                                                  
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}                                                                                                                                                                  
BuildVersion                   10.0.22000.832                                                                                                                                                                           
CLRVersion                     4.0.30319.42000                                                                                                                                                                          
WSManStackVersion              3.0                                                                                                                                                                                      
PSRemotingProtocolVersion      2.3                                                                                                                                                                                      
SerializationVersion           1.1.0.1

Module versions

ModuleType Version    Name                                ExportedCommands                                                                                                                                              
---------- -------    ----                                ----------------                                                                                                                                              
Script     2.10.1     Az.Accounts                         {Add-AzEnvironment, Clear-AzConfig, Clear-AzContext, Clear-AzDefault...}                                                                                      
Script     1.16.8     Az.DataFactory                      {Add-AzDataFactoryV2DataFlowDebugSessionPackage, Add-AzDataFactoryV2TriggerSubscription, Get-AzDataFactory, Get-AzDataFactoryActivityWindow...}               
Script     6.2.0      Az.Resources                        {Export-AzResourceGroup, Export-AzTemplateSpec, Get-AzDenyAssignment, Get-AzDeployment...}

Error output

Message        : Cannot validate argument on parameter 'Properties'. The argument is null or empty. Provide an argument that is not null or empty, and then try the command again.
StackTrace     :    at System.Management.Automation.ParameterBinderBase.BindParameter(CommandParameterInternal parameter, CompiledCommandParameter parameterMetadata, ParameterBindingFlags flags)
                    at System.Management.Automation.CmdletParameterBinderController.BindParameter(CommandParameterInternal argument, MergedCompiledCommandParameter parameter, ParameterBindingFlags flags)
                    at System.Management.Automation.CmdletParameterBinderController.BindParameter(UInt32 parameterSets, CommandParameterInternal argument, MergedCompiledCommandParameter parameter, 
                 ParameterBindingFlags flags)
                    at System.Management.Automation.CmdletParameterBinderController.BindParameters(UInt32 parameterSets, Collection`1 arguments)
                    at System.Management.Automation.CmdletParameterBinderController.BindCommandLineParametersNoValidation(Collection`1 arguments)
                    at System.Management.Automation.CmdletParameterBinderController.BindCommandLineParameters(Collection`1 arguments)
                    at System.Management.Automation.CommandProcessor.BindCommandLineParameters()
                    at System.Management.Automation.CommandProcessor.Prepare(IDictionary psDefaultParameterValues)
                    at System.Management.Automation.CommandProcessorBase.DoPrepare(IDictionary psDefaultParameterValues)
                    at System.Management.Automation.Internal.PipelineProcessor.Start(Boolean incomingStream)
                    at System.Management.Automation.Internal.PipelineProcessor.SynchronousExecuteEnumerate(Object input)
                 --- End of stack trace from previous location where exception was thrown ---
                    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
                    at System.Management.Automation.Internal.PipelineProcessor.SynchronousExecuteEnumerate(Object input)
                    at System.Management.Automation.PipelineOps.InvokePipeline(Object input, Boolean ignoreInput, CommandParameterInternal[][] pipeElements, CommandBaseAst[] pipeElementAsts, CommandRedirection[][] 
                 commandRedirections, FunctionContext funcContext)
                    at System.Management.Automation.Interpreter.ActionCallInstruction`6.Run(InterpretedFrame frame)
                    at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)
                    at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)
Exception      : System.Management.Automation.ParameterBindingValidationException
InvocationInfo : {New-AzResource}
Line           :               $Pipeline =  New-AzResource -ResourceType "Microsoft.DataFactory/factories/pipelines" -ResourceGroupName $resourceGroupName -Name "$dataFactoryName/$pipelineName" -ApiVersion 
                 "2018-06-01" -Properties $pipelineFile -IsFullObject -Force #| Format-List
Position       : At line:1 char:215
                 + ... elineName" -ApiVersion "2018-06-01" -Properties $pipelineFile -IsFull ...
                 +                                                     ~~~~~~~~~~~~~
HistoryId      : 11

Message        : The argument is null or empty. Provide an argument that is not null or empty, and then try the command again.
StackTrace     :    at System.Management.Automation.ValidateNotNullOrEmptyAttribute.Validate(Object arguments, EngineIntrinsics engineIntrinsics)
                    at System.Management.Automation.ParameterBinderBase.BindParameter(CommandParameterInternal parameter, CompiledCommandParameter parameterMetadata, ParameterBindingFlags flags)
Exception      : System.Management.Automation.ValidationMetadataException
InvocationInfo : {New-AzResource}
Line           :               $Pipeline =  New-AzResource -ResourceType "Microsoft.DataFactory/factories/pipelines" -ResourceGroupName $resourceGroupName -Name "$dataFactoryName/$pipelineName" -ApiVersion 
                 "2018-06-01" -Properties $pipelineFile -IsFullObject -Force #| Format-List
Position       : At line:1 char:215
                 + ... elineName" -ApiVersion "2018-06-01" -Properties $pipelineFile -IsFull ...
                 +                                                     ~~~~~~~~~~~~~
HistoryId      : 11



   HistoryId: 4


Message        : ResourceNotFound : The Resource 'Microsoft.DataFactory/factories/wb-dp-stag-df02' under resource group 'wbdp-stag-rg' was not found. For more details please go to 
                 https://aka.ms/ARMResourceNotFoundFix
                 CorrelationId: 3626ede1-b932-427c-8459-1888364ed219
StackTrace     :    at Microsoft.Azure.Commands.ResourceManager.Cmdlets.Implementation.ResourceManagerCmdletBase.HandleException(ExceptionDispatchInfo capturedException)
                    at Microsoft.Azure.Commands.ResourceManager.Cmdlets.Implementation.ResourceManagerCmdletBase.ExecuteCmdlet()
                    at Microsoft.WindowsAzure.Commands.Utilities.Common.AzurePSCmdlet.ProcessRecord()
Exception      : Microsoft.Azure.Commands.ResourceManager.Cmdlets.Entities.ErrorResponses.ErrorResponseMessageException
InvocationInfo : {New-AzResource}
Line           :                 New-AzResource -ResourceType "Microsoft.DataFactory/factories/pipelines" -ResourceGroupName $resourceGroupName -Name "$dataFactoryName/$pipelineName" -ApiVersion "2018-06-01" 
                 -Properties $pipelineFile -IsFullObject -Force | Format-List
                 
Position       : At C:\c\WHB\Whitbread-DataPlatform2\deployment\DeployArtefacts.ps1:215 char:17
                 + ...             New-AzResource -ResourceType "Microsoft.DataFactory/facto ...
                 +                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
HistoryId      : 4



   HistoryId: -1


Message        : Unable to find type [Microsoft.PowerShell.Commands.PowerShellGet.Telemetry].
StackTrace     :    at System.Management.Automation.ExceptionHandlingOps.CheckActionPreference(FunctionContext funcContext, Exception exception)
                    at System.Management.Automation.Interpreter.ActionCallInstruction`2.Run(InterpretedFrame frame)
                    at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)
                    at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)
Exception      : System.Management.Automation.RuntimeException
InvocationInfo : {}
Line           :     $telemetryMethods = ([Microsoft.PowerShell.Commands.PowerShellGet.Telemetry] | Get-Member -Static).Name
                 
Position       : At C:\Program Files\WindowsPowerShell\Modules\PowerShellGet\1.0.0.1\PSModule.psm1:715 char:26
                 + ... yMethods = ([Microsoft.PowerShell.Commands.PowerShellGet.Telemetry] | ...
                 +                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
HistoryId      : -1


The Azure PowerShell team is listening, please let us know how we are doing: https://aka.ms/azpssurvey?Q_CHL=ERROR.
@duttong13 duttong13 added bug This issue requires a change to an existing behavior in the product in order to be resolved. needs-triage This is a new issue that needs to be triaged to the appropriate team. labels Dec 7, 2022
@ghost ghost added customer-reported and removed needs-triage This is a new issue that needs to be triaged to the appropriate team. labels Dec 7, 2022
@dingmeng-xue dingmeng-xue added ARM - Core Tracking We will track status and follow internally and removed Tracking We will track status and follow internally labels Dec 8, 2022
@dingmeng-xue
Copy link
Member

@duttong13 , thanks for reporting.

Could you share the debug log when you execute New-AzResource? You can turn on debug via $DebugPreference = "Debug".

@dingmeng-xue dingmeng-xue added the needs-author-feedback More information is needed from author to address the issue. label Dec 8, 2022
@duttong13
Copy link
Author

@dingmeng-xue, thanks for your response. I have run the script we use under 6.4.0 and 6.5.0 with the following results. Please note the change in line 2402 to 2400 where a space was removed:
Expected (current behaviour under 6.4.0)
2022-12-08T10:22:17.2328766Z "notebookPath": "/Notebooks/ATEStore/LoadScripts/ATE DPExtract",

Unexpected (under 6.5.0)
2022-12-08T10:43:13.6405856Z "notebookPath": "/Notebooks/ATEStore/LoadScripts/ATEDPExtract",

New-AzResource under Az.Resource 6.5.0 (truncated to fit in commment)

`##[section]Starting: Deploy Data Factory Objects

Task : PowerShell
Description : Run a PowerShell script on Linux, macOS, or Windows
Version : 2.212.0
Author : Microsoft Corporation
Help : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/powershell

Generating script.
Formatted command: . 'D:\a\r1\a_DeploymentFiles_deploymentFiles\DeployArtefacts.ps1' -deploymentFile "deployment.master.json" -deploymentParameterFile deployment.params.dev.json -dataFactoryName wb-dp-dev-df03 -resourceGroupName wbdp-dev-rg -workingDirectory "D:\a\r1\a/DataFactory/_datafactory" -region northeurope -tenantId fdfed904-9e03-4e17-89c4-61053e0777be -subscription b976df9a-6270-4417-affb-ec653aeb3aab -spKey *** -spId *** -deploymentDirectory "D:\a\r1\a/_DeploymentFiles/_deploymentFiles"
========================== Starting Command Output ===========================
##[command]"C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe" -NoLogo -NoProfile -NonInteractive -ExecutionPolicy Unrestricted -Command ". 'D:\a_temp\7b41f2af-6a1b-4999-8599-1ddddee543ee.ps1'"
DEBUG:
using System;
using System.Net;

namespace Microsoft.PowerShell.Commands.PowerShellGet
{
///


/// Used by Ping-Endpoint function to supply webproxy to HttpClient
/// We cannot use System.Net.WebProxy because this is not available on CoreClr
///
public class InternalWebProxy : IWebProxy
{
Uri _proxyUri;
ICredentials _credentials;

    public InternalWebProxy(Uri uri, ICredentials credentials)
    {
        Credentials = credentials;
        _proxyUri = uri;
    }

    /// <summary>
    /// Credentials used by WebProxy
    /// </summary>
    public ICredentials Credentials
    {
        get
        {
            return _credentials;
        }
        set
        {
            _credentials = value;
        }
    }

    public Uri GetProxy(Uri destination)
    {
        return _proxyUri;
    }

    public bool IsBypassed(Uri host)
    {
        return false;
    }
}

}
DEBUG:
using System;
using System.Management.Automation;

namespace Microsoft.PowerShell.Commands.PowerShellGet
{
public static class Telemetry
{
public static void TraceMessageArtifactsNotFound(string[] artifactsNotFound, string operationName)
{
Microsoft.PowerShell.Telemetry.Internal.TelemetryAPI.TraceMessage(operationName, new { ArtifactsNotFound =
artifactsNotFound });
}

    public static void TraceMessageNonPSGalleryRegistration(string sourceLocationType, string sourceLocationHash,

string installationPolicy, string packageManagementProvider, string publishLocationHash, string
scriptSourceLocationHash, string scriptPublishLocationHash, string operationName)
{
Microsoft.PowerShell.Telemetry.Internal.TelemetryAPI.TraceMessage(operationName, new { SourceLocationType =
sourceLocationType, SourceLocationHash = sourceLocationHash, InstallationPolicy = installationPolicy,
PackageManagementProvider = packageManagementProvider, PublishLocationHash = publishLocationHash,
ScriptSourceLocationHash = scriptSourceLocationHash, ScriptPublishLocationHash = scriptPublishLocationHash });
}
}
}
DEBUG:
using System;
using System.Net;
using Microsoft.Win32.SafeHandles;
using System.Security.Cryptography;
using System.Runtime.InteropServices;
using System.Runtime.ConstrainedExecution;
using System.Runtime.Versioning;
using System.Security;

namespace Microsoft.PowerShell.Commands.PowerShellGet
{
[StructLayout(LayoutKind.Sequential, CharSet=CharSet.Unicode)]
public struct CERT_CHAIN_POLICY_PARA {
public CERT_CHAIN_POLICY_PARA(int size) {
cbSize = (uint) size;
dwFlags = 0;
pvExtraPolicyPara = IntPtr.Zero;
}
public uint cbSize;
public uint dwFlags;
public IntPtr pvExtraPolicyPara;
}

[StructLayout(LayoutKind.Sequential, CharSet=CharSet.Unicode)]
public struct CERT_CHAIN_POLICY_STATUS {
    public CERT_CHAIN_POLICY_STATUS(int size) {
        cbSize = (uint) size;
        dwError = 0;
        lChainIndex = IntPtr.Zero;
        lElementIndex = IntPtr.Zero;
        pvExtraPolicyStatus = IntPtr.Zero;
    }
    public uint   cbSize;
    public uint   dwError;
    public IntPtr lChainIndex;
    public IntPtr lElementIndex;
    public IntPtr pvExtraPolicyStatus;
}

// Internal SafeHandleZeroOrMinusOneIsInvalid class to remove the dependency on .Net Framework 4.6.
public abstract class InternalSafeHandleZeroOrMinusOneIsInvalid : SafeHandle
{
    protected InternalSafeHandleZeroOrMinusOneIsInvalid(bool ownsHandle)
        : base(IntPtr.Zero, ownsHandle)
    {
    }

    public override bool IsInvalid
    {
        get
        {
            return handle == IntPtr.Zero || handle == new IntPtr(-1);
        }
    }
}

// Internal SafeX509ChainHandle class to remove the dependency on .Net Framework 4.6.
[SecurityCritical]
public sealed class InternalSafeX509ChainHandle : InternalSafeHandleZeroOrMinusOneIsInvalid {
    private InternalSafeX509ChainHandle () : base(true) {}

    internal InternalSafeX509ChainHandle (IntPtr handle) : base (true) {
        SetHandle(handle);
    }

    internal static InternalSafeX509ChainHandle InvalidHandle {
        get { return new InternalSafeX509ChainHandle(IntPtr.Zero); }
    }

    [SecurityCritical]
    override protected bool ReleaseHandle()
    {
        CertFreeCertificateChain(handle);
        return true;
    }

    [DllImport("Crypt32.dll", SetLastError=true)]

    [SuppressUnmanagedCodeSecurity,
     ResourceExposure(ResourceScope.None),
     ReliabilityContract(Consistency.WillNotCorruptState, Cer.Success)]
    
    private static extern void CertFreeCertificateChain(IntPtr handle);
}

public class Win32Helpers
{
    [DllImport("Crypt32.dll", CharSet=CharSet.Auto, SetLastError=true)]
    public extern static
    bool CertVerifyCertificateChainPolicy(
        [In]     IntPtr                       pszPolicyOID,
        [In]     SafeX509ChainHandle  pChainContext,
        [In]     ref CERT_CHAIN_POLICY_PARA   pPolicyPara,
        [In,Out] ref CERT_CHAIN_POLICY_STATUS pPolicyStatus);

    [DllImport("Crypt32.dll", CharSet=CharSet.Auto, SetLastError=true)]
    public static extern
    SafeX509ChainHandle CertDuplicateCertificateChain(
        [In]     IntPtr pChainContext);

    [DllImport("Crypt32.dll", CharSet=CharSet.Auto, SetLastError=true)]

    [ResourceExposure(ResourceScope.None)]

    public static extern
    SafeX509ChainHandle CertDuplicateCertificateChain(
        [In]     SafeX509ChainHandle pChainContext);

    public static bool IsMicrosoftCertificate([In] SafeX509ChainHandle pChainContext)
    {
        //-------------------------------------------------------------------------
        //  CERT_CHAIN_POLICY_MICROSOFT_ROOT
        //
        //  Checks if the last element of the first simple chain contains a
        //  Microsoft root public key. If it doesn't contain a Microsoft root
        //  public key, dwError is set to CERT_E_UNTRUSTEDROOT.
        //
        //  pPolicyPara is optional. However,
        //  MICROSOFT_ROOT_CERT_CHAIN_POLICY_ENABLE_TEST_ROOT_FLAG can be set in
        //  the dwFlags in pPolicyPara to also check for the Microsoft Test Roots.
        //
        //  MICROSOFT_ROOT_CERT_CHAIN_POLICY_CHECK_APPLICATION_ROOT_FLAG can be set
        //  in the dwFlags in pPolicyPara to check for the Microsoft root for
        //  application signing instead of the Microsoft product root. This flag
        //  explicitly checks for the application root only and cannot be combined
        //  with the test root flag.
        //
        //  MICROSOFT_ROOT_CERT_CHAIN_POLICY_DISABLE_FLIGHT_ROOT_FLAG can be set
        //  in the dwFlags in pPolicyPara to always disable the Flight root.
        //
        //  pvExtraPolicyPara and pvExtraPolicyStatus aren't used and must be set
        //  to NULL.
        //--------------------------------------------------------------------------
        const uint MICROSOFT_ROOT_CERT_CHAIN_POLICY_ENABLE_TEST_ROOT_FLAG       = 0x00010000;
        const uint MICROSOFT_ROOT_CERT_CHAIN_POLICY_CHECK_APPLICATION_ROOT_FLAG = 0x00020000;
        //const uint MICROSOFT_ROOT_CERT_CHAIN_POLICY_DISABLE_FLIGHT_ROOT_FLAG    = 0x00040000;

        CERT_CHAIN_POLICY_PARA PolicyPara = new

CERT_CHAIN_POLICY_PARA(Marshal.SizeOf(typeof(CERT_CHAIN_POLICY_PARA)));
CERT_CHAIN_POLICY_STATUS PolicyStatus = new
CERT_CHAIN_POLICY_STATUS(Marshal.SizeOf(typeof(CERT_CHAIN_POLICY_STATUS)));
int CERT_CHAIN_POLICY_MICROSOFT_ROOT = 7;

        PolicyPara.dwFlags = (uint) MICROSOFT_ROOT_CERT_CHAIN_POLICY_ENABLE_TEST_ROOT_FLAG;
        bool isMicrosoftRoot = false;

        if(CertVerifyCertificateChainPolicy(new IntPtr(CERT_CHAIN_POLICY_MICROSOFT_ROOT),
                                            pChainContext,
                                            ref PolicyPara,
                                            ref PolicyStatus))
        {
            isMicrosoftRoot = (PolicyStatus.dwError == 0);
        }

        // Also check for the Microsoft root for application signing if the Microsoft product root verification is

unsuccessful.
if(!isMicrosoftRoot)
{
// Some Microsoft modules can be signed with Microsoft Application Root instead of Microsoft Product
Root,
// So we need to use the MICROSOFT_ROOT_CERT_CHAIN_POLICY_CHECK_APPLICATION_ROOT_FLAG for the
certificate verification.
// MICROSOFT_ROOT_CERT_CHAIN_POLICY_CHECK_APPLICATION_ROOT_FLAG can not be used
// with MICROSOFT_ROOT_CERT_CHAIN_POLICY_ENABLE_TEST_ROOT_FLAG,
// so additional CertVerifyCertificateChainPolicy call is required to verify the given certificate is
in Microsoft Application Root.
//
CERT_CHAIN_POLICY_PARA PolicyPara2 = new
CERT_CHAIN_POLICY_PARA(Marshal.SizeOf(typeof(CERT_CHAIN_POLICY_PARA)));
CERT_CHAIN_POLICY_STATUS PolicyStatus2 = new
CERT_CHAIN_POLICY_STATUS(Marshal.SizeOf(typeof(CERT_CHAIN_POLICY_STATUS)));
PolicyPara2.dwFlags = (uint) MICROSOFT_ROOT_CERT_CHAIN_POLICY_CHECK_APPLICATION_ROOT_FLAG;

            if(CertVerifyCertificateChainPolicy(new IntPtr(CERT_CHAIN_POLICY_MICROSOFT_ROOT),
                                                pChainContext,
                                                ref PolicyPara2,
                                                ref PolicyStatus2))
            {
                isMicrosoftRoot = (PolicyStatus2.dwError == 0);
            }
        }

        return isMicrosoftRoot;
    }
}

}
DEBUG: 00:00:00.0016776 Current running environment: Windows PowerShell.
DEBUG: 00:00:00.0548232 Trying provider assembly: C:\Program
Files\WindowsPowerShell\Modules\PackageManagement\1.4.8.1\fullclr\Microsoft.PackageManagement.NuGetProvider.dll
DEBUG: 00:00:00.0549108 Trying provider assembly: C:\Program
Files\WindowsPowerShell\Modules\PackageManagement\1.4.8.1\fullclr\Microsoft.PackageManagement.ArchiverProviders.dll
DEBUG: 00:00:00.0552520 Trying provider assembly: C:\Program
Files\WindowsPowerShell\Modules\PackageManagement\1.4.8.1\fullclr\Microsoft.PackageManagement.MetaProvider.PowerShell.d
ll
DEBUG: 00:00:00.0561352 Attempting loading of assembly: C:\Program
Files\WindowsPowerShell\Modules\PackageManagement\1.4.8.1\fullclr\Microsoft.PackageManagement.ArchiverProviders.dll
DEBUG: 00:00:00.0567645 Attempting loading of assembly: C:\Program
Files\WindowsPowerShell\Modules\PackageManagement\1.4.8.1\fullclr\Microsoft.PackageManagement.NuGetProvider.dll
DEBUG: 00:00:00.0575397 Attempting loading of assembly: C:\Program
Files\WindowsPowerShell\Modules\PackageManagement\1.4.8.1\fullclr\Microsoft.PackageManagement.MetaProvider.PowerShell.d
ll
DEBUG: 00:00:00.0640648 Trying provider assembly: C:\Program
Files\WindowsPowerShell\Modules\PackageManagement\1.4.8.1\fullclr\Microsoft.PackageManagement.CoreProviders.dll
DEBUG: 00:00:00.0660256 Attempting loading of assembly: C:\Program
Files\WindowsPowerShell\Modules\PackageManagement\1.4.8.1\fullclr\Microsoft.PackageManagement.CoreProviders.dll
DEBUG: 00:00:00.0796217 Trying provider assembly: C:\Program
Files\WindowsPowerShell\Modules\PackageManagement\1.4.8.1\fullclr\Microsoft.PackageManagement.MsuProvider.dll
DEBUG: 00:00:00.0799612 Attempting loading of assembly: C:\Program
Files\WindowsPowerShell\Modules\PackageManagement\1.4.8.1\fullclr\Microsoft.PackageManagement.MsuProvider.dll
DEBUG: 00:00:00.0863966 Trying provider assembly: C:\Program
Files\WindowsPowerShell\Modules\PackageManagement\1.4.8.1\fullclr\Microsoft.PackageManagement.MsiProvider.dll
DEBUG: 00:00:00.0875387 Attempting loading of assembly: C:\Program
Files\WindowsPowerShell\Modules\PackageManagement\1.4.8.1\fullclr\Microsoft.PackageManagement.MsiProvider.dll
DEBUG: 00:00:00.1185742 Registering providers via metaproviders for assembly
Microsoft.PackageManagement.MetaProvider.PowerShell.Internal.PowerShellMetaProvider
DEBUG: 00:00:00.1470252 Initializing provider 'Programs'
DEBUG: 00:00:00.1589363 Initializing provider 'msu'
DEBUG: 00:00:00.1693599 Initializing provider 'NuGet'
DEBUG: 00:00:00.1935129 Initializing provider 'msi'
DEBUG: 00:00:00.2038081 Trying to register metaprovider
DEBUG: 00:00:00.2092621 Initializing provider 'zipfile'
DEBUG: 00:00:00.2119359 Initializing PowerShell MetaProvider
DEBUG: 00:00:00.2167738 Provider 'zipfile' Initialized
DEBUG: 00:00:00.2168246 Using Archiver Provider zipfile
DEBUG: 00:00:00.2171397 Initializing provider 'Bootstrap'
DEBUG: 00:00:00.2231649 Initializing provider 'WebDownloader'
DEBUG: 00:00:00.2905295 Initialize Bootstrapper
DEBUG: 00:00:00.2930883 Provider 'WebDownloader' Initialized
DEBUG: 00:00:00.2931386 Using Downloader Provider WebDownloader
DEBUG: 00:00:00.3072103 Provider 'Bootstrap' Initialized
DEBUG: 00:00:00.3072651 Using Package Provider Bootstrap
DEBUG: 00:00:00.3353305 Calling 'Bootstrap::GetFeatures'
DEBUG: 00:00:00.3380584 The provider 'Bootstrap' is imported
DEBUG: 00:00:01.0453666 Calling 'Programs::InitializeProvider'
DEBUG: 00:00:01.0454272 Provider 'Programs' Initialized
DEBUG: 00:00:01.0454961 Calling 'msi::InitializeProvider'
DEBUG: 00:00:01.0455483 Provider 'msi' Initialized
DEBUG: 00:00:01.0455844 Using Package Provider msi
DEBUG: 00:00:01.0457305 Using Package Provider Programs
DEBUG: 00:00:01.0460102 Calling 'WebDownloader::GetFeatures'
DEBUG: 00:00:01.0461003 Calling 'zipfile::GetFeatures'
DEBUG: 00:00:01.0470419 Calling 'WebDownloader::DownloadFile'
'https://go.microsoft.com/fwlink/?LinkID=627338&clcid=0x409','C:\Users\VssAdministrator\AppData\Local\Temp\Microsoft.Pa
ckageManagement\xjur2kqo.ppp','40000','False'
DEBUG: 00:00:01.0472710 Calling 'NuGet'::'InitializeProvider'.
DEBUG: 00:00:01.0473482 Provider 'NuGet' Initialized
DEBUG: 00:00:01.0473863 Using Package Provider NuGet
DEBUG: 00:00:01.0475421 Calling 'msu::InitializeProvider'
DEBUG: 00:00:01.0475825 Provider 'msu' Initialized
DEBUG: 00:00:01.0476254 Using Package Provider msu
DEBUG: 00:00:01.0483197 SUCCESS provider assembly: C:\Program
Files\WindowsPowerShell\Modules\PackageManagement\1.4.8.1\fullclr\Microsoft.PackageManagement.ArchiverProviders.dll
DEBUG: 00:00:01.0522255 Calling 'msi::GetFeatures'
DEBUG: 00:00:01.0522962 The provider 'msi' is imported
DEBUG: 00:00:01.0526252 Calling 'Programs::GetFeatures'
DEBUG: 00:00:01.0526865 The provider 'Programs' is imported
DEBUG: 00:00:01.0528922 SUCCESS provider assembly: C:\Program
Files\WindowsPowerShell\Modules\PackageManagement\1.4.8.1\fullclr\Microsoft.PackageManagement.MsiProvider.dll
DEBUG: 00:00:01.0529707 SUCCESS provider assembly: C:\Program
Files\WindowsPowerShell\Modules\PackageManagement\1.4.8.1\fullclr\Microsoft.PackageManagement.CoreProviders.dll
DEBUG: 00:00:01.0605662 Calling 'NuGet'::'GetFeatures'.
DEBUG: 00:00:01.0617556 The provider 'NuGet' is imported
DEBUG: 00:00:01.0631014 SUCCESS provider assembly: C:\Program
Files\WindowsPowerShell\Modules\PackageManagement\1.4.8.1\fullclr\Microsoft.PackageManagement.NuGetProvider.dll
DEBUG: 00:00:01.0664974 Calling 'msu::GetFeatures'
DEBUG: 00:00:01.0665838 The provider 'msu' is imported
DEBUG: 00:00:01.0670331 SUCCESS provider assembly: C:\Program
Files\WindowsPowerShell\Modules\PackageManagement\1.4.8.1\fullclr\Microsoft.PackageManagement.MsuProvider.dll
DEBUG: 00:00:01.2020435 Calling 'WebDownloader::DownloadFile'
'https://onegetcdn.azureedge.net/providers/nuget-2.8.5.208.package.swidtag','C:\Users\VssAdministrator\AppData\Local\Te
mp\Microsoft.PackageManagement\hfh3ueta.n05','40000','False'
DEBUG: 00:00:01.2211981 Calling 'WebDownloader::DownloadFile'
'https://onegetcdn.azureedge.net/providers/psl-1.0.0.210.package.swidtag','C:\Users\VssAdministrator\AppData\Local\Temp
\Microsoft.PackageManagement\y5owaejo.npd','40000','False'
DEBUG: 00:00:01.2395331 Calling 'WebDownloader::DownloadFile'
'https://onegetcdn.azureedge.net/providers/ChocolateyPrototype-2.8.5.130.package.swidtag','C:\Users\VssAdministrator\Ap
pData\Local\Temp\Microsoft.PackageManagement\im4ykmwr.xqv','40000','False'
DEBUG: 00:00:01.5126545 Attempting to load PowerShell Provider Module [C:\Program
Files\WindowsPowerShell\Modules\PowerShellGet\2.2.5\PSModule.psm1]
DEBUG: 00:00:01.7964514 Loaded PowerShell package provider: '[C:\Program
Files\WindowsPowerShell\Modules\PowerShellGet\2.2.5\PSModule.psm1]'.
DEBUG: 00:00:01.8056378 Loaded PowerShell Provider: PowerShellGet
DEBUG: 00:00:01.8082665 Initializing provider 'PowerShellGet'
DEBUG: 00:00:01.8152504 Calling New() : MethodName = 'InitializeProvider'
DEBUG: 00:00:01.8208823 INVOKING PowerShell Fn Initialize-Provider with args that has length 0
DEBUG: 00:00:01.9037310 In PowerShellGet Provider - 'Initialize-Provider'.
DEBUG: 00:00:01.9041596 PowerShell Script 'PSModule' Function 'Initialize-Provider' returns null.
DEBUG: 00:00:01.9055494 Done calling powershell «Initialize-Provider» «PSModule»
DEBUG: 00:00:01.9056056 Provider 'PowerShellGet' Initialized
DEBUG: 00:00:01.9056445 Using Package Provider PowerShellGet
DEBUG: 00:00:01.9128421 Calling New() : MethodName = 'GetFeatures'
DEBUG: 00:00:01.9138361 INVOKING PowerShell Fn Get-Feature with args that has length 0
DEBUG: 00:00:01.9752295 In PowerShellGet Provider - 'Get-Feature'.
DEBUG: 00:00:01.9963670 Done calling powershell «Get-Feature» «PSModule»
DEBUG: 00:00:01.9965014 The provider 'PowerShellGet' is imported
DEBUG: 00:00:01.9980914 SUCCESS provider assembly: C:\Program
Files\WindowsPowerShell\Modules\PackageManagement\1.4.8.1\fullclr\Microsoft.PackageManagement.MetaProvider.PowerShell.d
ll
DEBUG: 00:00:02.0449944 Calling New() : MethodName = 'GetDynamicOptions'
DEBUG: 00:00:02.0460298 INVOKING PowerShell Fn Get-DynamicOptions with args Provider that has length 1
DEBUG: 00:00:02.1126328 In PowerShellGet Provider - 'Get-DynamicOptions'.
DEBUG: 00:00:02.1409408 Done calling powershell «Get-DynamicOptions» «PSModule»
DEBUG: 00:00:02.1457839 Calling New() : MethodName = 'GetDynamicOptions'
DEBUG: 00:00:02.1464532 INVOKING PowerShell Fn Get-DynamicOptions with args Install that has length 1
DEBUG: 00:00:02.1472735 In PowerShellGet Provider - 'Get-DynamicOptions'.
DEBUG: 00:00:02.1567942 Done calling powershell «Get-DynamicOptions» «PSModule»
DEBUG: 00:00:02.3793958 Calling New() : MethodName = 'GetInstalledPackages'
DEBUG: 00:00:02.3810570 MessageResolver: Microsoft.PowerShell.PackageManagement.Cmdlets.GetMessageString
DEBUG: 00:00:02.3826618 ProviderName: PowerShellGet
DEBUG: 00:00:02.3839151 Type: Module
DEBUG: 00:00:02.3853925 INVOKING PowerShell Fn Get-InstalledPackage with args , , , that has length 4
DEBUG: 00:00:02.4558924 In PowerShellGet Provider - 'Get-InstalledPackage'.
DEBUG: 00:00:02.4607167 OPTION: MessageResolver => Microsoft.PowerShell.PackageManagement.Cmdlets.GetMessageString
DEBUG: 00:00:02.4620213 OPTION: ProviderName => PowerShellGet
DEBUG: 00:00:02.4648511 OPTION: Type => Module
DEBUG: 00:00:04.7588463 Done calling powershell «Get-InstalledPackage» «PSModule»
Version Name Repository Description

4.1.220 AWSPowerShell PSGallery The AWS Tools for Windows PowerShell l...
1.0.0.8 DockerMsftProvider PSGallery PowerShell module with commands for di...
1.9 MarkdownPS PSGallery A module to help render Markdown from ...
1.18.0 Microsoft.Graph PSGallery Microsoft Graph PowerShell module
1.18.0 Microsoft.Graph.Applications PSGallery Microsoft Graph PowerShell Cmdlets
1.18.0 Microsoft.Graph.Authentication PSGallery Microsoft Graph PowerShell Authenticat...
1.18.0 Microsoft.Graph.Bookings PSGallery Microsoft Graph PowerShell Cmdlets
1.18.0 Microsoft.Graph.Calendar PSGallery Microsoft Graph PowerShell Cmdlets
1.18.0 Microsoft.Graph.ChangeNotifications PSGallery Microsoft Graph PowerShell Cmdlets
1.18.0 Microsoft.Graph.CloudCommunications PSGallery Microsoft Graph PowerShell Cmdlets
1.18.0 Microsoft.Graph.Compliance PSGallery Microsoft Graph PowerShell Cmdlets
1.18.0 Microsoft.Graph.CrossDeviceExper... PSGallery Microsoft Graph PowerShell Cmdlets
1.18.0 Microsoft.Graph.DeviceManagement PSGallery Microsoft Graph PowerShell Cmdlets
1.18.0 Microsoft.Graph.DeviceManagement... PSGallery Microsoft Graph PowerShell Cmdlets
1.18.0 Microsoft.Graph.DeviceManagement... PSGallery Microsoft Graph PowerShell Cmdlets
1.18.0 Microsoft.Graph.DeviceManagement... PSGallery Microsoft Graph PowerShell Cmdlets
1.18.0 Microsoft.Graph.DeviceManagement... PSGallery Microsoft Graph PowerShell Cmdlets
1.18.0 Microsoft.Graph.Devices.CloudPrint PSGallery Microsoft Graph PowerShell Cmdlets
1.18.0 Microsoft.Graph.Devices.Corporat... PSGallery Microsoft Graph PowerShell Cmdlets
1.18.0 Microsoft.Graph.Devices.ServiceA... PSGallery Microsoft Graph PowerShell Cmdlets
1.18.0 Microsoft.Graph.DirectoryObjects PSGallery Microsoft Graph PowerShell Cmdlets
1.18.0 Microsoft.Graph.Education PSGallery Microsoft Graph PowerShell Cmdlets
1.18.0 Microsoft.Graph.Files PSGallery Microsoft Graph PowerShell Cmdlets
1.18.0 Microsoft.Graph.Financials PSGallery Microsoft Graph PowerShell Cmdlets
1.18.0 Microsoft.Graph.Groups PSGallery Microsoft Graph PowerShell Cmdlets
1.18.0 Microsoft.Graph.Identity.Directo... PSGallery Microsoft Graph PowerShell Cmdlets
1.18.0 Microsoft.Graph.Identity.Governance PSGallery Microsoft Graph PowerShell Cmdlets
1.18.0 Microsoft.Graph.Identity.SignIns PSGallery Microsoft Graph PowerShell Cmdlets
1.18.0 Microsoft.Graph.Mail PSGallery Microsoft Graph PowerShell Cmdlets
1.18.0 Microsoft.Graph.ManagedTenants PSGallery Microsoft Graph PowerShell Cmdlets
1.18.0 Microsoft.Graph.Notes PSGallery Microsoft Graph PowerShell Cmdlets
1.18.0 Microsoft.Graph.People PSGallery Microsoft Graph PowerShell Cmdlets
1.18.0 Microsoft.Graph.PersonalContacts PSGallery Microsoft Graph PowerShell Cmdlets
1.18.0 Microsoft.Graph.Planner PSGallery Microsoft Graph PowerShell Cmdlets
1.18.0 Microsoft.Graph.Reports PSGallery Microsoft Graph PowerShell Cmdlets
1.18.0 Microsoft.Graph.SchemaExtensions PSGallery Microsoft Graph PowerShell Cmdlets
1.18.0 Microsoft.Graph.Search PSGallery Microsoft Graph PowerShell Cmdlets
1.18.0 Microsoft.Graph.Security PSGallery Microsoft Graph PowerShell Cmdlets
1.18.0 Microsoft.Graph.Sites PSGallery Microsoft Graph PowerShell Cmdlets
1.18.0 Microsoft.Graph.Teams PSGallery Microsoft Graph PowerShell Cmdlets
1.18.0 Microsoft.Graph.Users PSGallery Microsoft Graph PowerShell Cmdlets
1.18.0 Microsoft.Graph.Users.Actions PSGallery Microsoft Graph PowerShell Cmdlets
1.18.0 Microsoft.Graph.Users.Functions PSGallery Microsoft Graph PowerShell Cmdlets
1.18.0 Microsoft.Graph.WindowsUpdates PSGallery Microsoft Graph PowerShell Cmdlets
1.4.8.1 PackageManagement PSGallery PackageManagement (a.k.a. OneGet) is a...
5.3.3 Pester PSGallery Pester provides a framework for runnin...
2.2.5 PowerShellGet PSGallery PowerShell module with commands for di...
1.21.0 PSScriptAnalyzer PSGallery PSScriptAnalyzer provides script analy...
2.2.0.3 PSWindowsUpdate PSGallery This module contain cmdlets to manage ...
21.1.18256 SqlServer PSGallery This module allows SQL Server develope...
2.2.16 VSSetup PSGallery Visual Studio Setup PowerShell Module
2.10.4 Az.Accounts PSGallery Microsoft Azure PowerShell - Accounts ...
1.16.11 Az.DataFactory PSGallery Microsoft Azure PowerShell - Data Fact...
6.5.0 Az.Resources PSGallery Microsoft Azure PowerShell - Azure Res...
WARNING: Both Az and AzureRM modules were detected on this machine. Az and AzureRM modules cannot be imported in the
same session or used in the same script or runbook. If you are running PowerShell in an environment you control you can
use the 'Uninstall-AzureRm' cmdlet to remove all AzureRm modules from your machine. If you are running in Azure
Automation, take care that none of your runbooks import both Az and AzureRM modules. More information can be found
here: https://aka.ms/azps-migration-guide
DEBUG: Got version 0 of Az
DEBUG: Got version 0 of Az.Accounts
....
DEBUG: AzureQoSEvent: Module: Az.DataFactory:1.16.11; CommandName: Get-AzDataFactoryV2; PSVersion: 5.1.20348.859;
IsSuccess: True; Duration: 00:00:00.4532128
DEBUG: 10:43:09 AM - [ConfigManager] Got nothing from [EnableDataCollection], Module = [], Cmdlet = []. Returning
default value [True].
DEBUG: 10:43:09 AM - GetAzureDataFactoryCommand end processing.
Data Factory wb-dp-dev-df03 already exists
Deploying sub-area file: D:\a\r1\a/_DeploymentFiles/_deploymentFiles\SubAreaFiles\deployment.opera.json
Substituting variables for object [ATEStore_Common_DPExtract_Conformed_to_Clean]
Saving object [ATEStore_Common_DPExtract_Conformed_to_Clean] to file [C:\Users\VssAdministrator\AppData\Local\Temp\tmp582E.tmp]
Deploying Pipeline ATEStore_Common_DPExtract_Conformed_to_Clean from file C:\Users\VssAdministrator\AppData\Local\Temp\tmp582E.tmp
properties.activities before deployment
name : Load_Conformed_to_Clean
type : DatabricksNotebook
dependsOn : {}
policy : @{timeout=03:00:00; retry=0; retryIntervalInSeconds=30; secureOutput=False; secureInput=False}
userProperties : {}
typeProperties : @{notebookPath=/Notebooks/ATEStore/LoadScripts/ATE DPExtract; baseParameters=}
linkedServiceName : @{referenceName=LS_Databricks_Dynamic; type=LinkedServiceReference}
DEBUG: 10:43:12 AM - NewAzureResourceCmdlet begin processing with ParameterSet 'BySubscriptionLevel'.
DEBUG: 10:43:12 AM - using account id ''...
DEBUG: 10:43:12 AM - [ConfigManager] Got nothing from [DisplayBreakingChangeWarning], Module = [], Cmdlet = [].
Returning default value [True].
DEBUG: [Common.Authentication]: Authenticating using Account: '', environment:
'AzureCloud', tenant: 'fdfed904-9e03-4e17-89c4-61053e0777be'
DEBUG: 10:43:12 AM - [ServicePrincipalAuthenticator] Calling ClientSecretCredential.GetTokenAsync -
ApplicationId:'', TenantId:'fdfed904-9e03-4e17-89c4-61053e0777be',
Scopes:'https://management.core.windows.net//.default', AuthorityHost:'https://login.microsoftonline.com/'
DEBUG: ClientSecretCredential.GetToken invoked. Scopes: [ https://management.core.windows.net//.default ]
ParentRequestId:
DEBUG: False MSAL 4.46.2.0 MSAL.Desktop 4.8 or later Windows Server 2022 Datacenter [2022-12-08 10:43:12Z -
f3cff47e-0395-4390-b171-47803b0dbfde] MSAL MSAL.Desktop with assembly version '4.46.2.0'.
CorrelationId(f3cff47e-0395-4390-b171-47803b0dbfde)
DEBUG: False MSAL 4.46.2.0 MSAL.Desktop 4.8 or later Windows Server 2022 Datacenter [2022-12-08 10:43:12Z -
f3cff47e-0395-4390-b171-47803b0dbfde] === AcquireTokenForClientParameters ===
SendX5C: False
ForceRefresh: False
DEBUG: False MSAL 4.46.2.0 MSAL.Desktop 4.8 or later Windows Server 2022 Datacenter [2022-12-08 10:43:12Z -
f3cff47e-0395-4390-b171-47803b0dbfde]
=== Request Data ===
Authority Provided? - True
Scopes - https://management.core.windows.net//.default
Extra Query Params Keys (space separated) -
ApiId - AcquireTokenForClient
IsConfidentialClient - True
SendX5C - False
LoginHint ? False
IsBrokerConfigured - False
HomeAccountId - False
CorrelationId - f3cff47e-0395-4390-b171-47803b0dbfde
UserAssertion set: False
LongRunningOboCacheKey set: False
Region configured:
DEBUG: False MSAL 4.46.2.0 MSAL.Desktop 4.8 or later Windows Server 2022 Datacenter [2022-12-08 10:43:12Z -
f3cff47e-0395-4390-b171-47803b0dbfde] === Token Acquisition (ClientCredentialRequest) started:
Scopes: https://management.core.windows.net//.default
Authority Host: login.microsoftonline.com
DEBUG: False MSAL 4.46.2.0 MSAL.Desktop 4.8 or later Windows Server 2022 Datacenter [2022-12-08 10:43:12Z -
f3cff47e-0395-4390-b171-47803b0dbfde] [Region discovery] Not using a regional authority.
DEBUG: Request [fa74a047-323a-4b91-8550-98839001c31c] POST
https://login.microsoftonline.com/fdfed904-9e03-4e17-89c4-61053e0777be/oauth2/v2.0/token
x-client-SKU:REDACTED
x-client-Ver:REDACTED
x-client-CPU:REDACTED
x-client-OS:REDACTED
x-client-current-telemetry:REDACTED
x-client-last-telemetry:REDACTED
x-ms-lib-capability:REDACTED
client-request-id:REDACTED
return-client-request-id:REDACTED
x-app-name:REDACTED
x-app-ver:REDACTED
Content-Type:application/x-www-form-urlencoded
x-ms-client-request-id:fa74a047-323a-4b91-8550-98839001c31c
x-ms-return-client-request-id:true
User-Agent:azsdk-net-Identity/1.6.1 (.NET Framework 4.8.9105.0; Microsoft Windows 10.0.20348 )
client assembly: Azure.Identity
DEBUG: Response [fa74a047-323a-4b91-8550-98839001c31c] 200 OK (00.2s)
Pragma:no-cache
Strict-Transport-Security:REDACTED
X-Content-Type-Options:REDACTED
client-request-id:REDACTED
x-ms-request-id:8735ac4b-c670-43c2-ba89-5624bdaf6000
x-ms-ests-server:REDACTED
x-ms-clitelem:REDACTED
X-XSS-Protection:REDACTED
Cache-Control:no-store, no-cache
Content-Type:application/json; charset=utf-8
Expires:-1
P3P:REDACTED
Set-Cookie:REDACTED
Date:Thu, 08 Dec 2022 10:43:12 GMT
Content-Length:1379
DEBUG: False MSAL 4.46.2.0 MSAL.Desktop 4.8 or later Windows Server 2022 Datacenter [2022-12-08 10:43:13Z -
f3cff47e-0395-4390-b171-47803b0dbfde] ScopeSet was missing from the token response, so using developer provided scopes
in the result.
DEBUG: False MSAL 4.46.2.0 MSAL.Desktop 4.8 or later Windows Server 2022 Datacenter [2022-12-08 10:43:13Z -
f3cff47e-0395-4390-b171-47803b0dbfde] Checking client info returned from the server..
DEBUG: False MSAL 4.46.2.0 MSAL.Desktop 4.8 or later Windows Server 2022 Datacenter [2022-12-08 10:43:13Z -
f3cff47e-0395-4390-b171-47803b0dbfde] Saving token response to cache..
DEBUG: False MSAL 4.46.2.0 MSAL.Desktop 4.8 or later Windows Server 2022 Datacenter [2022-12-08 10:43:13Z -
f3cff47e-0395-4390-b171-47803b0dbfde] [SaveTokenResponseAsync] ID Token not present in response.
DEBUG: False MSAL 4.46.2.0 MSAL.Desktop 4.8 or later Windows Server 2022 Datacenter [2022-12-08 10:43:13Z -
f3cff47e-0395-4390-b171-47803b0dbfde] Cannot determine home account id - or id token or no client info and no subject
DEBUG: False MSAL 4.46.2.0 MSAL.Desktop 4.8 or later Windows Server 2022 Datacenter [2022-12-08 10:43:13Z -
f3cff47e-0395-4390-b171-47803b0dbfde] [SaveTokenResponseAsync] Saving AT in cache and removing overlapping ATs...
DEBUG: False MSAL 4.46.2.0 MSAL.Desktop 4.8 or later Windows Server 2022 Datacenter [2022-12-08 10:43:13Z -
f3cff47e-0395-4390-b171-47803b0dbfde] Looking for scopes for the authority in the cache which intersect with
https://management.core.windows.net//.default
DEBUG: False MSAL 4.46.2.0 MSAL.Desktop 4.8 or later Windows Server 2022 Datacenter [2022-12-08 10:43:13Z -
f3cff47e-0395-4390-b171-47803b0dbfde] Intersecting scope entries count - 0
DEBUG: False MSAL 4.46.2.0 MSAL.Desktop 4.8 or later Windows Server 2022 Datacenter [2022-12-08 10:43:13Z -
f3cff47e-0395-4390-b171-47803b0dbfde]
=== Token Acquisition finished successfully:
DEBUG: False MSAL 4.46.2.0 MSAL.Desktop 4.8 or later Windows Server 2022 Datacenter [2022-12-08 10:43:13Z -
f3cff47e-0395-4390-b171-47803b0dbfde] AT expiration time: 12/8/2022 11:43:12 AM +00:00, scopes:
https://management.core.windows.net//.default. source: IdentityProvider
DEBUG: False MSAL 4.46.2.0 MSAL.Desktop 4.8 or later Windows Server 2022 Datacenter [2022-12-08 10:43:13Z -
f3cff47e-0395-4390-b171-47803b0dbfde] Fetched access token from host login.microsoftonline.com.
DEBUG: ClientSecretCredential.GetToken succeeded. Scopes: [ https://management.core.windows.net//.default ]
ParentRequestId: ExpiresOn: 2022-12-08T11:43:12.1363674+00:00
DEBUG: [Common.Authentication]: Received token with LoginType 'User', Tenant: 'fdfed904-9e03-4e17-89c4-61053e0777be',
UserId: ''
DEBUG: ============================ HTTP REQUEST ============================

HTTP Method:
PUT

Absolute Uri:
https://management.azure.com/subscriptions/b976df9a-6270-4417-affb-ec653aeb3aab/resourceGroups/wbdp-dev-rg/providers/Mi
crosoft.DataFactory/factories/wb-dp-dev-df03/pipelines/ATEStore_Common_DPExtract_Conformed_to_Clean?api-version=2018-06
-01

Headers:
User-Agent : AzurePowershell/v0.0.0,PSVersion/v5.1.20348.859,Az.Resources/6.5.0
ParameterSetName : BySubscriptionLevel
CommandName : New-AzResource

Body:
{
"name": "ATEStore_Common_DPExtract_Conformed_to_Clean",
"properties": {
"activities": [
{
"name": "Load_Conformed_to_Clean",
"type": "DatabricksNotebook",
"dependsOn": [],
"policy": {
"timeout": "03:00:00",
"retry": 0,
"retryIntervalInSeconds": 30,
"secureOutput": false,
"secureInput": false
},
"userProperties": [],
"typeProperties": {
"notebookPath": "/Notebooks/ATEStore/LoadScripts/ATEDPExtract",
"baseParameters": {
"OutputPath": {
"value": "@pipeline().parameters.OutputPath",
"type": "Expression"
},
"ProcessName": {
"value": "@pipeline().parameters.ProcessName",
"type": "Expression"
},
"SliceDate": {
"value": "@formatDateTime(pipeline().parameters.SliceDate)",
"type": "Expression"
},
"OutputHeader": {
"value": "@pipeline().parameters.OutputHeader",
"type": "Expression"
},
"QuoteAllColumns": {
"value": "@pipeline().parameters.QuoteAllColumns",
"type": "Expression"
},
"QuoteCharacter": {
"value": "@pipeline().parameters.QuoteCharacter",
"type": "Expression"
},
"DataLakeArea": {
"value": "@pipeline().parameters.DataLakeArea",
"type": "Expression"
},
"Encoding": {
"value": "@pipeline().parameters.Encoding",
"type": "Expression"
},
"IgnoreLeadingWhitepace": {
"value": "@pipeline().parameters.ignoreLeadingWhitepace",
"type": "Expression"
},
"IgnoreTrailingWhiteSpace": {
"value": "@pipeline().parameters.IgnoreTrailingWhiteSpace",
"type": "Expression"
},
"FileSeparator": {
"value": "@pipeline().parameters.FileSeparator",
"type": "Expression"
}
}
},
"linkedServiceName": {
"referenceName": "LS_Databricks_Dynamic",
"type": "LinkedServiceReference"
}
}
],
"parameters": {
"OutputPath": {
"type": "string"
},
"ProcessName": {
"type": "string"
},
"SliceDate": {
"type": "string"
},
"OutputHeader": {
"type": "string",
"defaultValue": "true"
},
"QuoteAllColumns": {
"type": "string",
"defaultValue": "false"
},
"QuoteCharacter": {
"type": "string",
"defaultValue": """
},
"DataLakeArea": {
"type": "string",
"defaultValue": "clean"
},
"Encoding": {
"type": "string",
"defaultValue": "UTF-8"
},
"IgnoreLeadingWhitepace": {
"type": "string",
"defaultValue": "true"
},
"IgnoreTrailingWhiteSpace": {
"type": "string",
"defaultValue": "true"
},
"FileSeparator": {
"type": "string",
"defaultValue": ","
}
},
"folder": {
"name": "ATEStore/Common"
},
"annotations": [],
"lastPublishTime": "2022-10-18T20:23:40Z"
},
"type": "Microsoft.DataFactory/factories/pipelines"
}

DEBUG: ============================ HTTP RESPONSE ============================

Status Code:
OK

Headers:
Pragma : no-cache
x-ms-correlation-request-id : c9e347a6-2df5-4a61-9b91-d015998f838e
x-ms-ratelimit-remaining-subscription-writes: 1199
x-ms-request-id : c9e347a6-2df5-4a61-9b91-d015998f838e
x-ms-routing-request-id : NORTHEUROPE:20221208T104313Z:c9e347a6-2df5-4a61-9b91-d015998f838e
Strict-Transport-Security : max-age=31536000; includeSubDomains
X-Content-Type-Options : nosniff
Cache-Control : no-cache
Date : Thu, 08 Dec 2022 10:43:12 GMT
Server : Kestrel

Body:
{
"id":
"/subscriptions/b976df9a-6270-4417-affb-ec653aeb3aab/resourceGroups/wbdp-dev-rg/providers/Microsoft.DataFactory/factori
es/wb-dp-dev-df03/pipelines/ATEStore_Common_DPExtract_Conformed_to_Clean",
"name": "ATEStore_Common_DPExtract_Conformed_to_Clean",
"type": "Microsoft.DataFactory/factories/pipelines",
"properties": {
"activities": [
{
"name": "Load_Conformed_to_Clean",
"type": "DatabricksNotebook",
"dependsOn": [],
"policy": {
"timeout": "03:00:00",
"retry": 0,
"retryIntervalInSeconds": 30,
"secureOutput": false,
"secureInput": false
},
"userProperties": [],
"typeProperties": {
"notebookPath": "/Notebooks/ATEStore/LoadScripts/ATEDPExtract",
"baseParameters": {
"OutputPath": {
"value": "@pipeline().parameters.OutputPath",
"type": "Expression"
},
"ProcessName": {
"value": "@pipeline().parameters.ProcessName",
"type": "Expression"
},
"SliceDate": {
"value": "@formatDateTime(pipeline().parameters.SliceDate)",
"type": "Expression"
},
"OutputHeader": {
"value": "@pipeline().parameters.OutputHeader",
"type": "Expression"
},
"QuoteAllColumns": {
"value": "@pipeline().parameters.QuoteAllColumns",
"type": "Expression"
},
"QuoteCharacter": {
"value": "@pipeline().parameters.QuoteCharacter",
"type": "Expression"
},
"DataLakeArea": {
"value": "@pipeline().parameters.DataLakeArea",
"type": "Expression"
},
"Encoding": {
"value": "@pipeline().parameters.Encoding",
"type": "Expression"
},
"IgnoreLeadingWhitepace": {
"value": "@pipeline().parameters.ignoreLeadingWhitepace",
"type": "Expression"
},
"IgnoreTrailingWhiteSpace": {
"value": "@pipeline().parameters.IgnoreTrailingWhiteSpace",
"type": "Expression"
},
"FileSeparator": {
"value": "@pipeline().parameters.FileSeparator",
"type": "Expression"
}
}
},
"linkedServiceName": {
"referenceName": "LS_Databricks_Dynamic",
"type": "LinkedServiceReference"
}
}
],
"parameters": {
"OutputPath": {
"type": "string"
},
"ProcessName": {
"type": "string"
},
"SliceDate": {
"type": "string"
},
"OutputHeader": {
"type": "string",
"defaultValue": "true"
},
"QuoteAllColumns": {
"type": "string",
"defaultValue": "false"
},
"QuoteCharacter": {
"type": "string",
"defaultValue": """
},
"DataLakeArea": {
"type": "string",
"defaultValue": "clean"
},
"Encoding": {
"type": "string",
"defaultValue": "UTF-8"
},
"IgnoreLeadingWhitepace": {
"type": "string",
"defaultValue": "true"
},
"IgnoreTrailingWhiteSpace": {
"type": "string",
"defaultValue": "true"
},
"FileSeparator": {
"type": "string",
"defaultValue": ","
}
},
"folder": {
"name": "ATEStore/Common"
},
"annotations": [],
"lastPublishTime": "2022-10-18T20:23:40Z"
},
"etag": "5f005e1b-0000-0c00-0000-6391bfc10000"
}

DEBUG: [Common.Authentication]: Authenticating using Account: '', environment:
'AzureCloud', tenant: 'fdfed904-9e03-4e17-89c4-61053e0777be'
DEBUG: 10:43:13 AM - [ServicePrincipalAuthenticator] Calling ClientSecretCredential.GetTokenAsync -
ApplicationId:'', TenantId:'fdfed904-9e03-4e17-89c4-61053e0777be',
Scopes:'https://management.core.windows.net//.default', AuthorityHost:'https://login.microsoftonline.com/'
DEBUG: ClientSecretCredential.GetToken invoked. Scopes: [ https://management.core.windows.net//.default ]
ParentRequestId:
DEBUG: False MSAL 4.46.2.0 MSAL.Desktop 4.8 or later Windows Server 2022 Datacenter [2022-12-08 10:43:13Z -
50a8ae38-cd21-4097-9569-ad224121bde2] MSAL MSAL.Desktop with assembly version '4.46.2.0'.
CorrelationId(50a8ae38-cd21-4097-9569-ad224121bde2)
DEBUG: False MSAL 4.46.2.0 MSAL.Desktop 4.8 or later Windows Server 2022 Datacenter [2022-12-08 10:43:13Z -
50a8ae38-cd21-4097-9569-ad224121bde2] === AcquireTokenForClientParameters ===
SendX5C: False
ForceRefresh: False
DEBUG: False MSAL 4.46.2.0 MSAL.Desktop 4.8 or later Windows Server 2022 Datacenter [2022-12-08 10:43:13Z -
50a8ae38-cd21-4097-9569-ad224121bde2]
=== Request Data ===
Authority Provided? - True
Scopes - https://management.core.windows.net//.default
Extra Query Params Keys (space separated) -
ApiId - AcquireTokenForClient
IsConfidentialClient - True
SendX5C - False
LoginHint ? False
IsBrokerConfigured - False
HomeAccountId - False
CorrelationId - 50a8ae38-cd21-4097-9569-ad224121bde2
UserAssertion set: False
LongRunningOboCacheKey set: False
Region configured:
DEBUG: False MSAL 4.46.2.0 MSAL.Desktop 4.8 or later Windows Server 2022 Datacenter [2022-12-08 10:43:13Z -
50a8ae38-cd21-4097-9569-ad224121bde2] === Token Acquisition (ClientCredentialRequest) started:
Scopes: https://management.core.windows.net//.default
Authority Host: login.microsoftonline.com
DEBUG: False MSAL 4.46.2.0 MSAL.Desktop 4.8 or later Windows Server 2022 Datacenter [2022-12-08 10:43:13Z -
50a8ae38-cd21-4097-9569-ad224121bde2] [Region discovery] Not using a regional authority.
DEBUG: Request [5a1c3cb6-3961-40ab-bd7b-b8090139ea44] POST
https://login.microsoftonline.com/fdfed904-9e03-4e17-89c4-61053e0777be/oauth2/v2.0/token
x-client-SKU:REDACTED
x-client-Ver:REDACTED
x-client-CPU:REDACTED
x-client-OS:REDACTED
x-client-current-telemetry:REDACTED
x-client-last-telemetry:REDACTED
x-ms-lib-capability:REDACTED
client-request-id:REDACTED
return-client-request-id:REDACTED
x-app-name:REDACTED
x-app-ver:REDACTED
Content-Type:application/x-www-form-urlencoded
x-ms-client-request-id:5a1c3cb6-3961-40ab-bd7b-b8090139ea44
x-ms-return-client-request-id:true
User-Agent:azsdk-net-Identity/1.6.1 (.NET Framework 4.8.9105.0; Microsoft Windows 10.0.20348 )
client assembly: Azure.Identity
DEBUG: Response [5a1c3cb6-3961-40ab-bd7b-b8090139ea44] 200 OK (00.1s)
Pragma:no-cache
Strict-Transport-Security:REDACTED
X-Content-Type-Options:REDACTED
client-request-id:REDACTED
x-ms-request-id:d4298b78-4a10-4f8c-8f25-57efd13d3a00
x-ms-ests-server:REDACTED
x-ms-clitelem:REDACTED
X-XSS-Protection:REDACTED
Cache-Control:no-store, no-cache
Content-Type:application/json; charset=utf-8
Expires:-1
P3P:REDACTED
Set-Cookie:REDACTED
Date:Thu, 08 Dec 2022 10:43:12 GMT
Content-Length:1379
DEBUG: False MSAL 4.46.2.0 MSAL.Desktop 4.8 or later Windows Server 2022 Datacenter [2022-12-08 10:43:13Z -
50a8ae38-cd21-4097-9569-ad224121bde2] ScopeSet was missing from the token response, so using developer provided scopes
in the result.
DEBUG: False MSAL 4.46.2.0 MSAL.Desktop 4.8 or later Windows Server 2022 Datacenter [2022-12-08 10:43:13Z -
50a8ae38-cd21-4097-9569-ad224121bde2] Checking client info returned from the server..
DEBUG: False MSAL 4.46.2.0 MSAL.Desktop 4.8 or later Windows Server 2022 Datacenter [2022-12-08 10:43:13Z -
50a8ae38-cd21-4097-9569-ad224121bde2] Saving token response to cache..
DEBUG: False MSAL 4.46.2.0 MSAL.Desktop 4.8 or later Windows Server 2022 Datacenter [2022-12-08 10:43:13Z -
50a8ae38-cd21-4097-9569-ad224121bde2] [SaveTokenResponseAsync] ID Token not present in response.
DEBUG: False MSAL 4.46.2.0 MSAL.Desktop 4.8 or later Windows Server 2022 Datacenter [2022-12-08 10:43:13Z -
50a8ae38-cd21-4097-9569-ad224121bde2] Cannot determine home account id - or id token or no client info and no subject
DEBUG: False MSAL 4.46.2.0 MSAL.Desktop 4.8 or later Windows Server 2022 Datacenter [2022-12-08 10:43:13Z -
50a8ae38-cd21-4097-9569-ad224121bde2] [SaveTokenResponseAsync] Saving AT in cache and removing overlapping ATs...
DEBUG: False MSAL 4.46.2.0 MSAL.Desktop 4.8 or later Windows Server 2022 Datacenter [2022-12-08 10:43:13Z -
50a8ae38-cd21-4097-9569-ad224121bde2] Looking for scopes for the authority in the cache which intersect with
https://management.core.windows.net//.default
DEBUG: False MSAL 4.46.2.0 MSAL.Desktop 4.8 or later Windows Server 2022 Datacenter [2022-12-08 10:43:13Z -
50a8ae38-cd21-4097-9569-ad224121bde2] Intersecting scope entries count - 0
DEBUG: False MSAL 4.46.2.0 MSAL.Desktop 4.8 or later Windows Server 2022 Datacenter [2022-12-08 10:43:13Z -
50a8ae38-cd21-4097-9569-ad224121bde2]
=== Token Acquisition finished successfully:
DEBUG: False MSAL 4.46.2.0 MSAL.Desktop 4.8 or later Windows Server 2022 Datacenter [2022-12-08 10:43:13Z -
50a8ae38-cd21-4097-9569-ad224121bde2] AT expiration time: 12/8/2022 11:43:12 AM +00:00, scopes:
https://management.core.windows.net//.default. source: IdentityProvider
DEBUG: False MSAL 4.46.2.0 MSAL.Desktop 4.8 or later Windows Server 2022 Datacenter [2022-12-08 10:43:13Z -
50a8ae38-cd21-4097-9569-ad224121bde2] Fetched access token from host login.microsoftonline.com.
DEBUG: ClientSecretCredential.GetToken succeeded. Scopes: [ https://management.core.windows.net//.default ]
ParentRequestId: ExpiresOn: 2022-12-08T11:43:12.5856879+00:00
DEBUG: [Common.Authentication]: Received token with LoginType 'User', Tenant: 'fdfed904-9e03-4e17-89c4-61053e0777be',
UserId: '***'
Name : ATEStore_Common_DPExtract_Conformed_to_Clean
ResourceId : /subscriptions/b976df9a-6270-4417-affb-ec653aeb3aab/resourceGroups/wbdp-dev-rg/providers/Microsoft.
DataFactory/factories/wb-dp-dev-df03/pipelines/ATEStore_Common_DPExtract_Conformed_to_Clean
ResourceName : wb-dp-dev-df03/ATEStore_Common_DPExtract_Conformed_to_Clean
ResourceType : Microsoft.DataFactory/factories/pipelines
ResourceGroupName : wbdp-dev-rg
SubscriptionId : b976df9a-6270-4417-affb-ec653aeb3aab
Properties : @{activities=System.Object[]; parameters=; folder=; annotations=System.Object[];
lastPublishTime=2022-10-18T20:23:40Z}
ETag : 5f005e1b-0000-0c00-0000-6391bfc10000
DEBUG: AzureQoSEvent: Module: Az.Resources:6.5.0; CommandName: New-AzResource; PSVersion: 5.1.20348.859; IsSuccess:
True; Duration: 00:00:00.8099074
DEBUG: 10:43:13 AM - [ConfigManager] Got nothing from [EnableDataCollection], Module = [], Cmdlet = []. Returning
default value [True].
DEBUG: 10:43:13 AM - NewAzureResourceCmdlet end processing.

##[section]Finishing: Deploy Data Factory Objects
`

@ghost ghost added needs-team-attention This issue needs attention from Azure service team or SDK team and removed needs-author-feedback More information is needed from author to address the issue. labels Dec 8, 2022
@rachelxj-ms
Copy link
Contributor

This should be fixed in #20399

@RakeshMohanMSFT RakeshMohanMSFT self-assigned this Dec 14, 2022
@RakeshMohanMSFT RakeshMohanMSFT added issue-addressed and removed needs-team-attention This issue needs attention from Azure service team or SDK team labels Dec 14, 2022
@ghost
Copy link

ghost commented Dec 14, 2022

Hi @duttong13. Thank you for opening this issue and giving us the opportunity to assist. We believe that this has been addressed. If you feel that further discussion is needed, please add a comment with the text “/unresolve” to remove the “issue-addressed” label and continue the conversation.

@duttong13
Copy link
Author

@rachelxj-ms Many thanks for mentioning and raising the fix for this. I believe this will address the issue. When will this fix be merged and the version of Az.Resources be incremented?

@haui07 haui07 mentioned this issue Dec 14, 2022
Closed
@dingmeng-xue dingmeng-xue pinned this issue Dec 15, 2022
@ghost
Copy link

ghost commented Dec 22, 2022

Hi @duttong13, since you haven’t asked that we “/unresolve” the issue, we’ll close this out. If you believe further discussion is needed, please add a comment “/unresolve” to reopen the issue.

@ghost ghost closed this as completed Dec 22, 2022
@dingmeng-xue dingmeng-xue mentioned this issue Dec 26, 2022
Closed
@isra-fel isra-fel unpinned this issue May 11, 2023
This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@RakeshMohanMSFT RakeshMohanMSFT

Labels
ARM - Core bug This issue requires a change to an existing behavior in the product in order to be resolved. customer-reported issue-addressed
Projects
None yet
Milestone
January 2023 (2023-01-10)
Development

No branches or pull requests
4 participants
@RakeshMohanMSFT @dingmeng-xue @duttong13 @rachelxj-ms