Update-AzDatabricksWorkspace doesn't work as expected while enabling encryption.

[bobby86]

Description

Azure Databricks workspace is in failed state after enabling DBFS Encryption using Az PowerShell module and script below.
I have tried to enable the encryption for other dev workspace thru GUI, Azure portal, using the key from the same key vault and got the validation error (refer to snap01838).
For recovering the workspace, I have followed the MSFT troubleshooting guidelines (refer to snap01841).
When attempted in GUI, got an error that managed resource group name doesn’t match. (I have created the ADB workspace thru PowerShell and used our default naming convention for the managed resource group.
When attempted the recovery thru PowerShell script below, got a different error and we are not able to locate the failed deployment details.
a MSFT support ticket was raised for this issue and the support engineer reproduced the issue and asked us to follow up with the AZ PowerShell development team.

Issue script & Debug output

# enable Workspace encryption script
# Variables
$nmADBWSP = '*' # ADB Workspace Name
$nmWSPRG = '*' # ADB Workspace ResourceGroupName
$nmDSPRG = '*' # Key Vault ResourceGroupName
$nmKYV = '*' #Key Vault Name

# Configure customer-managed keys for DBFS encryption following: https://learn.microsoft.com/en-us/azure/databricks/security/keys/customer-managed-keys-dbfs/cmk-dbfs-powershell
# Prepare workspace for cutomer managed key encryption
$Wsp = Get-AzDatabricksWorkspace -ResourceGroupName $nmWSPRG -Name $nmADBWSP
#$updWsp = Update-AzDatabricksWorkspace -ResourceGroupName $nmWSPRG -Name $nmADBWSP -PrepareEncryption #doesn't work throws invalid EncryptionKeySource error (refer to snap01822).
$updWsp = Update-AzDatabricksWorkspace -ResourceGroupName $nmWSPRG -Name $nmADBWSP -PrepareEncryption -EncryptionKeySource 'Microsoft.Keyvault'

# retrieve Key Vault details
$kyvDev = Get-AzKeyVault -ResourceGroupName $nmDSPRG -VaultName $nmKYV
# Configure the Key Vault access policy
Set-AzKeyVaultAccessPolicy -VaultName $kyvDev.VaultName -ObjectId $updWsp.StorageAccountIdentityPrincipalId `
  -PermissionsToKeys wrapkey, unwrapkey, get
# Create a new key for the encryption
$key1 = Add-AzKeyVaultKey -VaultName $kyvDev.VaultName -Name "$nmADBWSP-DBFS" -Destination 'Software'
# Update DBFS encryption Key
Update-AzDatabricksWorkspace -ResourceGroupName $nmWSPRG -Name $nmADBWSP -EncryptionKeySource Microsoft.Keyvault `
  -EncryptionKeyName $key1.Name `
  -EncryptionKeyVersion $key1.Version `
  -EncryptionKeyVaultUri $kyvDev.VaultUri # throws error (refer to snap01836).

# recover Workspace script
# Variables
$nmADBWSP = '*' # ADB Workspace Name
$nmWSPRG = '*' # ResourceGroupName to deploy the ADB Workspace
$nmADBRG = '*' # ManagedResourceGroupName
$nmLctn = 'eastus2' # Deployment location
$nmSku = 'premium' # ADB Workspace SKU
$AdbWspArgs = @{
  Name                     = $nmADBWSP
  ResourceGroupName        = $nmWSPRG
  ManagedResourceGroupName = $nmADBRG
  Location                 = $nmLctn
  Sku                      = $nmSku
}
$Wsp = New-AzDatabricksWorkspace @AdbWspArgs ;

Environment data

PowerShell code was run in VS Code:
Version: 1.76.2 (user setup)
Commit: ee2b180d582a7f601fa6ecfdad8d9fd269ab1884
Date: 2023-03-14T17:55:54.936Z
Electron: 19.1.11
Chromium: 102.0.5005.196
Node.js: 16.14.2
V8: 10.2.154.26-electron.0
OS: Windows_NT x64 10.0.19044
Sandboxed: No
 
pwsh terminal-HostVersion
Name             : ConsoleHost
Version          : 7.3.0

Module versions

Az PowerShell module used and their versions:
Az.Accounts  2.12.1
Az.Databricks 1.5.0
Az.KeyVault 4.9.2

Error output

enable encryption Errors in the attached screenshots
workspace recovery error:
New-AzDatabricksWorkspace_CreateExpanded: /usr/local/share/powershell/Modules/Az.Databricks/1.5.0/custom/New-AzDatabricksWorkspace.ps1:299
Line |299 |              Az.Databricks.internal\New-AzDatabricksWorkspace @PSBound …
     |              ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | The operation to create appliance failed. Please check operations of deployment '*' under resource group '/subscriptions/*/resourceGroups/*'.
     | Error message: 'At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.'

[isra-fel]

Hi @lijinpei2008 please look into this issue. Thanks.