Set-AzStorageBlobTag does not allow clearing tags from blob

[johnwc]

Description

The documentation for blob tags states to remove all tags, pass a empty hash array to the API. Doing so in PowerShell causes an error.

$stg | Set-AzStorageBlobTag -Container '$web' -Blob 'my/blob.jpg' -Tag @{}

Set-AzStorageBlobTag: Cannot validate argument on parameter 'Tag'. The argument is null, empty, or an element of the argument collection contains a null value. Supply a collection that does not contain any null values and then try the command again.

Issue script & Debug output

Setting $DebugPreference='Continue' prior to running the cmdlet gives no difference in output.

Environment data

Name                           Value
----                           -----
PSVersion                      7.3.3
PSEdition                      Core
GitCommitId                    7.3.3
OS                             Microsoft Windows 10.0.22621
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

Module versions

ModuleType Version    PreRelease Name                                ExportedCommands
---------- -------    ---------- ----                                ----------------
Script     2.12.1                Az.Accounts                         {Add-AzEnvironment, Clear-AzConfig, Clear-AzContext, Clear-AzDefault…}
Script     5.5.0                 Az.Storage                          {Add-AzRmStorageContainerLegalHold, Add-AzStorageAccountManagementPolicyAction, Add-AzStorageAccountNetwo…

Error output

HistoryId: 41

Message        : Cannot validate argument on parameter 'Tag'. The argument is null, empty, or an element of the argument collection contains a null value. Supply a collection
                 that does not contain any null values and then try the command again.
StackTrace     :    at System.Management.Automation.ParameterBinderBase.BindParameter(CommandParameterInternal parameter, CompiledCommandParameter parameterMetadata,
                 ParameterBindingFlags flags)
                    at System.Management.Automation.CmdletParameterBinderController.BindParameter(CommandParameterInternal argument, MergedCompiledCommandParameter parameter,
                 ParameterBindingFlags flags)
                    at System.Management.Automation.CmdletParameterBinderController.BindParameter(UInt32 parameterSets, CommandParameterInternal argument,
                 MergedCompiledCommandParameter parameter, ParameterBindingFlags flags)
                    at System.Management.Automation.CmdletParameterBinderController.BindNamedParameter(UInt32 parameterSets, CommandParameterInternal argument,
                 MergedCompiledCommandParameter parameter)
                    at System.Management.Automation.ParameterBinderController.BindNamedParameters(UInt32 parameterSets, Collection`1 arguments)
                    at System.Management.Automation.CmdletParameterBinderController.BindCommandLineParametersNoValidation(Collection`1 arguments)
                    at System.Management.Automation.CmdletParameterBinderController.BindCommandLineParameters(Collection`1 arguments)
                    at System.Management.Automation.CommandProcessor.BindCommandLineParameters()
                    at System.Management.Automation.CommandProcessor.Prepare(IDictionary psDefaultParameterValues)
                    at System.Management.Automation.CommandProcessorBase.DoPrepare(IDictionary psDefaultParameterValues)
                    at System.Management.Automation.Internal.PipelineProcessor.Start(Boolean incomingStream)
                    at System.Management.Automation.Internal.PipelineProcessor.SynchronousExecuteEnumerate(Object input)
                 --- End of stack trace from previous location ---
                    at System.Management.Automation.Internal.PipelineProcessor.SynchronousExecuteEnumerate(Object input)
                    at System.Management.Automation.PipelineOps.InvokePipeline(Object input, Boolean ignoreInput, CommandParameterInternal[][] pipeElements, CommandBaseAst[]
                 pipeElementAsts, CommandRedirection[][] commandRedirections, FunctionContext funcContext)
                    at System.Management.Automation.Interpreter.ActionCallInstruction`6.Run(InterpretedFrame frame)
                    at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)
Exception      : System.Management.Automation.ParameterBindingValidationException
InvocationInfo : {Set-AzStorageBlobTag}
Line           : $stg | Set-AzStorageBlobTag -Container '$web' -Blob 'my/blob.jpg' -Tag @{}
Position       : At line:1 char:80
                 + … StorageBlobTag -Container '$web' -Blob 'my/blob.jpg' -Tag @{}
                 +                                                                     ~~~
HistoryId      : 41

Message        : The argument is null, empty, or an element of the argument collection contains a null value. Supply a collection that does not contain any null values and
                 then try the command again.
StackTrace     :    at System.Management.Automation.ValidateNotNullOrEmptyAttribute.Validate(Object arguments, EngineIntrinsics engineIntrinsics)
                    at System.Management.Automation.ParameterBinderBase.BindParameter(CommandParameterInternal parameter, CompiledCommandParameter parameterMetadata,
                 ParameterBindingFlags flags)
Exception      : System.Management.Automation.ValidationMetadataException
InvocationInfo : {Set-AzStorageBlobTag}
Line           : $stg | Set-AzStorageBlobTag -Container '$web' -Blob 'my/blob.jpg' -Tag @{}
Position       : At line:1 char:80
                 + … StorageBlobTag -Container '$web' -Blob 'my/blob.jpg' -Tag @{}
                 +                                                                     ~~~
HistoryId      : 41

[ghost]

Thank you for your feedback. This has been routed to the support team for assistance.

[SaurabhSharma-MSFT]

@johnwc Thanks for your feedback! We will investigate and update as appropriate.

[blueww]

@johnwc

Thanks for the reporting this issue!

I have found the root cause, and will raise a PR to fix it soon.

Besides that, before the issue is fixed, you can clean the tags on a blob with following PSH script:

$blob = Get-AzStorageBlob -Container '$web' -Blob 'my/blob.jpg' -Context $ctx
$Tags = New-Object System.Collections.Generic.Dictionary"[String,String]"
$blob.BlobBaseClient.SetTags($Tags)

[johnwc]

@blueww should there be a Delete-AzStorageBlobTags for this, instead of allowing a empty array?

[blueww]

@johnwc

"Blob Tags" is a whole object, we can only set/get it as a whole from server.
Currently we don't have Add/Remove blob tag cmdlet to be aligned with server. (Add these cmdlets will 1. raise the PSH maintain cost as it not aligned with server API; 2. raise the total storage cmdlets count which might will make customer not easy to find the cmdlet they need.)
Do you have any concern to clean blob tag by set an empty array?

[johnwc]

@blueww we have 100k+ blobs, was hoping not to have to pull the entire object for each.

[blueww]

@johnwc

Do you mean you don't want to Run Get-AzStorageBlob on each blob?
You can also list the blob object then set blob tag.
(There are many ways to get blob object, "Get-AzStorageBlob" on single blob is just a sample.)

List following script will first list all blobs in single container, then clear tags on each blob.

$blobs = Get-AzStorageBlob -Container '$web' -Context $ctx
$Tags = New-Object System.Collections.Generic.Dictionary"[String,String]"
foreach ($blob in $blobs)
{
    $blob.BlobBaseClient.SetTags($Tags)
}

[johnwc]

@blueww I'm not seeing the cmdlet using the correct endpoint when trying to list blobs by tag. I actually don't see it using the TagCondition at all.

It's calling https://xxxxxx.blob.core.windows.net/$web?restype=container&comp=list&prefix=&maxresults=12&include=Copy%2CMetadata%2CSnapshots when it should be calling https://myaccount.blob.core.windows.net/?comp=blobs&where=<expression>

$stg | Get-AzStorageBlob -Container '$web' -MaxCount 12 -TagCondition """TagName"" = 'abc'"
DEBUG: 11:00:59 AM - Init Operation Context for 'GetAzureStorageBlobCommand' with client request id  Azure-Storage-PowerShell-8e00dc58-17b5-4cd0-a416-0ad8d50dcb1c. If you want to get more details, please add "-Debug" to your command.
DEBUG: 11:00:59 AM - GetAzureStorageBlobCommand begin processing with ParameterSet 'BlobName'.
DEBUG: 11:00:59 AM - [ConfigManager] Got nothing from [DisplayBreakingChangeWarning], Module = [], Cmdlet = []. Returning default value [True].
DEBUG: 11:00:59 AM - Use storage account 'xxxxx' from storage context.
DEBUG: 11:00:59 AM - Start 35th remote call, method: , destination: https://xxxxxx.blob.core.windows.net/$web?restype=container.
DEBUG: Request [bf09fe15-ee23-4935-9b61-19799c5d9dc6] GET https://xxxxxx.blob.core.windows.net/$web?restype=container&comp=list&prefix=&maxresults=12&include=Copy%2CMetadata%2CSnapshots
x-ms-version:2021-10-04
Accept:application/xml
User-Agent:AzurePowershell/v1.0.0,azsdk-net-Storage.Blobs/12.14.0,(.NET 7.0.3; Microsoft Windows 10.0.22621)
x-ms-client-request-id:bf09fe15-ee23-4935-9b61-19799c5d9dc6
x-ms-return-client-request-id:true
x-ms-date:Mon, 17 Apr 2023 16:00:59 GMT
Authorization:REDACTED
client assembly: Azure.Storage.Blobs
DEBUG: Response [bf09fe15-ee23-4935-9b61-19799c5d9dc6] 200 OK (00.1s)
Transfer-Encoding:chunked
Server:Windows-Azure-Blob/1.0,Microsoft-HTTPAPI/2.0
x-ms-request-id:17c7e91a-701e-0019-3f45-714f5c000000
x-ms-client-request-id:bf09fe15-ee23-4935-9b61-19799c5d9dc6
x-ms-version:2021-10-04
Date:Mon, 17 Apr 2023 16:01:00 GMT
Content-Type:application/xml
Accept:application/xml
User-Agent:AzurePowershell/v1.0.0,azsdk-net-Storage.Blobs/12.14.0,(.NET 7.0.3; Microsoft Windows 10.0.22621)
x-ms-client-request-id:48d215e2-e8e6-417f-a64f-468b679dcd4b
x-ms-return-client-request-id:true
x-ms-date:Mon, 17 Apr 2023 15:58:34 GMT
Authorization:REDACTED
client assembly: Azure.Storage.Blobs
DEBUG: Response [48d215e2-e8e6-417f-a64f-468b679dcd4b] 200 OK (00.2s)
Transfer-Encoding:chunked
Server:Windows-Azure-Blob/1.0,Microsoft-HTTPAPI/2.0
x-ms-request-id:4f610f1e-401e-004f-5145-71beb3000000
x-ms-client-request-id:48d215e2-e8e6-417f-a64f-468b679dcd4b
x-ms-version:2021-10-04
Date:Mon, 17 Apr 2023 15:58:34 GMT
Content-Type:application/xml

[blueww]

@johnwc

I see you would like to find blob by tag filter, this is a different API from list blob from container, then you should use the following cmdlet ：

$blobs = Get-AzStorageBlobByTag -Container $containerName  -Context $ctx -TagFilterSqlExpression """tag1""='value1'" 

Please note, -Container is an optional parameter. You can use the cmdlet to find blobs by tag from all containers in a storage account or from a single container.

[johnwc]

@johnwc

I see you would like to find blob by tag filter, this is a different API from list blob from container, then you should use the following cmdlet ：

$blobs = Get-AzStorageBlobByTag -Container $containerName  -Context $ctx -TagFilterSqlExpression """tag1""='value1'" 

Please note, -Container is an optional parameter. You can use the cmdlet to find blobs by tag from all containers in a storage account or from a single container.

What's the purpose of TagCondition? Setting it doesn't seem to do anything.

[blueww]

@johnwc

For "-TagCondition" it mostly works on the single blob operation, only the blob matched the tag condition will execute the operation. See : https://learn.microsoft.com/en-us/rest/api/storageservices/specifying-conditional-headers-for-blob-service-operations#tags-conditional-operations

However, to filter multiple blobs from container / account, you should use -TagFilterSqlExpression, which is the search expression, see details in https://learn.microsoft.com/en-us/rest/api/storageservices/find-blobs-by-tags?tabs=azure-ad#constructing-a-search-expression.

BTW, Clear blob Tag with following command is already supported on Az.Storage 5.6.0.

Set-AzStorageBlobTag -Container $containerName -Blob test.txt -Context $ctx -Tag @{}

So I will close this issue.

Feel free to contact us again if you need any further assistance on Azure PowerShell.

[johnwc]

@blueww We have a couple 100k blobs that we need to set tags to, how can we update them in bulk rather than calling Set-AzStorageBlobTag per blob?

[blueww]

@johnwc

Powershell depends on .net SDK, and .net SDK depends on rest API.
Currently blob rest API only support set blob tag on single blob (or in upload/copy blob), but not support set blob tag in bulk.
So all Storage tools/SDK (include Powershell), can't set blob tag in bulk.

My suggestion is you can list blob chunk by chunk (see sample), and set blob tag on each list out blob.

If you would like rest API (and server) support set blob tag in bulk in the future, you can raise you request in MS portal like following picture.