[BUG] Managed Identity and SubmitTransactionAsync causes AtomFormatNotSupported exception

[jedjohan]

Describe the bug
When connecting to Azure Table with TokenCredential/Managed Identity and doing a SubmitTransactionAsync I get confusing "AtomFormatNotSupported" exception. When using full SA connectionstring it works fine. Also, regular AddEntityAsync works fine with Managed Identity.

Expected behavior
SubmitTransactionAsync should work when using Managed Identity

Actual behavior (include Exception or Stack Trace)
Exception:
"Service request failed.\r\nStatus: 415 (Unsupported Media Type)\r\n\r\nContent:\r\n<error xmlns="http://schemas.microsoft.com/ado/2007/08/dataservices/metadata\">AtomFormatNotSupported<message xml:lang="en-US">Atom format is not supported."

To Reproduce

1. Create User Assigned identity and give it the [Storage Table Data Contributor] role on the table
2. Connect to Storage Table with ClientId from previous step
3. Perform a SubmitTransactionAsync

something like this:

      TokenCredential tokenCredential = string.IsNullOrEmpty(Environment.GetEnvironmentVariable("WEBSITE_INSTANCE_ID"))
      ? new DefaultAzureCredential()
      : new ManagedIdentityCredential("Client id for User Assigned Identity");

      var Client = new TableClient(new Uri("url to table"), TableName, tokenCredential);
  
      try
      {
          var addEntitiesBatch = new List<TableTransactionAction>(100);
          foreach (var product in products)
          {
              addEntitiesBatch.Add(new TableTransactionAction(TableTransactionActionType.Add, product));
          }

          var response = await Client.SubmitTransactionAsync(addEntitiesBatch);
      }
      catch (Exception e)
      {
          throw e;
      }

Environment:

• Name and version of the Library package used: Azure.Data.Tables 12.2.0
• Hosting platform or OS and .NET runtime version:
  .NET SDK (reflecting any global.json):
  Version: 6.0.100-rc.1.21458.32
  Commit: d7c22323c4

Runtime Environment:
OS Name: Windows
OS Version: 10.0.19043
OS Platform: Windows
RID: win10-x64
Base Path: C:\Program Files\dotnet\sdk\6.0.100-rc.1.21458.32\

Host (useful for support):
Version: 6.0.0-rc.1.21451.13
Commit: d7619cd4b1
....... (long list of SDKs)

• IDE and version : Microsoft Visual Studio Professional 2022 Preview (64-bit) Version 17.0.0 Preview 4.0

[jsquire]

Thank you for your feedback. Tagging and routing to the team member best able to assist.

[jedjohan]

Small sample to reproduce. I guess System Assigned Identity would cause same issue but never tested.
https://github.com/jedjohan/SubmitBatchManagedIdentity

[christothes]

Hi @jedjohan - Thanks for the repro steps. Could you also provide the full stack trace from the exception mentioned above?

[jedjohan]

@christothes sure, hope this is ok ?

{"Service request failed.\r\nStatus: 415 (Unsupported Media Type)\r\n\r\nContent:\r\n<?xml version=\"1.0\" encoding=\"utf-8\"?><error xmlns=\"http://schemas.microsoft.com/ado/2007/08/dataservices/metadata\"><code>AtomFormatNotSupported</code><message xml:lang=\"en-US\">Atom format is not supported.\nRequestId:4e20d209-c002-003e-74fe-ae3856000000\nTime:2021-09-21T15:36:48.5801181Z</message></error>\r\n\r\nHeaders:\r\nTransfer-Encoding: chunked\r\nServer: Windows-Azure-Table/1.0,Microsoft-HTTPAPI/2.0\r\nx-ms-request-id: 4e20d209-c002-003e-74fe-ae3856000000\r\nx-ms-client-request-id: edefc13e-4306-4eb1-a2c7-c38c4579d084\r\nx-ms-version: REDACTED\r\nX-Content-Type-Options: REDACTED\r\nDate: Tue, 21 Sep 2021 15:36:48 GMT\r\nContent-Type: application/xml; charset=utf-8\r\n"}
    Data: {System.Collections.ListDictionaryInternal}
    ErrorCode: null
    HResult: -2146233088
    HasBeenThrown: true
    HelpLink: null
    InnerException: null
    Message: "Service request failed.\r\nStatus: 415 (Unsupported Media Type)\r\n\r\nContent:\r\n<?xml version=\"1.0\" encoding=\"utf-8\"?><error xmlns=\"http://schemas.microsoft.com/ado/2007/08/dataservices/metadata\"><code>AtomFormatNotSupported</code><message xml:lang=\"en-US\">Atom format is not supported.\nRequestId:4e20d209-c002-003e-74fe-ae3856000000\nTime:2021-09-21T15:36:48.5801181Z</message></error>\r\n\r\nHeaders:\r\nTransfer-Encoding: chunked\r\nServer: Windows-Azure-Table/1.0,Microsoft-HTTPAPI/2.0\r\nx-ms-request-id: 4e20d209-c002-003e-74fe-ae3856000000\r\nx-ms-client-request-id: edefc13e-4306-4eb1-a2c7-c38c4579d084\r\nx-ms-version: REDACTED\r\nX-Content-Type-Options: REDACTED\r\nDate: Tue, 21 Sep 2021 15:36:48 GMT\r\nContent-Type: application/xml; charset=utf-8\r\n"
    SerializationStackTraceString: "   at Azure.Data.Tables.TableRestClient.SendBatchRequestAsync(HttpMessage message, CancellationToken cancellationToken) in Azure.Data.Tables.dll:token 0x60002b7+0x2a1\r\n   at Azure.Data.Tables.TableClient.SubmitTransactionInternalAsync(IEnumerable`1 transactionalBatch, Guid batchId, Guid changesetId, Boolean async, CancellationToken cancellationToken) in Azure.Data.Tables.dll:token 0x60002fd+0x144\r\n   at Azure.Data.Tables.TableClient.SubmitTransactionAsync(IEnumerable`1 transactionActions, CancellationToken cancellationToken) in Azure.Data.Tables.dll:token 0x60002f9+0xba\r\n   at ShopitReceiver.Services.ShopitRawStorage.<Insert>b__5_2(ShopitRawMini[] chunk, CancellationToken token) in C:\\_code\\cdon\\ShopitImport\\src\\ShopitReceiver\\Services\\ShopitRawStorage.cs:line 59\r\n   at System.Threading.Tasks.Parallel.<>c__50`1.<<ForEachAsync>b__50_0>d.MoveNext() in System.Threading.Tasks.Parallel.dll:token 0x600007d+0x107\r\n--- End of stack trace from previous location ---\r
\n   at ShopitReceiver.Services.ShopitRawStorage.Insert(IEnumerable`1 products) in C:\\_code\\cdon\\ShopitImport\\src\\ShopitReceiver\\Services\\ShopitRawStorage.cs:line 55"
    SerializationWatsonBuckets: null
    Source: "Azure.Data.Tables"
    StackTrace: "   at Azure.Data.Tables.TableRestClient.<SendBatchRequestAsync>d__44.MoveNext()\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()\r\n   at Azure.Data.Tables.TableClient.<SubmitTransactionInternalAsync>d__62.MoveNext()\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()\r\n   at Azure.Data.Tables.TableClient.<SubmitTransactionAsync>d__58.MoveNext()\r\n   at System.Runtime.ExceptionService
s.ExceptionDispatchInfo.Throw()\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()\r\n   at ShopitReceiver.Services.ShopitRawStorage.<<Insert>b__5_2>d.MoveNext() in C:\\_code\\cdon\\ShopitImport\\src\\ShopitReceiver\\Services\\ShopitRawStorage.cs:line 59\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.ValueTaskAwaiter.GetResult()\r\n   at System.Threading.Tasks.Parallel.<>c__50`1.<<ForEachAsync>b__50_0>d.MoveNext()\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at System.Runtime.CompilerServices.Task
Awaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at ShopitReceiver.Services.ShopitRawStorage.<Insert>d__5.MoveNext() in C:\\_code\\cdon\\ShopitImport\\src\\ShopitReceiver\\Services\\ShopitRawStorage.cs:line 55"
    Status: 415
    TargetSite: {Void MoveNext()}
    _HResult: -2146233088
    _data: {System.Collections.ListDictionaryInternal}
    _dynamicMethods: null
    _exceptionMethod: null
    _helpURL: null
    _innerException: null
    _ipForWatsonBuckets: 0x00007ff9e1d44da5
    _message: "Service request failed.\r\nStatus: 415 (Unsupported Media Type)\r\n\r\nContent:\r\n<?xml version=\"1.0\" encoding=\"utf-8\"?><error xmlns=\"http://schemas.microsoft.com/ado/2007/08/dataservices/metadata\"><code>AtomFormatNotSupported</code><message xml:lang=\"en-US\">Atom format is not supported.\nRequestId:4e20d209-c002-003e-74fe-ae3856000000\nTime:2021-09-21T15:36:48.5801181Z</message></error>\r\n\r\nHeaders:\r\nTransfer-Encoding: chunked\r\nServer: Windows-Azure-Table/1.0,Microsoft-HTTPAPI/2.0\r\nx-ms-request-id: 4e20d209-c002-003e-74fe-ae3856000000\r\nx-ms-client-request-id: edefc13e-4306-4eb1-a2c7-c38c4579d084\r\nx-ms-version: REDACTED\r\nX-Content-Type-Options: REDACTED\r\nDate: Tue, 21 Sep 2021 15:36:48 GMT\r\nContent-Type: application/xml; charset=utf-8\r\n"
    _remoteStackTraceString: null
    _source: null
    _stackTrace: {byte[1536]}
    _stackTraceString: null
    _watsonBuckets: null
    _xcode: -532462766
    _xptrs: 0x0000000000000000

[christothes]

Thanks - Could I ask for one more thing? Would you mind enabling logging on the client to show the batch request that is returning this error?

It should look something like this:

[Informational] Azure-Core: Request [1f2e9de9-aae5-46ab-b804-6b31eb493954] POST https://somestorageaccount.table.core.windows.net/$batch
x-ms-version:REDACTED
DataServiceVersion:3.0
x-ms-client-request-id:1f2e9de9-aae5-46ab-b804-6b31eb493954
x-ms-return-client-request-id:true
User-Agent:azsdk-net-Data.Tables/12.2.0,(.NET 5.0.10; Microsoft Windows 10.0.19043)
Authorization:REDACTED
Content-Type:multipart/mixed; boundary=batch_82b898e9-fcc7-42ac-b8ef-17de0993e3c1
client assembly: Azure.Data.Tables
[Informational] Azure-Core: Response [d2caba4a-a7b7-429a-a979-ae88bca6c453] 204 No Content (00.1s)
Cache-Control:no-cache
ETag:W/"datetime'2021-09-21T20%3A46%3A47.321551Z'"
Server:Windows-Azure-Table/1.0,Microsoft-HTTPAPI/2.0
x-ms-request-id:e45c2d9c-7002-006f-2d29-afe2a1000000
x-ms-client-request-id:d2caba4a-a7b7-429a-a979-ae88bca6c453
x-ms-version:REDACTED
X-Content-Type-Options:REDACTED
Date:Tue, 21 Sep 2021 20:46:47 GMT
Content-Length:0

[jedjohan]

@christothes absolutely, hope this helps :)

[Informational] Azure-Core: Request [e6503aca-d257-4b10-9234-9aa832001ff3] POST https://myaccount.table.core.windows.net/shopitrawdata/$batch x-ms-version:REDACTED DataServiceVersion:3.0 x-ms-client-request-id:e6503aca-d257-4b10-9234-9aa832001ff3 x-ms-return-client-request-id:true User-Agent:azsdk-net-Data.Tables/12.2.0,(.NET 6.0.0-preview.7.21377.19; Microsoft Windows 10.0.19043) Authorization:REDACTED Content-Type:multipart/mixed; boundary=batch_8f6518ab-12f4-482c-966e-8615f413830d client assembly: Azure.Data.Tables [Informational] Azure-Core: Request [e6503aca-d257-4b10-9234-9aa832001ff3] POST https://myaccount.table.core.windows.net/shopitrawdata/$batch x-ms-version:REDACTED DataServiceVersion:3.0 x-ms-client-request-id:e6503aca-d257-4b10-9234-9aa832001ff3 x-ms-return-client-request-id:true User-Agent:azsdk-net-Data.Tables/12.2.0,(.NET 6.0.0-preview.7.21377.19; Microsoft Windows 10.0.19043) Authorization:REDACTED Content-Type:multipart/mixed; boundary=batch_8f6518ab-12f4-482c-966e-8615f413830d client assembly: Azure.Data.Tables [2021-09-21T21:16:57.526Z] [Warning] Azure-Core: Error response [419d43e0-f4ed-462c-9794-a024656da2b3] 415 Unsupported Media Type (00.0s) Transfer-Encoding:chunked Server:Windows-Azure-Table/1.0,Microsoft-HTTPAPI/2.0 x-ms-request-id:ef331ed3-2002-0019-742e-af2f92000000 x-ms-client-request-id:419d43e0-f4ed-462c-9794-a024656da2b3 x-ms-version:REDACTED X-Content-Type-Options:REDACTED Date:Tue, 21 Sep 2021 21:16:57 GMT Content-Type:application/xml; charset=utf-8

[christothes]

Thanks - Although I don't know why this reproduces for only in this scenario, I have an idea what the problem is. I think the batch requests are missing an Accept application/json header. When not provided, the service assumes application/atom+xml

[jedjohan]

Ahh, ok. Sounds reasonable. I looked after ways to set it on the TableClient but couldn't find any way to force json. As I understand it atom is not supported (on the Table Rest API) since quite a few years back ?

[christothes]

This fix will be in next months package release.