Helix API erroring 500 on Arcade PRs

[missymessa]

D:\a\1\s\src\Microsoft.DotNet.Helix\Sdk\tools\Microsoft.DotNet.Helix.Sdk.MonoQueue.targets(47,5): error : RestApiException`1: The response contained an invalid status code 500 Internal Server Error [D:\a\1\s\tests\UnitTests.proj]
D:\a\1\s\src\Microsoft.DotNet.Helix\Sdk\tools\Microsoft.DotNet.Helix.Sdk.MonoQueue.targets(47,5): error :  [D:\a\1\s\tests\UnitTests.proj]
D:\a\1\s\src\Microsoft.DotNet.Helix\Sdk\tools\Microsoft.DotNet.Helix.Sdk.MonoQueue.targets(47,5): error : Body: {"Message":"An error occured.","ActivityId":"4faff895cd66cc39cb9efa781b8aac33"} [D:\a\1\s\tests\UnitTests.proj]
D:\a\1\s\src\Microsoft.DotNet.Helix\Sdk\tools\Microsoft.DotNet.Helix.Sdk.MonoQueue.targets(47,5): error :    at Microsoft.DotNet.Helix.Client.Job.OnNewFailed(Request req, Response res) in /_/src/Microsoft.DotNet.Helix/Client/CSharp/generated-code/Job.cs:line 169 [D:\a\1\s\tests\UnitTests.proj]
D:\a\1\s\src\Microsoft.DotNet.Helix\Sdk\tools\Microsoft.DotNet.Helix.Sdk.MonoQueue.targets(47,5): error :    at Microsoft.DotNet.Helix.Client.Job.NewAsync(JobCreationRequest body, String idempotencyKey, CancellationToken cancellationToken) in /_/src/Microsoft.DotNet.Helix/Client/CSharp/generated-code/Job.cs:line 131 [D:\a\1\s\tests\UnitTests.proj]
D:\a\1\s\src\Microsoft.DotNet.Helix\Sdk\tools\Microsoft.DotNet.Helix.Sdk.MonoQueue.targets(47,5): error :    at Microsoft.DotNet.Helix.Client.JobDefinition.SendAsync(Action`1 log, CancellationToken cancellationToken) in /_/src/Microsoft.DotNet.Helix/JobSender/JobDefinition.cs:line 242 [D:\a\1\s\tests\UnitTests.proj]
D:\a\1\s\src\Microsoft.DotNet.Helix\Sdk\tools\Microsoft.DotNet.Helix.Sdk.MonoQueue.targets(47,5): error :    at Microsoft.DotNet.Helix.Sdk.SendHelixJob.ExecuteCore(CancellationToken cancellationToken) in /_/src/Microsoft.DotNet.Helix/Sdk/SendHelixJob.cs:line 270 [D:\a\1\s\tests\UnitTests.proj]
D:\a\1\s\src\Microsoft.DotNet.Helix\Sdk\tools\Microsoft.DotNet.Helix.Sdk.MonoQueue.targets(47,5): error :    at Microsoft.DotNet.Helix.Sdk.HelixTask.Execute() in /_/src/Microsoft.DotNet.Helix/Sdk/HelixTask.cs:line 61 [D:\a\1\s\tests\UnitTests.proj]
D:\a\1\s\src\Microsoft.DotNet.Helix\Sdk\tools\Microsoft.DotNet.Helix.Sdk.MonoQueue.targets(47,5): error :  [D:\a\1\s\tests\UnitTests.proj]
##[error]src\Microsoft.DotNet.Helix\Sdk\tools\Microsoft.DotNet.Helix.Sdk.MonoQueue.targets(47,5): error : (NETCORE_ENGINEERING_TELEMETRY=Helix) RestApiException`1: The response contained an invalid status code 500 Internal Server Error

Body: {"Message":"An error occured.","ActivityId":"4faff895cd66cc39cb9efa781b8aac33"}
   at Microsoft.DotNet.Helix.Client.Job.OnNewFailed(Request req, Response res) in /_/src/Microsoft.DotNet.Helix/Client/CSharp/generated-code/Job.cs:line 169
   at Microsoft.DotNet.Helix.Client.Job.NewAsync(JobCreationRequest body, String idempotencyKey, CancellationToken cancellationToken) in /_/src/Microsoft.DotNet.Helix/Client/CSharp/generated-code/Job.cs:line 131
   at Microsoft.DotNet.Helix.Client.JobDefinition.SendAsync(Action`1 log, CancellationToken cancellationToken) in /_/src/Microsoft.DotNet.Helix/JobSender/JobDefinition.cs:line 242
   at Microsoft.DotNet.Helix.Sdk.SendHelixJob.ExecuteCore(CancellationToken cancellationToken) in /_/src/Microsoft.DotNet.Helix/Sdk/SendHelixJob.cs:line 270
   at Microsoft.DotNet.Helix.Sdk.HelixTask.Execute() in /_/src/Microsoft.DotNet.Helix/Sdk/HelixTask.cs:line 61

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

[missymessa]

Things we've investigated:

• re-enabled previously disabled storage accounts
• have not cycled keys or rebooted the service (other repos do not seem to be affected right now? Saw something similar in a Helix Machines PR, but the error was a 302)

[missymessa]

FR conversation here: https://teams.microsoft.com/l/message/19:[email protected]/1723050990188?tenantId=72f988bf-86f1-41af-91ab-2d7cd011db47&groupId=147df318-61de-4f04-8f7b-ecd328c256bb&parentMessageId=1723050990188&teamName=.NET%20Eng%20Services&channelName=FR%20Internal&createdTime=1723050990188

This morning, we were still able to trace the failures back to the re-enabled storage accounts. We rebooted the nodes for Helix Service in Service Fabric and that seemed to put things back into a good state. We're assuming caching was occurring somewhere that caused us to be in a bad state when the storage accounts were previously disabled.

PR builds in Arcade seem to be working now.