CreateXHarnessAppleWorkItems task failed unexpectedly when building iOS arm64 AllSubsets_Mono #67349
Labels
arch-arm64
area-Infrastructure-mono
blocking-clean-ci
Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms'
os-ios
Apple iOS
Comments
joperezr
added
arch-arm64
blocking-clean-ci
|
Tagging subscribers to this area: @directhex Issue DetailsRolling Build failed this morning on the For convenience, I'm pasting the stack trace of the failure below: @radical do you mind tagging the right people that should be looking into this?
|
dotnet-issue-labeler
bot
added
the
untriaged
|
cc @steveisok |
|
@premun looks like there might be a problem trying to grab the provisioning profile. |
|
I guess this is a fluke? Or possibly something to do with the TLS 1.2 changes happening all around, maybe the agent didn't have this enabled somehow? I tried to run this again and seems to work but I will add retries: dotnet/arcade#8703 |
|
Thanks, @premun. |
|
This doesn't seem fixed, it failed again in today's build: https://dev.azure.com/dnceng/public/_build/results?buildId=1694490&view=logs&j=f4520fb1-1559-5885-1d9c-3cb3f6a85e23&t=6c7a8cfe-f92e-569a-eef9-b2ad3e13056d It does seem flaky as we have had successful builds in between the two with the failure, but it does indeed happen some times so should be investigated as it is now twice this week where this is the only error keeping the rolling build from being green. |
|
I am prioritizing dotnet/arcade#8703 and will have a look. There was no code change in that area for many months so it might be actually related to TLS (for which retries won't help, I am afraid). But I will see what can be done. |
|
@wfurt @bartonjs is there a way to get more info from should it log the name of the cert, or the endpoint? |
|
RevocationStatusUnknown means one of
Since no other status code was reported (or the exception only prints one) it sounds like just intermittency (unless it's a firewall configuration problem). But for a revocation host problem it may take longer to recover than "try three times". |
|
Thanks for the explanation! I am adding 10 retries with exponential backoff here: dotnet/arcade#8747 TimelineIssues
| where Type == "error" and Message contains "CreateXHarnessAppleWorkItems"
| join kind=inner TimelineBuilds on BuildId
| where StartTime > now() - 14d |
|
Happened again in today's rolling build: https://dev.azure.com/dnceng/public/_build/results?buildId=1702530&view=results |
|
Probably will keep happening before we merge the fix and flow Arcade to runtime |
|
Happened again in the runtime-extra-platforms rolling build https://dev.azure.com/dnceng/public/public%20Team/_build/results?buildId=1702568&view=logs&j=a5078f86-b345-5a4a-85ee-f64916152c6f&t=b1d0531b-2b6c-5e64-cc59-e2e1ffcc72bf&l=76
|
|
@carlossanlop please note that the fix has already been merged into Arcade, however Arcade hasn't been promoted for well over a week because runtime has problem with ingestion of it. This is probably something to look into for the @dotnet/runtime-infrastructure team? https://dnceng.visualstudio.com/internal/_build/results?buildId=1706764&view=results |
|
@hoyosjs please let me know when we update the arcade dependency in runtime so we consume this fix and close this issue. |
|
@premun Is there any way to bring in a fix for this without updating the version of the SDK we use in runtime? That's a big task and might take us some time |
|
@agocke I'm not sure but this SSL error is a transient occasional error which is probably not blocking anything so I am not sure whether it makes sense to try to somehow build some .NET 6 Arcade with it. Do you have some ETA for the Arcade bump? |
The work is happening here #67771 |
akoeplinger
added a commit
to akoeplinger/runtime
that referenced
this issue
Apr 14, 2022
Extracted from dotnet#67771 to get the fix for dotnet#67349
carlossanlop
removed
the
untriaged
This was referenced Apr 16, 2022
akoeplinger
added a commit
that referenced
this issue
Apr 21, 2022
Extracted from #67771 to get the fix for #67349 Updates the WASM tests to always use net6.0 until we can upgrade to a net7.0 SDK. Co-authored-by: Premek Vysoky <[email protected]>
directhex
pushed a commit
to directhex/runtime
that referenced
this issue
Apr 21, 2022
Extracted from dotnet#67771 to get the fix for dotnet#67349 Updates the WASM tests to always use net6.0 until we can upgrade to a net7.0 SDK. Co-authored-by: Premek Vysoky <[email protected]>
|
A fix was merged to main on April 21st and the issue hasn't happened since (on main): I use this query to verify this: TimelineIssues
| where Type == "error" and Message contains "CreateXHarnessAppleWorkItems"
| join kind=inner TimelineBuilds on BuildId
| where StartTime > now() - 17d and SourceBranch == "main"
| order by StartTime desc
| project QueueTime, Project, Repository, Reason, BuildNumber, Definition, SourceBranchWe can give it a few more days and check again but it seems like the fix is helping. |
|
There were no occurrences in the last 6 days so feels like the fix helped. I think we can close this @joperezr |
ghost
locked as resolved and limited conversation to collaborators
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Rolling Build failed this morning on the
Build iOS arm64 Release AllSubsets_Monojob at the Send to Helix step, with a problem when trying to use the toolmicrosoft.dotnet.xharness.clito run theCreateXHarnessAppleWorkItemstask. The issue seems to be just a failure in the SSL handshake communication. To view more details, you can get to the log of the failing build here.For convenience, I'm pasting the stack trace of the failure below:
@radical do you mind tagging the right people that should be looking into this?
The text was updated successfully, but these errors were encountered: