crossgen-comparison Linux arm checked fails with timeouts #1282

sandreenko · 2020-01-03T19:35:47Z

It happens in all runs, the timeout is 60 minutes. We have not seen that in coreclr.

Log example:

Starting: Run native crossgen and compare output to baseline (Unix)
==============================================================================
Task         : Command line
Description  : Run a command line script using Bash on Linux and macOS and cmd.exe on Windows
Version      : 2.151.2
Author       : Microsoft Corporation
Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/command-line
==============================================================================
Generating script.
Script contents:
$BUILD_SOURCESDIRECTORY/eng/common/msbuild.sh $BUILD_SOURCESDIRECTORY/eng/common/helixpublish.proj /restore /t:Test /bl:$BUILD_SOURCESDIRECTORY/artifacts/log/$BuildConfig/SendToHelix.binlog
========================== Starting Command Output ===========================
/bin/bash --noprofile --norc /__w/_temp/7553b9eb-4204-4db6-9f64-3d08371fe862.sh
/__w/3/s/.dotnet/sdk/5.0.100-alpha1-015772/MSBuild.dll /nologo -distributedlogger:Microsoft.DotNet.Tools.MSBuild.MSBuildLogger,/__w/3/s/.dotnet/sdk/5.0.100-alpha1-015772/dotnet.dll*Microsoft.DotNet.Tools.MSBuild.MSBuildForwardingLogger,/__w/3/s/.dotnet/sdk/5.0.100-alpha1-015772/dotnet.dll -maxcpucount /m -verbosity:m /v:minimal /bl:/__w/3/s/artifacts/log/Checked/SendToHelix.binlog /clp:Summary /nr:true /p:TreatWarningsAsErrors=true /p:ContinuousIntegrationBuild=false /restore /t:Test /warnaserror /__w/3/s/eng/common/helixpublish.proj
  Restore completed in 599.36 ms for /__w/3/s/eng/common/helixpublish.proj.
  Restore completed in 2.35 ms for /__w/3/s/eng/common/helixpublish.proj.
  Starting Azure Pipelines Test Run (Ubuntu.1804.Arm32.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7-30f6673-20190814153226
  Uploading payloads for Job on (Ubuntu.1804.Arm32.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7-30f6673-20190814153226...
  Finished uploading payloads for Job on (Ubuntu.1804.Arm32.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7-30f6673-20190814153226...
  Sending Job to (Ubuntu.1804.Arm32.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7-30f6673-20190814153226...
  Sent Helix Job 40b0be17-33d6-41c3-bdce-fd02aba91a98
  Waiting for completion of job 40b0be17-33d6-41c3-bdce-fd02aba91a98
##[error]The operation was canceled.
Finishing: Run native crossgen and compare output to baseline (Unix)

https://dev.azure.com/dnceng/public/_build/results?buildId=471248

The text was updated successfully, but these errors were encountered:

BruceForstall · 2020-01-03T19:45:10Z

Is this a regression, or has it always failed in runtime?

sandreenko · 2020-01-03T19:51:06Z

I saw it green a few times, for example here https://dev.azure.com/dnceng/public/_build/results?buildId=456520&view=logs&j=e14ed261-6507-56f9-47b6-1ad25f52112f.

jashook · 2020-01-03T23:36:33Z

Generally this is failure happens due to us running at capacity for the helix queue. With the amount of machines that were added last month, there is most likely something that has changed with what jobs are submitted and from where to cause this, or there are a lot of machine that have gone offline.

jashook · 2020-01-03T23:38:30Z

Is this a regression, or has it always failed in runtime?

This has not always failed, although the amount of jobs submitted has probably increased this month over last month.

sandreenko · 2020-01-03T23:40:52Z

Thanks, @jashook . We can make a quick surgical fix and stop running everything twice (under runtime and under runtime-coreclr pipelines, delete duplicates from https://github.com/dotnet/runtime/blob/master/eng/pipelines/runtime-official.yml and https://github.com/dotnet/runtime/blame/master/eng/pipelines/coreclr/pr.yml) or wait for a few weeks when @safern replaces existing pipelines.

jashook · 2020-01-03T23:51:45Z

Seems like the wait times are not too high in the queue, if I remember right these jobs ran close to the timeout. I would suggest upping the timeout by 15 to 30 minutes.

safern · 2020-01-06T19:04:04Z

or wait for a few weeks when @safern replaces existing pipelines.

I should be removing duplication this week. Hope to have a PR before Wednesday.

Disable crossgen comparison runs that systematically fail in CoreCLR outerloop runs. Tracking issue: dotnet#1282 Thanks Tomas

Disable crossgen comparison runs that systematically fail in CoreCLR outerloop runs. Tracking issue: #1282 Thanks Tomas

safern · 2020-01-16T01:15:09Z

I just merged: #1473 which removes the duplication of pipelines and it will now only be ran once per PR and once on CI. Will leave this open to monitor the outcome after this change and react if needed.

jaredpar · 2020-02-25T21:25:36Z

This is presently impacting about 9% of our builds

Build	Kind	Timeline Record
534481	PR #2107	Test crossgen-comparison Linux arm checked
534999	PR #2107	Test crossgen-comparison Linux arm checked
535931	PR #32402	Test crossgen-comparison Linux arm checked
535806	PR #32794	Test crossgen-comparison Linux arm checked
534710	PR #32762	Test crossgen-comparison Linux arm checked
534703	PR #32330	Test crossgen-comparison Linux arm checked
534497	PR #32499	Test crossgen-comparison Linux arm checked
534413	PR #32626	Test crossgen-comparison Linux arm checked
534345	Rolling	Test crossgen-comparison Linux arm checked

BruceForstall · 2020-02-25T21:47:56Z

@jashook You thought before that this was failing due to overloaded Linux arm hardware, leading to the timeouts. Do you still believe that?

jashook · 2020-02-25T21:48:30Z

Yes, I think there is more pressure on the hardware with mono running arm now.

echesakov · 2020-02-28T01:32:52Z

It look that we missed a real regression in crossgen_comparison due to our inability to run this test without timing out #32951

@jashook Can we increase timeouts for this job?

jaredpar · 2020-03-02T18:54:14Z

Removing blocking label now that mitigations are in place.

ViktorHofer · 2020-03-02T19:13:25Z

Is this issue still actionable now that the mitigation is in place? Should we open another issue or link to an existing one which tracks the ARM hardware capacity issue?

BruceForstall · 2021-03-02T17:11:23Z

@echesakovMSFT I noticed that the crossgen-comparison job is still commented out in ci.yml, so it's not running in the outerloop job. Should we fix that?

echesakov · 2021-03-03T20:01:56Z

@BruceForstall I think it is okay.

We still have crossgen-comparison job triggered in PRs (if they affect code under src/coreclr). One of the latest runs is green - https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-48830-merge-9b43a99dd41241aaa5/WorkItem/console.42781747.log?sv=2019-07-07&se=2021-03-23T08%3A59%3A24Z&sr=c&sp=rl&sig=Eu7gjhAYzP2ozwKcOMC%2FsAzV9caUqAe2qehEdfwvEjM%3D

In addition to that, @davidwrighton added crossgen2-outerloop pipeline that I believe runs similar comparison jobs but with crossgen2. In theory, the changes that break JIT cross-bitness, cross-architecture or cross-os compatibility should be caught by these jobs. In reality, these jobs have been red for a while (e.g. https://dev.azure.com/dnceng/public/_build?definitionId=701). I opened #49077 to track the issue.

davidwrighton · 2021-03-03T22:01:36Z

@echesakovMSFT I intend to get around to fixing some of those crossgen2 comparision issues soon.

echesakov · 2021-03-03T23:19:54Z

@davidwrighton Thank you, David!

sandreenko added the area-Infrastructure-coreclr label Jan 3, 2020

Dotnet-GitSync-Bot added the untriaged New issue has not been triaged by the area owner label Jan 3, 2020

sandreenko mentioned this issue Jan 3, 2020

Disable failing jobs in PR testing. #1283

Merged

trylek added a commit to trylek/runtime that referenced this issue Jan 13, 2020

Temporarily disable outerloop crossgen comparison runs

2bf011c

Disable crossgen comparison runs that systematically fail in CoreCLR outerloop runs. Tracking issue: dotnet#1282 Thanks Tomas

trylek mentioned this issue Jan 13, 2020

Temporarily disable outerloop crossgen comparison runs #1695

Merged

trylek added a commit that referenced this issue Jan 14, 2020

Temporarily disable outerloop crossgen comparison runs (#1695)

2c54bd9

Disable crossgen comparison runs that systematically fail in CoreCLR outerloop runs. Tracking issue: #1282 Thanks Tomas

safern mentioned this issue Jan 24, 2020

Increase crossgen-comparison job timeout #2160

Merged

jaredpar added the blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' label Feb 25, 2020

jaredpar assigned jashook Feb 25, 2020

jashook mentioned this issue Feb 25, 2020

Infrastructure - Status/Health #702

Closed

jaredpar mentioned this issue Feb 27, 2020

Enable workflow of building Mono and Libraries without CoreCLR #32881

Merged

hoyosjs mentioned this issue Feb 28, 2020

Package libcoreclrtraceptprovider.so with sharedFramework #32949

Merged

jaredpar removed the blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' label Mar 2, 2020

jashook closed this as completed Mar 2, 2020

ghost locked as resolved and limited conversation to collaborators Dec 11, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

crossgen-comparison Linux arm checked fails with timeouts #1282

crossgen-comparison Linux arm checked fails with timeouts #1282

sandreenko commented Jan 3, 2020 •

edited

Loading

BruceForstall commented Jan 3, 2020

sandreenko commented Jan 3, 2020

jashook commented Jan 3, 2020

jashook commented Jan 3, 2020

sandreenko commented Jan 3, 2020 •

edited

Loading

jashook commented Jan 3, 2020

safern commented Jan 6, 2020

safern commented Jan 16, 2020

jaredpar commented Feb 25, 2020

BruceForstall commented Feb 25, 2020

jashook commented Feb 25, 2020

echesakov commented Feb 28, 2020

jaredpar commented Mar 2, 2020

ViktorHofer commented Mar 2, 2020

BruceForstall commented Mar 2, 2021

echesakov commented Mar 3, 2021

davidwrighton commented Mar 3, 2021

echesakov commented Mar 3, 2021

crossgen-comparison Linux arm checked fails with timeouts #1282

crossgen-comparison Linux arm checked fails with timeouts #1282

Comments

sandreenko commented Jan 3, 2020 • edited Loading

BruceForstall commented Jan 3, 2020

sandreenko commented Jan 3, 2020

jashook commented Jan 3, 2020

jashook commented Jan 3, 2020

sandreenko commented Jan 3, 2020 • edited Loading

jashook commented Jan 3, 2020

safern commented Jan 6, 2020

safern commented Jan 16, 2020

jaredpar commented Feb 25, 2020

BruceForstall commented Feb 25, 2020

jashook commented Feb 25, 2020

echesakov commented Feb 28, 2020

jaredpar commented Mar 2, 2020

ViktorHofer commented Mar 2, 2020

BruceForstall commented Mar 2, 2021

echesakov commented Mar 3, 2021

davidwrighton commented Mar 3, 2021

echesakov commented Mar 3, 2021

sandreenko commented Jan 3, 2020 •

edited

Loading

sandreenko commented Jan 3, 2020 •

edited

Loading