Disable known failing device tests using xharness #4884
Conversation
chamons
changed the title
|
Build success |
|
Issues related to mono should ideally be narrowed down and filed in the mono repo, otherwise they'll never be fixed (even less if we ignore them). So I had a look again at https://github.com/xamarin/maccore/issues/1014, and luckily I was able to reproduce it, so I fixed it (#4897). I also tried reproducing the mini timeouts (https://github.com/xamarin/maccore/issues/1011) again, but no luck there. Have you tried it locally? I'm fine with disabling the F# extension tests, those are useless right now until the corresponding bug is fixed. The general case of all extension tests is a bit more complicated though, because they do actually confirm one thing even if they crash: that they build. We've almost released XI versions with extension build bugs... and the extension tests were disabled at the time. What's needed here is a way to be able to make tests build-only somehow (https://github.com/xamarin/maccore/issues/1076), which is a somewhat larger project. |
|
Build failure |
|
@rolfbjarne I'm going to try to isolate the test failures more, but thought on the PR now? |
|
Build failure |
|
Build success |
tests/xharness/Jenkins.cs
Outdated
| ForceExtensionBuildOnly = true; | ||
|
|
||
| // https://github.com/xamarin/maccore/issues/1009 | ||
| foreach (var fsharp in Harness.IOSTestProjects.Where (x => x.Name == "fsharp" || x.Name == "fsharplibrary")) |
There was a problem hiding this comment.
The F# extensions tests don't even build, so these can be completely ignored.
tests/xharness/Jenkins.cs
Outdated
| @@ -3140,6 +3169,9 @@ public async Task<bool> BuildAsync () | |||
|
|
|||
| protected override async Task ExecuteAsync () | |||
| { | |||
| if (BuildOnly) | |||
There was a problem hiding this comment.
How do these tests show up the html report now (after running them)?
There was a problem hiding this comment.
I did not realize that you could run them locally from the runner and debug from VSfM. Looking into this now.
|
Build failure |
|
Build success |
dotnet#4884 introduced the logic of only building certain `RunTestTask`. This was meant to disable iOS Extensions as part of a fix to xamarin/maccore#1008. However this didn't quite work and iOS extensions were still running (and failing). The reason being that `BuildOnly` was set to a `RunDeviceTask` that's added to a list which is then given to `CreateTestVariations` which creates new instances of `RunDeviceTask`. We now propagate `BuildOnly` to the new variation instance.
#4884 introduced the logic of only building certain `RunTestTask`. This was meant to disable iOS Extensions as part of a fix to xamarin/maccore#1008. However this didn't quite work and iOS extensions were still running (and failing). The reason being that `BuildOnly` was set to a `RunDeviceTask` that's added to a list which is then given to `CreateTestVariations` which creates new instances of `RunDeviceTask`. We now propagate `BuildOnly` to the new variation instance.
#4884 introduced the logic of only building certain `RunTestTask`. This was meant to disable iOS Extensions as part of a fix to xamarin/maccore#1008. However this didn't quite work and iOS extensions were still running (and failing). The reason being that `BuildOnly` was set to a `RunDeviceTask` that's added to a list which is then given to `CreateTestVariations` which creates new instances of `RunDeviceTask`. We now propagate `BuildOnly` to the new variation instance.
…5980) * [tests] Don't run BitcodeNotSupported tests on watchOS. (#5744) Ref: mono/mono#13007 * [tests] Check all architectures when verifying public symbols. (#5745) This also means updating the whitelisted symbols. * [XHarness] Don't run BitcodeNotSupported BCL tests on watchOS. (#5750) * [xharness] Remove bitcode from tvOS apps. (#5781) We don't need the bitcode for testing, it makes test apps bigger (so big that the Apple TV might refuse to install them). * [XHarness] Reduce the number of test applications for the BCL tests. (#5768) This commit improves the state of the BCL testing in the following ways: 1. Improve the device tets running. Less apps, faster results. 2. WatchOS apps are left as they were to ensure that we do not have deplouyment/run issues. We now support the ignore files per assembly name to simplify the tracking of the ignored tests. All * [XHarness] Ensure we do not crash if the framework node is missing. (#5792) * [xharness] Fix BuildOnly logic (#5813) #4884 introduced the logic of only building certain `RunTestTask`. This was meant to disable iOS Extensions as part of a fix to xamarin/maccore#1008. However this didn't quite work and iOS extensions were still running (and failing). The reason being that `BuildOnly` was set to a `RunDeviceTask` that's added to a list which is then given to `CreateTestVariations` which creates new instances of `RunDeviceTask`. We now propagate `BuildOnly` to the new variation instance. * [XHarness] Fix IntermediateOutputPath getting inserted in the wrong place. (#5841) * Ignore tests for devices (#5807) * [XHarness] Ignore some tests that fail on device. Added the ignore, which can be later removed on the new mono 2019-02 since the issues do not happen there. Fixes: xamarin/maccore#1495 * [XHarness] Ignore corlib tets failing on devices. Added the required ignores to get the devices green and workaround mono issue mono/mono#13641 * [XHarness] Ignore a failing test on devices. (#5848) * [XHarness] Update ignore for some failing tests. (#5887) The mono bump added some new tests that fail on device + release configurations. This is a known issue by mono and they'll fix it in the next version but will not be backported to 2018-10. * [XHarness] Ignore monotouch_Microsoft.CSharp_xunit-test.dll until issue 5868 is fixed. (#5869) If not ignored, we will have an issue building the test app on device resulting on a failure. * [d16-1][xharness] Skip fsharp iOS tests until #5886 is merged This can be reverted if #5886 makes it in. * [XHarness] Fix issues building the today extensions for the BCL tests apps. (#5854) * [d16-1][xharness] Ignore failing device tests because mono backports are impossible * [d16-1][xharness] Ignore correct interp tests and more BCL group 1 * [d16-1][xharness] Ignore mscorlib interpreter (debug) test * [tests] Adjust SetupBlockPerfTest to expect at least a 4x speedup instead of 6x We hit this: ``` Expected: greater than 6 But was: 4.9317220348827826d ``` So adjusting to 4 to avoid those random failures. See xamarin/maccore#649 for more details.
So far these four are disabled:
I tested locally that TestsToSkipOnDevice.Contains hits but I did not do full device test run locally.