[Tests] Try different urls before failing the test. #6237

mandel-macaque · 2019-06-10T20:45:31Z

We do have a test that fails when there are issues accessing the remote
resource. If an error is raised, rather than setting the tests as an
error, set them to inconclusive with the message of the error.

Related issue: https://github.com/xamarin/maccore/issues/1725

We do have a test that fails when there are issues accessing the remote resource. If an error is raised, rather than setting the tests as an error, set them to inconclusive with the message of the error. Related issue: xamarin/maccore#1725

spouliot · 2019-06-10T20:52:04Z

tests/monotouch-test/Foundation/NSDataTest.cs

+					Assert.Inconclusive ($"Could not access url error is '{error.Description}'");
+				} else {
+					Assert.That ((x != null) && (x.Length > 0));
+				}


the test was to detect an error...
with that change I don't think you can trigger the 2nd assert (unless an empty file was present)
so basically the test can't fail (so it's not worth much)

@spouliot we can check the domain of the error and decide what to do?

Well.. thx Cocoa, we would get the not so useful "NSCocoaErrorDomain Code=256", which means an unknown error happened.

yeah, a reachability (to host) error would be fine to ignore, but other errors would not be

I have used several company urls, lets hope trying several will work, else we can do an increasing timeout retry.

monojenkins · 2019-06-10T21:53:53Z

Build failure
✅ Build succeeded
✅ API Diff (from stable)
🔥 Failed to compare API and create generator diff 🔥
    ** Error: Working directory isn't clean:
    HEAD detached at 1c56879
    Untracked files:
    (use "git add ..." to include in what will be committed)

    tests/bcl-test/BCL tests group 1.csproj
    tests/bcl-test/BCL tests group 2.csproj
    tests/bcl-test/BCL tests group 3.csproj
    tests/bcl-test/BCL tests group 4.csproj
    tests/bcl-test/BCL tests group 6.csproj
    tests/bcl-test/Mac OS X BCL tests group 1-mac-full.csproj
    tests/bcl-test/Mac OS X BCL tests group 1-mac-modern.csproj
    tests/bcl-test/Mac OS X BCL tests group 2-mac-full.csproj
    tests/bcl-test/Mac OS X BCL tests group 2-mac-modern.csproj
    tests/bcl-test/Mac OS X BCL tests group 3-mac-full.csproj
    tests/bcl-test/Mac OS X BCL tests group 3-mac-modern.csproj
    tests/bcl-test/Mac OS X BCL tests group 4-mac-full.csproj
    tests/bcl-test/Mac OS X BCL tests group 4-mac-modern.csproj
    tests/bcl-test/generated/
    tests/bcl-test/mscorlib-mac-full.csproj
    tests/bcl-test/mscorlib-mac-modern.csproj
    tests/bcl-test/mscorlib.csproj

    nothing added to commit but untracked files present (use "git add" to track)
    Search for Comparing API & creating generator diff in the log to view the complete log.
✅ Test run succeeded

spouliot · 2019-06-11T01:57:48Z

for reference

	[INCONCLUSIVE] NSDataTest.Https : Could not access url error is 'Error Domain=NSCocoaErrorDomain Code=256 "The file “robots.txt” couldn’t be opened." UserInfo={NSURL=https://www.microsoft.com/robots.txt}'
		  at MonoTouchFixtures.Foundation.NSDataTest.Https () [0x00021] in /Users/builder/jenkins/workspace/xamarin-macios-pr-builder/tests/monotouch-test/Foundation/NSDataTest.cs:162 
		  at (wrapper managed-to-native) System.Reflection.RuntimeMethodInfo.InternalInvoke(System.Reflection.RuntimeMethodInfo,object,object[],System.Exception&)
		  at System.Reflection.RuntimeMethodInfo.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x0006a] in /Library/Frameworks/Xamarin.iOS.framework/Versions/Current/src/Xamarin.iOS/mcs/class/corlib/System.Reflection/RuntimeMethodInfo.cs:391

https://jenkins.mono-project.com/job/xamarin-macios-pr-builder/10214/Test_20Report/

rolfbjarne

Two ideas to improve this test:

Ignore only when doing CI (like 7d67e05), this way we'll eventually see failures when running into them locally.
Have a list of URLs to try, and pass the test if any of them work.

tests/monotouch-test/Foundation/NSDataTest.cs

spouliot · 2019-06-11T15:54:00Z

it's unclear why it fails so it's possible the complete list will fail (even more since it's all MS hosts) but if it works then it's much better than the current test :)

monojenkins · 2019-06-11T16:54:44Z

Build failure
✅ Build succeeded
✅ API Diff (from stable)
🔥 Failed to compare API and create generator diff 🔥
    ** Error: Working directory isn't clean:
    HEAD detached at 79f7c48
    Untracked files:
    (use "git add ..." to include in what will be committed)

    tests/bcl-test/BCL tests group 1.csproj
    tests/bcl-test/BCL tests group 2.csproj
    tests/bcl-test/BCL tests group 3.csproj
    tests/bcl-test/BCL tests group 4.csproj
    tests/bcl-test/BCL tests group 6.csproj
    tests/bcl-test/Mac OS X BCL tests group 1-mac-full.csproj
    tests/bcl-test/Mac OS X BCL tests group 1-mac-modern.csproj
    tests/bcl-test/Mac OS X BCL tests group 2-mac-full.csproj
    tests/bcl-test/Mac OS X BCL tests group 2-mac-modern.csproj
    tests/bcl-test/Mac OS X BCL tests group 3-mac-full.csproj
    tests/bcl-test/Mac OS X BCL tests group 3-mac-modern.csproj
    tests/bcl-test/Mac OS X BCL tests group 4-mac-full.csproj
    tests/bcl-test/Mac OS X BCL tests group 4-mac-modern.csproj
    tests/bcl-test/generated/
    tests/bcl-test/mscorlib-mac-full.csproj
    tests/bcl-test/mscorlib-mac-modern.csproj
    tests/bcl-test/mscorlib.csproj

    nothing added to commit but untracked files present (use "git add" to track)
    Search for Comparing API & creating generator diff in the log to view the complete log.
🔥 Test run failed 🔥

Test results

4 tests failed, 90 tests passed.

Failed tests

apitest/Mac Classic/NonSystem: BuildFailure
dont link/Mac Classic/All: BuildFailure
introspection/Mac Classic/Modern: BuildFailure
MSBuild tests/iOS: Failed (Execution failed with exit code 1)

mandel-macaque · 2019-06-11T16:59:43Z

Issues are related to the msbuild problems we have, all other tests, specially the NSDataTests passed.

Known issues are:

mandel-macaque · 2019-06-11T18:03:00Z

@monojenkins backport to d16-2

mandel-macaque · 2019-06-11T18:03:12Z

@monojenkins backport to d16-1

VincentDondain · 2019-06-12T16:15:37Z

@monojenkins backport to xcode11

spouliot suggested changes Jun 10, 2019

View reviewed changes

rolfbjarne reviewed Jun 11, 2019

View reviewed changes

tests/monotouch-test/Foundation/NSDataTest.cs Outdated Show resolved Hide resolved

Use a list of urls to try several times.

54bb886

spouliot approved these changes Jun 11, 2019

View reviewed changes

mandel-macaque changed the title ~~[Tests] If there are network issues do not set the tests as a failure.~~ [Tests] Try different urls before failing the test. Jun 11, 2019

mandel-macaque merged commit 46c16eb into xamarin:master Jun 11, 2019

mandel-macaque deleted the nsdatatest-account-network-issues branch June 11, 2019 17:04

This was referenced Jun 11, 2019

[d16-2] [Tests] Try different urls before failing the test. #6265

Merged

[d16-1] [Tests] Try different urls before failing the test. #6266

Merged

monojenkins mentioned this pull request Jun 12, 2019

[xcode11] [Tests] Try different urls before failing the test. #6276

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Tests] Try different urls before failing the test. #6237

[Tests] Try different urls before failing the test. #6237

mandel-macaque commented Jun 10, 2019

spouliot Jun 10, 2019

mandel-macaque Jun 10, 2019

mandel-macaque Jun 10, 2019

spouliot Jun 10, 2019

mandel-macaque Jun 11, 2019

monojenkins commented Jun 10, 2019

spouliot commented Jun 11, 2019

rolfbjarne left a comment

spouliot commented Jun 11, 2019

monojenkins commented Jun 11, 2019

Failed tests

mandel-macaque commented Jun 11, 2019 •

edited

Loading

mandel-macaque commented Jun 11, 2019

mandel-macaque commented Jun 11, 2019

VincentDondain commented Jun 12, 2019

[Tests] Try different urls before failing the test. #6237

[Tests] Try different urls before failing the test. #6237

Conversation

mandel-macaque commented Jun 10, 2019

spouliot Jun 10, 2019

Choose a reason for hiding this comment

mandel-macaque Jun 10, 2019

Choose a reason for hiding this comment

mandel-macaque Jun 10, 2019

Choose a reason for hiding this comment

spouliot Jun 10, 2019

Choose a reason for hiding this comment

mandel-macaque Jun 11, 2019

Choose a reason for hiding this comment

monojenkins commented Jun 10, 2019

spouliot commented Jun 11, 2019

rolfbjarne left a comment

Choose a reason for hiding this comment

spouliot commented Jun 11, 2019

monojenkins commented Jun 11, 2019

Test results

Failed tests

mandel-macaque commented Jun 11, 2019 • edited Loading

mandel-macaque commented Jun 11, 2019

mandel-macaque commented Jun 11, 2019

VincentDondain commented Jun 12, 2019

mandel-macaque commented Jun 11, 2019 •

edited

Loading