Travis CI is stalling in some builds

[tcard]

Travis builds sometimes fail with something like this:

No output has been received in the last 10m0s, this potentially indicates a stalled build or something wrong with the build itself.
The build has been terminated

This happens most often after running the Swift suite and before starting the Obj-C suite. But I've also seen it happen in the middle of the Swift suite, with a message like "main thread has stalled".

This happens inconsistently.

[ricardopereira]

I will try to join both tests into one target to have only one execution. Right now it executes the Swift target and then the Objective-C target and in some way it gets stuck. It should resolve this issue.

I will also do some changes on the project because I want to use the bridging header to import the library code to the Swift test target like Apple suggest: Importing Objective-C into Swift. CocoaPods in a Swift project and with the use_frameworks! setting use also a bridging header so I think it is a way to guarantee we use it in the same way on our ablySpec(Swift one).

[ricardopereira]

Could be related: xcodebuild hangs when a test causes EXC_BAD_ACCESS kernel exception https://openradar.appspot.com/24222858.

[ricardopereira]

I tried to join both test targets into one but it isn't possible because there are too different. I will change the Travis file to use scan --verbose to see if we get anything from the log.

[ricardopereira]

I found out my test suite was hanging when I was checking #171. I run the test suite with scan then only with xcodebuild and it hanged as well. I run it from Xcode and the test stopped because of an exception. So, I assume that there is something raising an exception on the CI as well. Something to look further.

[ricardopereira]

There is a bug in xcodebuild which has been fixed in the latest release of XCode, 7.2.1 (which was released an hour ago). From the release notes

Fixed in Xcode 7.2.1:
• Command line tool ‘xcodebuild test’ will no longer time out waiting for Simulator.app to launch

[mattheworiordan]

What is the status on this?

[tcard]

No news. There are periods in which this happens less often. I still don't know how we could debug this, we would need to make Travis builds more explicit somehow, haven't looked into it though.

[mattheworiordan]

@tcard @ricardopereira should we not raise an issue with Travis themselves to get their help? I can't believe that their system doesn't work given how ubiquitous it is in the open source community. Can one of you do that?

[ricardopereira]

The --verbose from scan doesn't help. I can update the Xcode version and see if it fixes.

[mattheworiordan]

Ok, and then lets just raise an issue with Travis, they are incentivised to make it work so will be able to help I am sure

[ricardopereira]

bitrise-io/bitrise.io#5

In our swift code we had a [unowned self], which should have been a [weak self] and it would fail unit tests randomly with exit code 65

One more note: it seems that if you use xcpretty you'll get a similar issue (that you don't get any error just the exit code 65) in cases where the xcodebuild output might include the error.

[mattheworiordan]

Are we saying that with the releaseChannel disabled and the Xcode upgrade this is now no longer an issue?

[ricardopereira]

Yes, none stalled test at the moment with #265.

[ricardopereira]

Still happening 💤

[ricardopereira]

With xcodebuild -workspace ./ably.xcworkspace -list we have different results:

My copy and @tcard:

➜  ably-ios git:(master) xcodebuild -workspace ./ably.xcworkspace -list
Information about workspace "ably":
    Schemes:
        Ably
        ablySpec
        ablyTests
        Aspects
        Nimble
        Pods
        Pods-ablySpec
        Pods-ablyTests
        Quick
        SwiftWebSocket
        SwiftyJSON

CI:

+ set -e
+ xcodebuild -workspace ./ably.xcworkspace -list
2016-03-02 01:14:33.079 xcodebuild[1110:5940] Failed to locate a valid instance of CoreSimulatorService in the bootstrap.  Adding it now.
Information about workspace "ably":
    Schemes:
        ably
        ablySpec
        ablyTests
        Aspects
        Nimble
        Pods
        Pods-ablySpec
        Pods-ablyTests
        Quick
        SwiftWebSocket
        SwiftyJSON

If xcodebuild can't find the scheme then it's quite possible the test gets stopped. I will uniformize the project name and scheme to Ably.

[ricardopereira]

@tcard @mattheworiordan I finally have an answer for this. I had a huge help from Bitrise for this one. So, the test suite sometimes "hangs" because the xcodebuild, for some reason, doesn’t print the output. Travis checks if there’s any activity on stdout and if not then this can happen: "No output has been received in the last 10m0s, this potentially indicates a stalled build or something wrong with the build itself". It seems like the test suite hangs but once all the tests finish the output is actually shown. Currently the test suite takes ~20 minutes. I have a free plan on Bitrise with a 10 minutes limit so that’s why it fails as well. They gave me a 15 trial plan to run all the needed tests to check this problem and currently it works but sometimes the complete log is only printed after the tests are finished.

[mattheworiordan]

Sounds encouraging... keep us posted

[ricardopereira]

Currently the test suite, in normal conditions, takes around 22 min to finish. So, if we get a Timeout (30 minutes) reached. Terminating "scan --scheme Ably --open_report false" from Travis, it's a sign that the sandbox is broken.