Test Suite issues

[ricardopereira]

Currently there are tests that doesn't pass even running them individually:

❌ **RTP8h**
[09:09:12]: ▸ -[RealtimeClientPresence Presence__enter__should_result_in_an_error_if_the_client_does_not_have_required_presence_permission()]
Failed with: expected to contain <Channel denied access based on given capability>, got <Invalid clientId for credentials>

❌ **RTP9e**
[09:09:12]: ▸ -[RealtimeClientPresence Presence__update__should_result_in_an_error_if_the_client_does_not_have_required_presence_permission()]
Failed with: expected to contain <Channel denied access based on given capability>, got <Invalid clientId for credentials>

❌ **RTP10e**
[09:09:12]: ▸ -[RealtimeClientPresence Presence__leave__should_result_in_an_error_if_the_client_does_not_have_required_presence_permission()]
Failed with: expected to contain <Channel denied access based on given capability>, got <Invalid clientId for credentials>

❌ **RTP12c**
[09:09:12]: ▸ -[RealtimeClientPresence Presence__history__should_return_a_PaginatedResult_page()]
Failed with: expected to all pass a condition, but failed first at element ...

❌ **RSP5**
[09:09:12]: ▸ -[RestClientPresence Presence__presence_messages_retrieved_are_decoded_in_the_same_way_that_messages_are_decoded()]
Failed with: expected to equal <{test = 1;}>, got <nil>

---

It is frustrating when failing tests start to pass after you didn’t change anything, only tried debugging it for a few minutes.

I would like to ask if in a different client, e.g. JS or Ruby, if you run the test suite a couple of times one after another, the result is consistent? Always a green?

---

Problems that I'm facing a lot by running the test suite:

1. Fatal errors

@tcard already explained what happens when a test crash.
I forced a nil pointer exception and the crash is imperceptible by looking at the logs:

Test Suite 'AblySpec.xctest' started at 2016-08-25 16:46:58.399
Test Suite 'RealtimeClientChannel' started at 2016-08-25 16:46:58.400
Test Case '-[AblySpec.RealtimeClientChannel Channel__history__should_return_a_PaginatedResult_page]' started.
2016-08-25 16:46:58.464 xcodebuild[83986:5211227] [MT] IDETestOperationsObserverDebug: (47F45A20-7EF8-4C91-B20A-CEABF3AE2057) Beginning test session 47F45A20-7EF8-4C91-B20A-CEABF3AE2057 with Xcode 7D1014 on target <DVTiPhoneSimulator: 0x7fe1ec641a20> {
            SimDevice: SimDevice : iPhone 5s (0A3E2636-1B03-4AAC-A8A3-9EE5538B92E3) : state={ Booted } deviceType={ SimDeviceType : com.apple.CoreSimulator.SimDeviceType.iPhone-5s } runtime={ SimRuntime : 9.3 (13E230) - com.apple.CoreSimulator.SimRuntime.iOS-9-3 }
} (9.3 (13E230))
2016-08-25 16:46:58.465 xcodebuild[83986:5211268]  IDETestOperationsObserverDebug: Writing diagnostic log for test session to:
/Users/ricardopereira/Library/Developer/Xcode/DerivedData/Ably-ctlkecuaicwkgscexhopiibucqom/Logs/Test/E1DDE652-D5CB-46CF-AFF5-8DAEC84048A0/Session-2016-08-25_16:46:58-CiGUEb.log
16:47:01.855 xctest[84514:5213384] _XCT_testBundleReadyWithProtocolVersion:minimumVersion: reply received
16:47:01.864 xctest[84514:5213384] _IDE_startExecutingTestPlanWithProtocolVersion:16

The test suite stops and we only see a ** TEST FAILED **.

The Realtime specs has a lot of code like realtime.transport as! TestProxyTransport. The transport is created only when the connection starts connecting and when there is an error, e.g. the client couldn't retrieve a valid token, the transport remains nil and a bad access is raised by using a force unwrap. That's the most common.

When this happens, I basically run the test individually and see what happens. In cases like this, should we retry the test?

BTW, the CI use xcpretty indirectly by using scan. xcpretty is a formatter for xcodebuild log but maybe we should have the raw log as well. It could help when we need to investigate a problem. I don't know if Travis supports it but we could try do something like xcodebuild [flags] | tee raw_xcodebuild.log | xcpretty, by using tee and save the raw log.

2. Done closure's called multiple times

Wait asynchronously until the done closure is called or the timeout has been reached.

The done() closure is used to indicate the waiting has completed, so it should be called only once. This error happens randomly, so I suppose that sometimes the event emitter is still emitting after the test has finished.

waitUntil(timeout: testTimeout) { done in
    client.connection.on { stateChange in
        let stateChange = stateChange!
        let state = stateChange.current
        let errorInfo = stateChange.reason
        switch state {
        case .Failed:
            AblyTests.checkError(errorInfo, withAlternative: "Failed state")
            done()
        }
        ...
    }
}

This type of error leads to many consequent errors.

[ricardopereira]

Seems that legacy tests also receives a account blocked (connection limits exceeded) error from sandbox. Need to be fixed.

[19:45:28]: ▸ ✓ testMultipleText_1000_10 (13.919 seconds)
[19:45:37]: ▸ ✓ testPublish_10_1000 (9.616 seconds)
[19:45:48]: ▸ ✗ testPublishArray, ((errorInfo) == nil) failed: "ARTErrorInfo with code 401, message: account blocked (connection limits exceeded)"
[19:45:48]: ▸ ✗ testPublishArray, Asynchronous wait failed: Exceeded timeout of 10 seconds, with unfulfilled expectations: "-[ARTRealtimeMessageTest testPublishArray]".

[mattheworiordan]

Any update on this?

[ricardopereira]

Current failures were fixed with #505 and #506.
Done closure's called multiple times fixed with 75d1815.