Custom test runner for iOS target

[ildarsharafutdinov]

hi,

This is basically follow up of #1620, which states Currently Gradle plugin doesn't provide a testing DSL but this is in short-term plans..
Is there news on that?

The question is caused by the fact there is the only way to run K/N framework's tests on iOS: build K/N part as executable(instead of framework) with gradle task linkTestDebugExecutableIos. Unfortunately its not always desirable. E.g. framework can depend on the host app resources/plist/other frameworks/etc.

I guess building K/N part as framework with tests(linkTestDebugFrameworkIos doesn't work) and "overriding" K/N test runner will be enough to run tests in the host app(the same way Xcode does).

Is there a way to make framework include tests/get a list of tests/override K/N test runner to achieve something like that?

[kylejbrock]

I'm interacting with the KeyChain, so I have to execute my tests with a host app. I tried to do this without using XCode, but the amount of work required was too much to make it worth it. Instead, the approach I took was to create a "TestBridge" interface in Kotlin, plus a "TestExpectation". Then I implement those interfaces in swift like so:

import XCTest
@testable import RED_Test_App
@testable import test

class TestExpectation : NSObject, Expectation {
    
    private let expectation: XCTestExpectation
    
    init(expectation: XCTestExpectation) {
        self.expectation = expectation
    }
    
    func fulfill() {
        self.expectation.fulfill()
    }    
}

class RED_Test_AppTests: XCTestCase, TestBridge {
    
    func doNewPlatform() -> RedCorePlatform {
        let path = NSSearchPathForDirectoriesInDomains(.documentDirectory, .userDomainMask, true)[0] as String
        return RedCorePlatform(applicationContext: RedPlatformContext())
    }
    
    func doNewExpectation() -> Expectation {
        return TestExpectation(expectation: expectation(description: "expectation"))
    }
    
    func waitForExpectations() {
        waitForExpectations(timeout: 30, handler: nil)
    }
    
    func testRedTestSuite() {
        TestBridgeHandle().bridge = self
        XCTAssertEqual(TestSuite().run(), 0)
    }

}

In Kotlin, I have:

import kotlin.native.internal.test.testLauncherEntryPoint

object TestSuite {

    fun run(): Int = testLauncherEntryPoint(emptyArray())
}

The testLauncherEntryPoint(emptyArray()) is the hook into the Kotlin generated test suite.

Then, in gradle, I have a task:

task iosTest {
    def device = project.findProperty("iosDevice")?.toString() ?: "iPhone XS Max"
    dependsOn "linkTestDebugFrameworkIos"
    group = JavaBasePlugin.VERIFICATION_GROUP
    description = "Runs tests for target 'ios' on an iOS simulator"

    doLast {
        def binary = project.kotlin.targets.ios.compilations.test.getBinary("FRAMEWORK", "DEBUG")
        copyIosResources(binary)
        copyIosTestResources(binary)
        println("Running on Device: $device :: $binary")
        exec {
            commandLine "xcrun", "simctl", "boot", device
        }
        exec {
            workingDir = file("$rootProject.projectDir/ios/iosTestApp/RED Test App")
            commandLine "xcodebuild", "test", "-scheme", "RED Test App", "-destination", "platform=iOS Simulator,name=$device"
        }
        exec {
            commandLine "xcrun", "simctl", "shutdown", device
        }
    }
}

It will boot the simulator, execute the tests using XCode, and then shutdown the simulator.

I also have a task that will reset the simulator if it ever gets in a funky state:

xcrun simctl shutdown "$1"
xcrun simctl erase "$1"

Finally, my BaseTest class in Kotlin common code looks like this:

interface TestBridge {
    fun newPlatform(): CorePlatform
    fun newExpectation(): Expectation
    fun waitForExpectations()
}

interface Expectation {

    fun fulfill()
}

object TestBridgeHandle {

    private val _bridge = AtomicReference(defaultTestBridge())

    var bridge: TestBridge
        get() = _bridge.value
        set(value) {
            _bridge.set(value)
        }
}

open class BaseTest {

    init {
        Core.staging = true
        Core.initialize(
            TestBridgeHandle.bridge.newPlatform(),
            clientId,
            clientSecret
        )
    }

    fun newExpectation(): Expectation = TestBridgeHandle.bridge.newExpectation()

    fun waitForExpectations() = TestBridgeHandle.bridge.waitForExpectations()

    fun <T> assertSuccess(future: Future<T>): T? {
        waitFor(future)
        future.error?.printStackTrace()
        assertNull(future.error)
        assertTrue(future.complete)
        return future.value
    }

    fun <T> waitFor(future: Future<T>): Future<T> {
        val expectation = newExpectation()
        future.addCallback { _, _ ->
            expectation.fulfill()
        }
        waitForExpectations()
        assertTrue(future.complete)
        return future
    }
}

With this approach, I'm able to have Android & iOS tests written in common code that execute in their respective emulators & simulators.

Hope this helps.

[ildarsharafutdinov]

@kylejbrock ,

Thank you.

testLauncherEntryPoint(emptyArray()) is probably what I was looking for.

Also dependsOn "linkTestDebugFrameworkIos" results in Task with path 'linkTestDebugFrameworkIos' not found in my case.
./gradlew tasks displays only the following ones:

linkDebugFrameworkIos - Links an Objective-C framework from the 'main' compilation for target 'native'.
linkReleaseFrameworkIos - Links an Objective-C framework from the 'main' compilation for target 'native'.
linkTestDebugExecutableIos - Links an executable from the 'test' compilation for target 'native'.

Did you do any additional setup?

[kylejbrock]

Try this, specifically compilations.test.outputKinds("FRAMEWORK"):

kotlin {
    targets {
            fromPreset(presets.android, "android")
            fromPreset(getIosPreset(), "ios") {
                compilations.main.outputKinds("FRAMEWORK")
                compilations.test.outputKinds("FRAMEWORK")
            }
        }
}

I think that's what will cause linkTestDebugFrameworkIos to be available.

[ildarsharafutdinov]

@kylejbrock ,

You're right, compilations.test.outputKinds("FRAMEWORK") makes linkTestDebugFrameworkIos available.
Thanks.

[cquemin]

Am I correct to assume that CorePlatform and defaultTestBridge() are classes that you have created? I have been trying to implement your workaround with kotlin mpp 1.3.20 and few things are different. Would you be able to share a sample project with your workaround @kylejbrock or @ildarsharafutdinov ?

[ildarsharafutdinov]

@cquemin ,

I introduced an empty protocol to mark all Xcode specific tests. XCTestSuite.defaultTestSuite uses obj-c runtime to find all classes which implement MyXcodeTestMarkerProtocol and creates an instance of XCTestsCase for each test with + (instancetype)testCaseWithSelector:(SEL)selector;.

This allows me to have a number of tests which are run within XCTest environment.

@interface KtAllTests: XCTestCase

@end

@implementation KtAllTests

+ (XCTestSuite *)defaultTestSuite {
    __auto_type testCaseClasses = [self kotlinTestCaseClasses]; // find all classes which implement MyXcodeTestMarkerProtocol
    __auto_type testCases = [self allTestCasesFrom:testCaseClasses]; // create an instance of XCTestCase for every test in each MyXcodeTestMarkerProtocol 
    
    XCTestSuite *suite = [[XCTestSuite alloc] initWithName:NSStringFromClass(self.class)];
    for(XCTestCase *test in testCases) {
        [suite addTest:test];
    }
     
    return suite;
}

...
@end 

[kylejbrock]

@cquemin Yes, it is. I can't post everything, but here are some more details.

For iOS (Swift):

class TestExpectation : NSObject, Expectation {
    
    private let expectation: XCTestExpectation
    
    init(expectation: XCTestExpectation) {
        self.expectation = expectation
    }
    
    func fulfill() {
        self.expectation.fulfill()
    }
    
}

class RED_Test_AppTests: XCTestCase, TestBridge {
    
    func doNewPlatform() -> RedCorePlatform {
        let path = NSSearchPathForDirectoriesInDomains(.documentDirectory, .userDomainMask, true)[0] as String
        return RedCorePlatform(applicationContext: RedPlatformContext())
    }
    
    func doNewExpectation() -> Expectation {
        return TestExpectation(expectation: expectation(description: "expectation"))
    }
    
    func waitForExpectations() {
        waitForExpectations(timeout: 30, handler: nil)
    }
    
    func testRedTestSuite() {
        TestBridgeHandle().bridge = self
        XCTAssertEqual(TestSuite().run(), 0)
    }

}

For Android:

internal class TestBridgeImpl : TestBridge {

    internal val condition = Condition()
    internal val waitingExpectations = AtomicInt(0)
    internal val waiting = AtomicBoolean()

    init {
        (Dispatchers.mainDispatcher as MainDispatcher).notifier.setAssertTrue(object : DispatchNotifier {
            override fun dispatched() {
                condition.signalAll()
            }
        })
    }

    override fun newPlatform(): CorePlatform =
        CorePlatform(ApplicationProvider.getApplicationContext())

    override fun newExpectation(): Expectation {
        if (waiting.get())
            throw IllegalStateException("Already waiting")
        waitingExpectations.incrementAndGet()
        return ExpectationImpl(this)
    }

    override fun waitForExpectations() {
        waiting.set(true)
        try {
            while (waitingExpectations.get() > 0) {
                condition.await(60_000)
            }
        } finally {
            waiting.set(false)
        }
    }
}

actual fun defaultTestBridge(): TestBridge = TestBridgeImpl()

actual fun runTest(block: () -> Unit) {
    block()
}

internal class ExpectationImpl(val bridge: TestBridgeImpl) : Expectation {

    override fun fulfill() {
        bridge.waitingExpectations.decrementAndGet()
        bridge.condition.signalAll()
    }
}

[evant]

I would defiantly be interested in an XCTest runner backend for kotlin multiplatform tests. This would make it easier to define cross-platform integration tests and have better xcode integration for running them.

[cquemin]

Same here to be honest. I still haven't found an easy way to have integration test that run both on Android and iOS with an app attached to it.

…

On Mon, 3 Jun 2019, 13:38 Evan Tatarka, ***@***.***> wrote: I would defiantly be interested in an XCTest runner backend for kotlin multiplatform tests. This would make it easier to define cross-platform integration tests and have better xcode integration for running them. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2362?email_source=notifications&email_token=AAPUABS2HY4WHOMQ7ZBBX73PYV6NZA5CNFSM4GFVAZ32YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODW2T64A#issuecomment-498417520>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAPUABS275UZLUUVK75CAFLPYV6NZANCNFSM4GFVAZ3Q> .

[SvyatoslavScherbina]

We are closing issue tracking on GitHub, please follow https://youtrack.jetbrains.com/issue/KT-48089 instead.