Run Watch tests in CI

[dfed]

Since watchOS does not have an XCTest framework, we created a mock of the XCTest framework and built a watch app to host & run the tests.

This watch app allows us to manually verify that Valet works as expected on the watch, but it doesn't allow us to catch regressions in CI. It'd be great for us to improve this situation by running our watch app in CI.

Since the watch app exits with code 0 after successfully running tests, we're most of the way there. The crux of the remaining challenge is determining how to launch the watch app in the simulator from the command line.

I've done some digging, and it looks like we may need to rely one some of the following commands:

• xcrun simctl list # this command will allow us to determine which sims are on the CI machine, and allow us to find the UUIDs of phone+watch pairs.
• xcrun simctl boot <uuid> # this command will boot the sims with the uuids we've found above. We'll likely need to launch both the phone and watch sims.
• xcrun simctl install <uuid> <path-to-phone-app AND/OR path-to-watch-app> # this command should install the app on the simulator
• xcrun simctl launch <uuid> <path-to-phone-app AND/OR path-to-watch-app> # this command should allow us to launch the app(s) we installed above

I'm likely missing a command above, as I haven't managed to get a script working with the above. Pro-tip: you can monitor the logs emanating from your simulators by running the command xcrun simctl spawn <uuid> log stream. Note that the phone host app is ValetTouchIDTest, and the watch app is Valet watchOS Test Host App.

Help would be greatly appreciated, since I'm brand new to command-line simulator manipulation.

[SlaunchaMan]

FWIW I filed a bug about this (rdar://30201876) in 2017, but I got the following response:

Please know that our engineering team has determined that this issue behaves as intended based on the information provided.

Additionally, my forum post got no replies. For what it’s worth, I think the correct approach here is going to need to be to create a host iOS app that the watchOS app communicates with using WatchConnectivity; the host app would then record the test status and return the correct exit status.

[dfed]

Fascinating @SlaunchaMan. Thanks for the info. I think it should be possible to hack the xcrun output to get the right test in CI... but your approach sounds like it might be a lot simpler 🙂. I'll admit that I'm a bit of a newb on watchOS – last I actually worked on a fully fledged WatchKit app it was built on WatchKit 1.

For what it's worth, I'm super open to help on this one.

[WF-Home]

@dfed is this still relevant with the release of xcode 12.5 adding support to XCTest for watchOS? If it is, I would love to give this issue a shot.

[dfed]

Hi @WF-Home! Thanks for letting me know that Xcode 12.5 was adding this support – that's great news! I'm also excited that you want to help out 🙂

I think there's still value in taking this project on, since this project would enable us to run unit tests on watchOS versions prior to watchOS 7.4.

However... getting our tests compiling on watchOS 7.4 will be problematic if we're still linking the custom XCTest framework – we'll almost certainly encounter a linker failure due to the fact that we've duplicated a bunch of XCTest symbols. If you have ideas about how to solve that issue, I'm all ears.

My guess is the simplest path forward will be to delete the custom XCTest framework, and then create a new CI config to run Watch tests utilizing Xcode 12.5 and watchOS 7.4 once Xcode 12.5 GMs. I'm not sure how much ROI there is to get tests running in CI prior to 7.4. What do you think?

[WF-Home]

@dfed I definitely agree that there is still value in being able to run the unit tests prior to watchOS 7.4 as long as there are still a good amount users on this watchOS version.

As for the linker issue I'm not too sure how that will behave to be completely honest but I'm thinking we could use something like sed to remove all the reference to that custom XCTest on the project.pbxproj when we build for testing on watchOS 7.4+ on the CI but that still leaves a linker issue when testing locally.

Would it make sense to create another target specifically for testing versions lower than 7.4 which will have the reference to the custom XCTest and then remove the reference to XCTest from the existing target? Actually not 100% how the test target would look with watchOS, I'll probably download the beta this weekend just to check it out. Might have a better idea after that.

Disclaimer: I am not an expert on this at all ahaha just a mobile CI/CD and command line enthusiast

[dfed]

Apologies for dropping the ball here. I'm also not entirely sure what this test target would look like, though I'd expect that we would want separate test targets for pre-7.4 and post-7.4. We can likely use the existing test target for pre-7.4, and create a new one for post-7.4 that uses Apple's XCTest target.

I think this Issue should track converting the existing target to test watchOS versions prior to 7.4.

[dfed]

At this point I think the path forward is to roll a Valet 5.0 and drop support for watchOS targets older than 7.4. I don't think that's happening in the near future, but leaving this issue open since:

1. It is still a real issue
2. I intend to fix it as part of a breaking change release