Test watchOS and tvOS support

[czechboy0]

• watchOS support
• tvOS support

[czechboy0]

WatchOS support will be interesting. Watch Bots actually look exactly like iOS bots, just don't have a testing mode. Investigate more, but in the meantime, having watchOS bots is not that useful anyway (if you can't run tests).

Waiting for Xcode 7.1 to look at tvOS.

[voidrender]

I've been researching this a little and wanted to share some findings. One thing I've found is that many tvOS apps will be a new target in the same Xcode project as the corresponding iOS app. This is so the apps can take advantage of Universal Purchase. The other thing is that tvOS bots seem to look and behave just like iOS bots, and they can run tests just the same.

The iOS and tvOS targets can be built and tested separately using different schemes, but it would be nice if Buildasaur syncers could be created with some kind of filter. For example, if the pull request contains the string "[tvOS]", the syncer configured to build the tvOS scheme will pick up the pull request. If the pull request contains the string "[iOS]", the iOS syncer will pick it up and build using the iOS scheme. Or, maybe this could all be configured into a single syncer.

Is this sort of thing on the roadmap? Is there a way to do this sort of separation of schemes/targets/pull requests using existing Xcode or Buildasaur configuration?

[czechboy0]

Thanks for sharing the findings, @ioveracker!

I noticed that the tvOS bots are pretty much identical to iOS bots, but I haven't actually tested that they work with Buildasaur properly. The main question I have is: how would you want to use them? Do you expect to want to test both iOS & tvOS tests in one bot? Isn't there a way to have an aggregate target which tests both, but is still one scheme?

Buildasaur retains quite a lot of details about the bots & PRs internally, so I don't think it'd be difficult to introduce some sort of PR-based actions or filters, but at the moment I'd like to understand what the ultimate goal is.

[voidrender]

Thinking about a project which contains both iOS and tvOS targets, there will likely be a lot of pull requests that focus just on changes to the iOS target, and likewise, pull requests that focus just on changes to the tvOS target. Because of this, it would be nice to be able to build and test just the applicable target without the added time of building and testing the target that isn't affected by the changes. However, there could also be times where a pull request touches both targets--for example, if there are classes shared between the two targets (for networking code, etc.). In this case, it would be nice if the bot could build and test both targets.

It seems to me like all of those scenarios could be handled by creating schemes specific to the scenario. For example:

1. iOS-only scheme, build and test only the iOS target
2. tvOS-only scheme, build and test only the tvOS target
3. Everything scheme, build and test both the iOS and the tvOS targets

The tricky part will be determining the best way for Buildasaur to know which scheme to use based on the pull request.

The simplest approach would be to always build using the Everything scheme, but this would mean that pull requests changing just tvOS target code would also build and test iOS code, and vice versa. This would add ~10 minutes to our ~4 minute tvOS build. Not a huge deal, but I could see there being a much larger delta between building and testing the targets in some cases.

An alternative would be to use some kind of string filtering based on the pull request title, but that could be error-prone. For example, if the pull request contains the string "[tvOS]", Buildasaur picks it up and builds using a syncer configured to build and test the tvOS target.

Perhaps there is a more clever way to determine which scheme to use, but I'm not familiar enough with the low-level details of the GitHub API or project/scheme metadata.

[voidrender]

By the way, I created a really simple project that contains an iOS target, a tvOS target, and code shared between the two: https://github.com/ioveracker/Zen

This Friday, I'm planning on taking a look at updating Buildasaur to see what happens when it builds that project's tvOS target, but if you're interested in the meantime, that should be a pretty quick way to get started.

[czechboy0]

Thanks, that sounds great. I won't have much time these couple of days unfortunately, so please let me know once you try to get this project building and I can reply to comments then.

[voidrender]

I played around with it a bit today and it was trivial to get syncers going for tvOS. See #258.

As for building both iOS and tvOS targets, what I have going right now for our manual bot setup is as follows:

1. iOS bot which builds our develop branch On Commit,
2. tvOS bot which builds our develop branch On Commit.

After we merge a PR, both bots fire off and report back to our Slack channel with their results. So, I don't have any conditional stuff happening for our manually created bots. I'm starting to think that always building both targets is best because shared code can break for one target and not the other. Perhaps a combined scheme will be the right way to go, but I haven't had any luck getting a manually bot to build a combined scheme. Maybe a syncer that can be configured to build multiple schemes would be a good solution?

[czechboy0]

@ioveracker When you have two syncers operating on the same branch, don't they "fight" over the build status? I think I'll need to make a change to Buildasaur to report each syncer under a unique ID to github so that you can see the results separately, i.e. "Failed on iOS, succeeded on tvOS".

[voidrender]

Both syncers create a bot (with the same name, due to the BotNaming conventions), but only one bot starts an integration. That bot continues as expected, but the other bot never does anything.

I think that's the right path, having both syncers report back to GitHub with a unique ID. Alternatively, if that's not supported by GitHub, Builda could report back with the status only when both syncers have finished.

[voidrender]

I still haven't been able to make a mixed scheme or aggregate target that will work for building both iOS and tvOS targets, but I did come up with a work around today. Run two Buildasaurs! One of them is running on our dedicated CI server and builds the iOS target, the other is running on my workstation and builds the tvOS target.