Proposal: Add ability to create multiple test targets with different kwargs

[dostrander]

What

Add an ability to create multiple test targets with the same bundle with different kwargs sent into each rule. Bundle them all into a single test_suite at the end.

Why

Tests on iOS can be long running, specifically UI tests. Bazel has a sharding mechanism that is per target on iOS this means to create multiple sims (1 per shard on the target) to run. These simulators take up a lot of resources and you can't run very many at a time without having simulator and underlying test issues. This proposal will allow for splitting up of test targets to do your own sharding which can be split across multiple machines, thus can lower the time for each "shard"

You could ask then why not just use the --test_filter and have it send the same target to different machines with different filters. The main reason for that is keeping the sanctity of using bazel test //Foo to run all of the shards, similar to how it would be done today. However, if we want to test a specific shard it would be as easy as running bazel test //Foo_shard_1 or however we name it.

Other reasons to create a thing like this would be to create production vs staging targets or any other variances that you would want to have running.

How

Add a new rule to rules_apple to take in a dictionary of suffixes -> kwargs

def split_ios_unit_test_suite(name, splits):

We'd then want to add a way to handle this in the apple_test_assembler

def _assemble_split(name, splits):

This would call into the normal _assemble for creating a test target with the suffix and kwargs passed in, then wrap those tests into a suite on the outside.

Conclusion

This could be way off-base for something like bazel and there maybe better ways to achieve this. However this is something i felt like warranted discussion as I feel this could be a common thing folks run into. Would love feedback and hear folks thoughts ❤️

[jerrymarino]

Having an improved /standard pattern and solution for this would be awesome 👍 . I assumed this was in the context where you have a large MyTestSuite.m with multiple test methods of varying runtime profiles, which typically execute that inside of Xcode.app or bazel build :MyTestSuite.

Today you can have custom fanout in Bazel w/o rules_apple: e.g. generating a **_test for each shard or permutation prior to the analysis phase: a repo_rule parser/generator, or macro:

# a BUILD file that shards out with custom_runner and custom_args
shard1=struct(name="Shard1", custom_args=["--test_filter=MyMethod1"], custom_runner=None)
shard2=struct(name="Shard2", custom_args=["--test_filter=MyMethod2"], testrunner="@build_bazel_rules_ios//:macOSVM_runner")

for shard in [shard1, shard2]:
    ios_unit_test(name=name+shard.name, deps=[":bundle"], shard.custom_args, testrunner=shard.testrunner)

test_suite(name=name + "_suite", tests = [name+shard.name for shard in shards])

Of course, now you'd need to feed custom_args into the runner, templates, and other changes to pull it off!

The other thing to consider is reproducing the CI runs locally with Xcode's specialized GUI inside via rules_ios/XCHammer. Should we somehow indicate if shards should be brought into Xcode or not? Say, you had sharded optimally for remote execution at the method level: how that would work in the GUI? You may want MyTestSuite.m in an Xcode project with or without the target level fanout.. Also, unfortunately some iOS tests out there may have implicit ordering requirements, and of course specialized sharding.

[keith]

In our case we actually took the same splits idea, but instead we just created separate bazel targets from each of those splits instead. At which point they're just kinda "normal" tests, so you get multiple schemes in Xcode etc based on what splits people define. Of course this meant we had to split out any shared code into a separate library they both depended on, but we had actually already done that since we had different targets per app before.

In general doing something the way you're suggesting sounds totally reasonable, I think Jerry's example is pretty close to something that would work and probably makes sense to just have for your use case vs a custom rule? The only thing is right now test selection is only determined using this env var bazel sets

rules_apple/apple/testing/default_runner/ios_test_runner.template.sh

Lines 136 to 140 in 77155d9

	# Use the TESTBRIDGE_TEST_ONLY environment variable set by Bazel's --test_filter
	# flag to set tests_to_run value in ios_test_runner's launch_options.
	# Any test prefixed with '-' will be passed to "skip_tests". Otherwise the tests
	# is passed to "tests_to_run"
	if [[ -n "$TESTBRIDGE_TEST_ONLY" ]]; then

, but I imagine we could accept an argument / attribute and combine that with this logic?

[dostrander]

@keith @jerrymarino Let me play around with this and see what I come up with. I remember trying to do this and there were issues creating the same library twice, specifically this error:

"Error in resources_filegroup: resources_filegroup rule 'Foo_Unit_Tests_data' in package 'Foo' conflicts with existing resources_filegroup rule"

This comes from creating the same test library multiple times under the hood of ios_unit_test here: https://github.com/bazel-ios/rules_ios/blob/master/rules/test.bzl#L65 .

We could abstract that so we can create all the internal libraries (and middlemen) into it's own library then have an optional module passed into ios_unit_test which it will use if it's there and create a new one if it's not

@keith IRT:

In our case we actually took the same splits idea, but instead we just created separate bazel targets from each of those splits instead.

Do you do this manually in the build file? Something like:

apple_framework(name = "Foo", srcs = glob(["Sources/**/*.swift"]))
ios_unit_test(name = "Foo-Unit-Tests_1", srcs = ["Tests/Foo1.swift"])
ios_unit_test(name = "Foo-Unit-Tests_2", srcs = ["Tests/Foo2.swift"])
ios_unit_test(name = "Foo-Unit-Tests_3", srcs = ["Tests/Foo3.swift"])

[keith]

We don't use rules_ios, so I can't comment on any of those issues

We defined a ui_test macro that takes a directory of sources and creates an underlying swift_library, and ios_ui_test. So we have a separate swift_library per ui test, but that works for our setup since we explicitly separate tests in different classes instead of running the same tests with different cases filtering.

[dostrander]

Got it that makes sense. Let me play around with that