Provide a general facility for bazel-in-bazel integration testing

[alexeagle]

Broken out from #11

Summary from that issue:

• There are a few implementations today: go_bazel_test from rules_go, a bespoke test runner in rules_nodejs and rules_python, and @cgrindel recently working on https://github.com/cgrindel/rules_bazel_integration_test
• One issue with these is how to select the workspace files that are part of the test fixture. Bazel makes it hard to glob() the sources from under a directory with BUILD files.
• @ittaiz offers that we could fix up https://github.com/bazelbuild/bazel-integration-testing if we think that's the right thing to do

I'd love for the SIG to figure out what ought to be done, likely reviewing the recent implementation by @cgrindel

[alexeagle]

To avoid churn on developers and keep the ecosystem as simple as possible, my preference would be to make bazelbuild/bazel-integration-testing fit the current need, rather than make a new repo.

[jsharpe]

One thing that I've particularly found challenging to test is different spawn_strategies; in rules_foreign_cc we added a specific configuration to test this but abused the fact that building in -c dbg gives a different cache key as the spawn_strategy isn't factored into the cache key. This means a local spawn_strategy can pick up the build cache from a sandboxed execution run and so doesn't give confidence in the testing of the given strategy; a reliable way of testing this would be great.

[aherrmann]

We've recently reviewed the bazel-in-bazel testing options for rules_haskell.

• https://github.com/cgrindel/rules_bazel_integration_test
  A big plus here is that it can test against multiple Bazel versions, including the option to provide your own Bazel binary. The test runner is also configurable.
• go_bazel_test from rules_go
  This one has the advantage that it constructs a persistent bazel repository to run the tests in, such that subsequent bazel test runs are incremental.

Ideally, we'd have both the testing against multiple versions and the incremental tests. As far as I understand most of the setup of the persistent repository happens inside go_bazel_test's test runner. So, perhaps one could get the best of both worlds by using rules_bazel_integration_test with go_bazel_test's test runner.

That said, go_bazel_test currently always uses the same path for the persistent repository. It may be better to use separate directories for each Bazel version.

[gravypod]

Do either of the existing integration testing libraries have the ability to hook into a RBE environment? We may also want to be able to test permutations of (host_platform, build_platform, target_platform) to make sure cross compiling features work. This is something we need to be careful about in rules_docker.

[cgrindel]

The rules_bazel_integration_test does not have anything that explicitly supports RBE. My understanding is that selection of the various platforms is driven by the flags passed to Bazel or by defaults determined by the host system. So, with Bazel integration tests, are we talking about having the primary Bazel (i.e., the one driving the tests) being given the appropriate flags, or are we talking about executing the "test" Bazel with the appropriate flags (i.e., making the RBE calls)? If the latter, then we would need to pass along RBE configuration to the "test" Bazel. What do you think would need to be added to support this well?

[ittaiz]

Bazel-integration-testing works with RBE.
Don't remember what I needed to get it done but it was hard :)

[alexeagle]

https://github.com/bazel-contrib/rules_bazel_integration_test is now in bazel-contrib, so this should solve the general case.