Add test coverage for per-thread default streams

[brycelelbach]

PR NVIDIA/thrust#1128 added support for using the per-thread default stream in Thrust if it has been requested via CUDA_API_PER_THREAD_DEFAULT_STREAM. We don't have any tests for this codepath, or in fact any multi-threaded host tests. We should probably add some.

[rongou]

Great idea! I'm seeing some synchronization issues in spark with per-thread default stream enabled, but not sure if the problem is with thrust or cudf (or both). Having some test coverage would definitely help.

[rongou]

@brycelelbach I think I might have some time to work on this. Do you have any idea on how to test this effectively?

[brycelelbach]

Spawn a thread with std::thread, get the stream in the parent thread and the child thread by calling thrust::cuda_cub::stream, then compare them for equality?

We should probably also make this a CMake option so we can run tests with and without this.

[rongou]

Ah yes I think we can easily test that we are using the correct streams. I'm also wondering if we can test the multi-threaded behavior, checking for race conditions, etc.

[alliepiper]

As discussed in today's meeting, we want to update both the per-thread stream and RDC options as follows:

• Add a small number of dedicated tests that always enable / always disable these features (small, quick sanity checks)
• Add a global option that enables / disables these features in all targets, when applicable

This will give each build some coverage, while letting us add wider coverage with CI.