Enable multiconfig builds for testing / validation

[alliepiper]

Problem

Currently, a Thrust build must be reconfigured and rebuilt to test and validate various host/device system combinations. This quickly gets tedious when testing whether a change will break any of the backends.

Being an NVIDIA product, the core Thrust developers tend to focus on the HOST=CPP DEVICE=CUDA configuration, and bugs sometimes slip into the other backends that go unnoticed until a user reports them.

By simplifying the process of testing other configurations, we can provide a more stable and reliable product to users that rely on the OpenMP and TBB backends. This will also be useful as our CI improves, as we can have a single CI build generate multiple configurations.

Overview

I propose adding a THRUST_ENABLE_MULTICONFIG cmake option. By default, this will be off and everything will work as it does now. But when enabled, this will:

• Hide and ignore the THRUST_HOST_SYSTEM / THRUST_DEVICE_SYSTEM cache options.
• Create new THRUST_MULTICONFIG_WORKLOAD option with settings FULL, LARGE, MEDIUM, SMALL (Default MEDIUM).
• Create new THRUST_MULTICONFIG_ENABLE_${system} boolean options (Default ON).
• Create new THRUST_MULTICONFIG_ENABLE_CPPXX boolean options for C++11, C++14, and C++17. Default is just C++14.
• Generate tests, examples, and header-tests for a variety of host/device configuration.

Configurations

The THRUST_MULTICONFIG_ENABLE_${system} options are used to specify the expected systems that a particular build should use. Dependencies of disabled systems will not be searched for, and any configurations that use a disabled system will not be tested. If a system is requested but an associated dependency is not found, a configuration error will occur.

The THRUST_MULTICONFIG_ENABLE_CPP11, THRUST_MULTICONFIG_ENABLE_CPP14, and THRUST_MULTICONFIG_ENABLE_CPP17 options will produce configurations for each of the indicated C++ dialects. By default, only C++14 is enabled.

The THRUST_MULTICONFIG_WORKLOAD option controls how much of the configuration space is tested. The full cross product of HOST={CPP, TBB, OMP} and DEVICE={CUDA, TBB, OMP, CPP} is quite expensive to build, so the workload may be adapted to suite available resources.

The full configuration space is divided into groups that maximize coverage while eliminating redundant or unusual configurations and minimizing expense. This provides a convenient way to balance the amount of coverage needed for a particular usecase (local pre-CI testing, CI, pre-release testing, etc) with the amount of computational resources.

• SMALL: [3 configs] Minimal coverage and validation of each device system against the CPP host.
• MEDIUM: [6 configs] Cheap extended coverage.
• LARGE: [8 configs] Expensive extended coverage. Include all useful build configurations.
• FULL: [12 configs] The complete cross product of all possible build configurations.

Config	FULL	LARGE	MEDIUM	SMALL	Value	Expense	Note
CPP/CUDA	X	X	X	X	Essential	Expensive	Validates CUDA against CPP
CPP/OMP	X	X	X	X	Essential	Cheap	Validates OMP against CPP
CPP/TBB	X	X	X	X	Essential	Cheap	Validates TBB against CPP
CPP/CPP	X	X	X		Important	Cheap	Tests CPP as device
OMP/OMP	X	X	X		Important	Cheap	Tests OMP as host
TBB/TBB	X	X	X		Important	Cheap	Tests TBB as host
TBB/CUDA	X	X			Important	Expensive	Validates TBB/CUDA interop
OMP/CUDA	X	X			Important	Expensive	Validates OMP/CUDA interop
TBB/OMP	X				Not useful	Cheap	Mixes CPU-parallel systems
OMP/TBB	X				Not useful	Cheap	Mixes CPU-parallel systems
TBB/CPP	X				Not Useful	Cheap	Parallel host, serial device
OMP/CPP	X				Not Useful	Cheap	Parallel host, serial device

Target Naming

The test/example names will need to be modified to reflect which configuration the executable belongs to. For instance:

thrust.[test|example].name --> thrust.[cpp|omp|tbb].[cuda|cpp|omp|tbb].cpp[11|14|17].[test|example].yyy

For non-multiconfig builds, the names will stay the same as they currently are.

Meta-targets

Custom CMake targets will be added to make it easier to build subsets of a single configuration.

Target	Description
thrust.<host>.<device>.<dialect>.all	All targets associated with the configuration
thrust.<host>.<device>.<dialect>.headers	All header tests associated with the configuration
thrust.<host>.<device>.<dialect>.tests	All tests associated with the configuration
thrust.<host>.<device>.<dialect>.examples	All examples associated with the configuration
`thrust.meta.[test	example].{name}`

Testing

Tests can be filtered by configuration by using -R <regex> when invoking CTest, e.g.:

ctest -R tbb.cuda.cpp17

Build Time Considerations

A detailed exploration of the compile time requirements of each configuration is in this comment. An overview of the relative build times for each workload and dialect is shown below.

[jaredhoberock]

The cross product I used to test with Jenkins was

{cpp, omp, tbb} X {cuda, omp, tbb}

I don't know how useful it is to test e.g. host=tbb, device=tbb, but it's a possible configuration, so there must be some users out there doing that.

[alliepiper]

Ok, I'll start with that full cross product and see how expensive it is to build. If it's too much, we can split the configs up into a full set (for CI) and a "common" set for quick (well, for some definition of 'quick') local testing.

[jaredhoberock]

SGTM

I'm not familiar with the current state of Thrust's build, but I'd say the most valuable things we could do would be something like this:

1. Make the common build painless enough that it can be run frequently, and
2. Make any build combination convenient to initiate

[alliepiper]

I've updated the "Configurations" section of the issue with something that I think will work for most cases.

re: the current state of things, by default we just build and test CPP/CUDA. We can change the host and/or device for the entire build by reconfiguring with CMake.

After this change, that will still be the default behavior. This is most useful for feature development and iterative changes targeted at a single backend.

Turning on THRUST_ENABLE_MULTICONFIG will build and test multiple configs at a time. A THRUST_MULTICONFIG_WORKLOAD option will allow the number of configs to be tuned for hardware / usecase while optimizing coverage.

[brycelelbach]

If we do this, we should probably do this for all the other variants, which I believe is:

• C++ dialect (at least for this)
• SM architecture (for CUDA backend)
• RDC (for CUDA backend)

e.g. thrust.xxx.yyy.HOST_CPP.DEVICE_CUDA.CPP_DIALECT

Maybe we should do at least C++ dialect in this PR.

[alliepiper]

Updated proposal with C++ dialect options for C++11, 14, and 17. It defaults to just C++17 since I figure that'll give the most coverage.

[alliepiper]

Updated names to be less harsh and more useful for categorizing targets:

thrust.test.reduce.HOST_CPP.DEVICE_CUDA.CPP14 is now thrust.cpp.cuda.cpp14.test.reduce.

[alliepiper]

Here are some of my findings from testing out multiconfig builds this weekend. Hardware specs and raw data are in this gist. This is on my NVIDIA provided workstation, so these numbers should be similar (or better) for the other members of the NVIDIA developer team.

There is currently an issue preventing the tbb.cuda configurations from building. These build times are estimated to be similar to the cpp.cuda and omp.cuda measurements in the discussion below.

MultiConfig Cost Summary

The key takeaways are:

• CPU backends are practically free. For a FULL build with all dialects enabled, the non-CUDA backends take up ~12% of the total build time -- a little over an hour for all 27 CPU-only host/device/dialect combinations.
• Most of the expense comes from the CUDA backends, which account for the remaining ~88% of the total build time. The 9 host/cuda/dialect configurations take around an hour each.

The workload grouping has been updated to reflect this:

• SMALL Minimal coverage: cpp.cuda, cpp.omp, cpp.tbb.
• MEDIUM Cheap extended coverage: Adds cpp.cpp, omp.omp, tbb.tbb.
• LARGE Expensive extended coverage: Adds tbb.cuda, omp.cuda.
• FULL Complete coverage: Adds omp.cpp, omp.tbb, tbb.cpp, tbb.omp.

The MEDIUM workload seems to be the best value, as it adds all useful CPU-only systems to a cpp.cuda build for only ~12 minutes/dialect (a 20% increase over ~60 minutes/dialect for just cpp.cuda).

LARGE workloads bring in the additional heavy [omp|tbb].cuda configurations for a cost of ~2 hours/dialect. This provides complete coverage of all useful configurations.

Cost of increasing workload

To determine which workload to build, it's useful to know the cost of increasing to the next largest workload. This section tabulates the time needed to build only the new configurations in each workload.

Workload	cpp11	cpp14	cpp17	All Dialects
cpp.cuda	0:56:52	0:54:59	0:55:27	2:47:18
SMALL	0:04:21	0:04:21	0:04:28	0:13:10
MEDIUM	0:07:41	0:07:33	0:08:00	0:23:14
LARGE	2:05:31	0:02:36	2:06:39	6:14:46
FULL	0:11:03	0:11:24	0:11:43	0:34:10

Total build time per workload

To estimate the amount of time needed for a given workload / dialect combination, this section compares the full build times.

Workload	cpp11	cpp14	cpp17	All Dialects
cpp.cuda	0:56:52	0:54:59	0:55:27	2:47:18
SMALL	1:01:13	0:59:20	0:59:55	3:00:28
MEDIUM	1:08:54	1:06:53	1:07:55	3:23:42
LARGE	3:14:25	3:09:29	3:14:34	9:38:28
FULL	3:25:28	3:20:53	3:26:17	10:12:38

Details

CPUs backends are super cheap

Each build that doesn't require CUDA takes 1-4 minutes. Using the host==device configs to compare their build times, the trend here is about what you'd expect:

Config	Build Time
cpp.cpp	~1:35
omp.omp	~2:20
tbb.tbb	~3:45

The full cross product of the CPU backends -- including the unlikely/silly combos -- only takes about 25 minutes to compile for a single cpp dialect.

Most of the time is spent building CUDA configs

All configurations that use the CUDA backend take a full hour to build when all SM versions are enabled (except 5.3, 6.2, and 7.2, which don't build with RDC). Most of this is due to toolchain overhead in nvcc, but there's also more CUDA code to compile. A non-CUDA build has ~515 targets, but CUDA has ~895.

Some CUDA targets are very expensive

There are a few monster translation units that take a very long time on CUDA.

The most expensive targets in the cpp.cuda.cpp14 configuration are:

Translation Unit	Build Time
testing/scan.cu	~25:00
testing/scan_by_key.cu	~20:00
testing/minmax_element.cu	~17:00
testing/min_element.cu	~16:00
testing/max_element.cu	~16:00
testing/async_sort.cu	~13:00

and quite a few others in the 6-10 minute range. A timeline of the cpp.cuda.cpp14 build on 12 cores is shown below.

Download an interactive html version here.

[alliepiper]

Did some additional tests varying the number of target SMs on cpp.cuda.cpp14:

#SMs	compute_75	Walltime (-j13)
0	Yes	10 minutes
1	No	11 minutes
1 (SM75)	Yes	11 minutes
2	No	17 minutes
3	No	23 minutes
4	No	29 minutes
5	No	34 minutes
6	No	40 minutes
9	Yes	60 minutes

I've switched to just building the 2 SMs I can test locally, and this changes the total build time to for MEDIUM/cpp14 to around 30 minutes. This seems like a reasonable amount of coverage/expense for pre-commit testing.