Feature/openmp

[maddyscientist]

This is mainly a quality of life PR, as well as adding some building work for future work:

• Properly enable OpenMP host compilation
• OMP parallelization of some of the tests codes (gauge force/path, hisq force, fat/long link construction, staggered dslash reference)
• Acceleration and robustness improvement of (staggered_)dslash_ctest; we now use the same gauge field across different partitioning
  • this has an especially dramatic improvement on the execution time for the improved staggered fermion tests when constructing the fat/long links
• some sundry cleanup resulting from the above additions

[mathiaswagner]

Is QUDA_OPENMP exposed in cmake ?

[maddyscientist]

Is QUDA_OPENMP exposed in cmake ?

Yes, it has been for about 5 years 🙂

[maddyscientist]

Latest few commits fix a couple of additional issues I found in testing:

• QUDA_CTEST_LAUNCH needed an additional FORCE to enable the user to override the setting, e.g., to run ctest on different process counts without nuking the build
• For the dslash ctests, make the host quark field creation a one-time event, created when the host gauge field is created. This reduces the test overhead, and ensures we use the random source field for all partitions and precision.

Also noting that the improvements from this PR reduce the ctest overhead on the CSCS CI by over 100 seconds, so a welcome boost to CI throughput.

[mathiaswagner]

@Jenkins test this please.

[mathiaswagner]

@Jenkins test this please.

[mathiaswagner]

@Jenkins test this please.

[weinbe2]

As a thought, in the spirit of the raw_pointer function for the raw gauge field pointer, it'd also be nice to have a raw_pointer_ghost (or whatnot) that returns a void** for CPU-ordered fields

[mathiaswagner]

Is QUDA_OPENMP exposed in cmake ?

Yes, it has been for about 5 years 🙂

Guess it is one of the options I never use.
Should we turn it ON by default?
We could also rely on cmake to detect support and make QUDA_OPENMP=ON mean request OpenMP if supported by the compiler.

[maddyscientist]

Is QUDA_OPENMP exposed in cmake ?

Yes, it has been for about 5 years 🙂

Guess it is one of the options I never use. Should we turn it ON by default? We could also rely on cmake to detect support and make QUDA_OPENMP=ON mean request OpenMP if supported by the compiler.

I didn't turn it on by default since I still consider it very experimental. In particular Eigen has huge slowdowns with OMP enabled, and until we work that out, I don't think we should enable by default.

[weinbe2]

Is QUDA_OPENMP exposed in cmake ?

Yes, it has been for about 5 years 🙂

Guess it is one of the options I never use. Should we turn it ON by default? We could also rely on cmake to detect support and make QUDA_OPENMP=ON mean request OpenMP if supported by the compiler.

I didn't turn it on by default since I still consider it very experimental. In particular Eigen has huge slowdowns with OMP enabled, and until we work that out, I don't think we should enable by default.

I can't test it right now, but it looks like there's a compile-time define EIGEN_DONT_PARALLELIZE (circa 2019...) that can be used to disable OpenMP. Source: https://stackoverflow.com/a/56613641

[weinbe2]

Approved, thanks @maddyscientist !