Tempus_*_MPI_1 failing in Trilinos-atdm-ats1-knl_intel-19.0.4_mpich-7.7.15_openmp_static_opt build starting 2021-10-15

[ZUUL42]

CC: @trilinos/tempus, @ccober6 (Trilinos Nonlinear Solvers / Discretizations Triage/ATDM Contact)

Next Action Status

Description

As shown in this query (click "Shown Matching Output" in upper right) the tests:

• Tempus_BDF2_MPI_1
• Tempus_BackwardEuler_MPI_1
• Tempus_DIRK_MPI_1
• Tempus_ExplicitRK_MPI_1
• Tempus_UnitTest_ERK_MPI_1

in the builds:

• Trilinos-atdm-ats1-knl_intel-19.0.4_mpich-7.7.15_openmp_static_opt

started failing on testing day 2021-10-15.

Error: srun: error: nid00195: task 0: Floating point exception (core dumped)

Current Status on CDash

Run the above query adjusting the "Begin" and "End" dates to match today any other date range or just click "CURRENT" in the top bar to see results for the current testing day.

Steps to Reproduce

One should be able to reproduce this failure as described in:

• https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md

and the system-specific instructions at:

• https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#specific-instructions-for-each-system

Just log into any of the associated machines and copy and paste the full CDash build name <build-name> listed above and run commands like:

$ cd <some_build_dir>/

$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh <build-name>

$ cmake \
 -GNinja \
 -DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
 -DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_<package-name>=ON \
 $TRILINOS_DIR

$ make NP=16

$ <command-to-run-on-compute-node> ctest -j4

where <package-name> is any package that you want to enable to reproduce build and/or test results.

Again, for exact system-specific details on what commands to run to build and run tests, see:

• https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#specific-instructions-for-each-system

If you can't figure out what commands to run to reproduce the problem given this documentation, then please post a comment here and we will give you the exact minimal commands.

[ccober6]

@ZUUL42 is this still a problem. I think the above merge should have fixed it.

[ZUUL42]

@ccober6, the CDash query seems to indicate this failure hasn't presented itself since 11/24.
So, ya, I think we can call this one closed.