Ifpack2_MTSGS_belos_MPI_1 randomly failing in build Trilinos-atdm-cee-rhel6_intel-18.0.2_mpich2-3.2_openmp_static_opt starting before 2019-04-24

[fryeguy52]

Bug Report

CC: @trilinos/ifpack2, @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe, @fryeguy52

Next Action Status

Description

As shown in this query the test:

• Ifpack2_MTSGS_belos_MPI_1

is failing in the build:

• Trilinos-atdm-cee-rhel6_intel-18.0.2_mpich2-3.2_openmp_static_opt
• Trilinos-atdm-ats1-knl_­intel-19.0.4_­mpich-7.7.15_­openmp_­static_­opt

This has failed 9 times in the last 4 weeks with something similar to:

Achieved tolerance: 6.44895e-10
Actual iters(20) > expected number of iterations (19), or resid-norm(0) >= 1.e-7
proc 0 total program time: 0.0258739
End Result: TEST FAILED

Current Status on CDash

Current 2 week history on CDash

Steps to Reproduce

One should be able to reproduce this failure on a machine with a cee rhel6 environment as described in:

• https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md

More specifically, the commands given for a machine with a cee rhel6 environment are provided at:

• https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#cee-rhel6-environment

The exact commands to reproduce this issue should be:

$ cd <some_build_dir>/
$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh Trilinos-atdm-cee-rhel6_intel-18.0.2_mpich2-3.2_openmp_static_opt
$ cmake \
 -GNinja \
 -DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
 -DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_Ifpack2=ON \
 $TRILINOS_DIR
$ make NP=16
$ ctest -j16

[bartlettroscoe]

FYI: I think this thing started randomly failing when we (based on @nmhamster's suggestion) updated OMP_NUM_THREADS from 2 to 3 for this build. This just shows we need to be testing with a greater variety of threads (hence #2422).

[grover-trilinos]

Test results for issue #5006 as of 2020-08-16

Tests with issue trackers Passed: twip=1

Detailed test results: (click to expand)

Tests with issue trackers Passed: twip=1

Site	Build Name	Test Name	Status	Details	Consec­utive Pass Days	Non-pass Last 30 Days	Pass Last 30 Days	Issue Tracker
cee-rhel6	Trilinos-atdm-cee-rhel6_­intel-18.0.2_­mpich2-3.2_­openmp_­static_­opt	Ifpack2_­MTSGS_­belos_­MPI_­1	Passed	Completed	3	11	19	#5006

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

[grover-trilinos]

Test results for issue #5006 as of 2020-08-23

Tests with issue trackers Failed: twif=1

Detailed test results: (click to expand)

Tests with issue trackers Failed: twif=1

Site	Build Name	Test Name	Status	Details	Consec­utive Non-pass Days	Non-pass Last 30 Days	Pass Last 30 Days	Issue Tracker
cee-rhel6	Trilinos-atdm-cee-rhel6_­intel-18.0.2_­mpich2-3.2_­openmp_­static_­opt	Ifpack2_­MTSGS_­belos_­MPI_­1	Failed	Completed (Failed)	1	11	18	#5006

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

[grover-trilinos]

Test results for issue #5006 as of 2020-08-30

Tests with issue trackers Failed: twif=1

Detailed test results: (click to expand)

Tests with issue trackers Failed: twif=1

Site	Build Name	Test Name	Status	Details	Consec­utive Non-pass Days	Non-pass Last 30 Days	Pass Last 30 Days	Issue Tracker
cee-rhel6	Trilinos-atdm-cee-rhel6_­intel-18.0.2_­mpich2-3.2_­openmp_­static_­opt	Ifpack2_­MTSGS_­belos_­MPI_­1	Failed	Completed (Failed)	1	10	19	#5006

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

[grover-trilinos]

Test results for issue #5006 as of 2020-09-06

Tests with issue trackers Passed: twip=1

Detailed test results: (click to expand)

Tests with issue trackers Passed: twip=1

Site	Build Name	Test Name	Status	Details	Consec­utive Pass Days	Non-pass Last 30 Days	Pass Last 30 Days	Issue Tracker
cee-rhel6	Trilinos-atdm-cee-rhel6_­intel-19.0.3_­mpich2-3.2_­openmp_­static_­opt	Ifpack2_­MTSGS_­belos_­MPI_­1	Passed	Completed	2	0	2	#5006

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

[grover-trilinos]

Test results for issue #5006 as of 2020-09-13

Tests with issue trackers Passed: twip=1

Detailed test results: (click to expand)

Tests with issue trackers Passed: twip=1

Site	Build Name	Test Name	Status	Details	Consec­utive Pass Days	Non-pass Last 30 Days	Pass Last 30 Days	Issue Tracker
cee-rhel6	Trilinos-atdm-cee-rhel6_­intel-19.0.3_­mpich2-3.2_­openmp_­static_­opt	Ifpack2_­MTSGS_­belos_­MPI_­1	Passed	Completed	1	3	6	#5006

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

[grover-trilinos]

Test results for issue #5006 as of 2020-09-20

Tests with issue trackers Passed: twip=1

Detailed test results: (click to expand)

Tests with issue trackers Passed: twip=1

Site	Build Name	Test Name	Status	Details	Consec­utive Pass Days	Non-pass Last 30 Days	Pass Last 30 Days	Issue Tracker
cee-rhel6	Trilinos-atdm-cee-rhel6_­intel-19.0.3_­mpich2-3.2_­openmp_­static_­opt	Ifpack2_­MTSGS_­belos_­MPI_­1	Passed	Completed	1	5	11	#5006

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

[grover-trilinos]

Test results for issue #5006 as of 2020-09-27

Tests with issue trackers Passed: twip=1

Detailed test results: (click to expand)

Tests with issue trackers Passed: twip=1

Site	Build Name	Test Name	Status	Details	Consec­utive Pass Days	Non-pass Last 30 Days	Pass Last 30 Days	Issue Tracker
cee-rhel6	Trilinos-atdm-cee-rhel6_­intel-19.0.3_­mpich2-3.2_­openmp_­static_­opt	Ifpack2_­MTSGS_­belos_­MPI_­1	Passed	Completed	4	7	16	#5006

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

[grover-trilinos]

Test results for issue #5006 as of 2020-10-04

Tests with issue trackers Passed: twip=1

Detailed test results: (click to expand)

Tests with issue trackers Passed: twip=1

Site	Build Name	Test Name	Status	Details	Consec­utive Pass Days	Non-pass Last 30 Days	Pass Last 30 Days	Issue Tracker
cee-rhel6	Trilinos-atdm-cee-rhel6_­intel-19.0.3_­mpich2-3.2_­openmp_­static_­opt	Ifpack2_­MTSGS_­belos_­MPI_­1	Passed	Completed	1	9	21	#5006

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

[grover-trilinos]

Test results for issue #5006 as of 2020-10-11

Tests with issue trackers Passed: twip=1

Detailed test results: (click to expand)

Tests with issue trackers Passed: twip=1

Site	Build Name	Test Name	Status	Details	Consec­utive Pass Days	Non-pass Last 30 Days	Pass Last 30 Days	Issue Tracker
cee-rhel6	Trilinos-atdm-cee-rhel6_­intel-19.0.3_­mpich2-3.2_­openmp_­static_­opt	Ifpack2_­MTSGS_­belos_­MPI_­1	Passed	Completed	8	7	23	#5006

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

[grover-trilinos]

Test results for issue #5006 as of 2020-10-18

Tests with issue trackers Passed: twip=1

Detailed test results: (click to expand)

Tests with issue trackers Passed: twip=1

Site	Build Name	Test Name	Status	Details	Consec­utive Pass Days	Non-pass Last 30 Days	Pass Last 30 Days	Issue Tracker
cee-rhel6	Trilinos-atdm-cee-rhel6_­intel-19.0.3_­mpich2-3.2_­openmp_­static_­opt	Ifpack2_­MTSGS_­belos_­MPI_­1	Passed	Completed	15	5	25	#5006

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

[grover-trilinos]

Test results for issue #5006 as of 2020-10-25

Tests with issue trackers Passed: twip=1

Detailed test results: (click to expand)

Tests with issue trackers Passed: twip=1

Site	Build Name	Test Name	Status	Details	Consec­utive Pass Days	Non-pass Last 30 Days	Pass Last 30 Days	Issue Tracker
cee-rhel6	Trilinos-atdm-cee-rhel6_­intel-19.0.3_­mpich2-3.2_­openmp_­static_­opt	Ifpack2_­MTSGS_­belos_­MPI_­1	Passed	Completed	6	3	27	#5006

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

[grover-trilinos]

Test results for issue #5006 as of 2020-11-01

Tests with issue trackers Passed: twip=1

Detailed test results: (click to expand)

Tests with issue trackers Passed: twip=1

Site	Build Name	Test Name	Status	Details	Consec­utive Pass Days	Non-pass Last 30 Days	Pass Last 30 Days	Issue Tracker
cee-rhel6	Trilinos-atdm-cee-rhel6_­intel-19.0.3_­mpich2-3.2_­openmp_­static_­opt	Ifpack2_­MTSGS_­belos_­MPI_­1	Passed	Completed	13	2	28	#5006

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

[grover-trilinos]

Test results for issue #5006 as of 2020-11-08

Tests with issue trackers Passed: twip=1

Detailed test results: (click to expand)

Tests with issue trackers Passed: twip=1

Site	Build Name	Test Name	Status	Details	Consec­utive Pass Days	Non-pass Last 30 Days	Pass Last 30 Days	Issue Tracker
cee-rhel6	Trilinos-atdm-cee-rhel6_­intel-19.0.3_­mpich2-3.2_­openmp_­static_­opt	Ifpack2_­MTSGS_­belos_­MPI_­1	Passed	Completed	20	1	29	#5006

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

[grover-trilinos]

Test results for issue #5006 as of 2020-11-15

Tests with issue trackers Passed: twip=1

Detailed test results: (click to expand)

Tests with issue trackers Passed: twip=1

Site	Build Name	Test Name	Status	Details	Consec­utive Pass Days	Non-pass Last 30 Days	Pass Last 30 Days	Issue Tracker
cee-rhel6	Trilinos-atdm-cee-rhel6_­intel-19.0.3_­mpich2-3.2_­openmp_­static_­opt	Ifpack2_­MTSGS_­belos_­MPI_­1	Passed	Completed	27	1	29	#5006

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

[grover-trilinos]

Test results for issue #5006 as of 2020-11-22

Tests with issue trackers Passed: twip=1

Detailed test results: (click to expand)

Tests with issue trackers Passed: twip=1

Site	Build Name	Test Name	Status	Details	Consec­utive Pass Days	Non-pass Last 30 Days	Pass Last 30 Days	Issue Tracker
cee-rhel6	Trilinos-atdm-cee-rhel6_­intel-19.0.3_­mpich2-3.2_­openmp_­static_­opt	Ifpack2_­MTSGS_­belos_­MPI_­1	Passed	Completed	5	1	29	#5006

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

[grover-trilinos]

Test results for issue #5006 as of 2020-11-29

Tests with issue trackers Passed: twip=1

Detailed test results: (click to expand)

Tests with issue trackers Passed: twip=1

Site	Build Name	Test Name	Status	Details	Consec­utive Pass Days	Non-pass Last 30 Days	Pass Last 30 Days	Issue Tracker
cee-rhel6	Trilinos-atdm-cee-rhel6_­intel-19.0.3_­mpich2-3.2_­openmp_­static_­opt	Ifpack2_­MTSGS_­belos_­MPI_­1	Passed	Completed	12	1	29	#5006

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

[grover-trilinos]

Test results for issue #5006 as of 2021-04-25

Tests with issue trackers Passed: twip=1

Detailed test results: (click to expand)

Tests with issue trackers Passed: twip=1

Site	Build Name	Test Name	Status	Details	Consec­utive Pass Days	Non-pass Last 30 Days	Pass Last 30 Days	Issue Tracker
cee-rhel7	Trilinos-atdm-cee-rhel7_­intel-19.0.3_­mpich2-3.2_­openmp_­static_­opt	Ifpack2_­MTSGS_­belos_­MPI_­1	Passed	Completed	16	2	28	#5006

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

[grover-trilinos]

Test results for issue #5006 as of 2021-05-02

Tests with issue trackers Passed: twip=2

Detailed test results: (click to expand)

Tests with issue trackers Passed: twip=2

Site	Build Name	Test Name	Status	Details	Consec­utive Pass Days	Non-pass Last 30 Days	Pass Last 30 Days	Issue Tracker
mutrino	Trilinos-atdm-ats1-knl_­intel-19.0.4_­mpich-7.7.15_­openmp_­static_­opt	Ifpack2_­MTSGS_­belos_­MPI_­1	Passed	Completed	7	8	19	#5006
cee-rhel7	Trilinos-atdm-cee-rhel7_­intel-19.0.3_­mpich2-3.2_­openmp_­static_­opt	Ifpack2_­MTSGS_­belos_­MPI_­1	Passed	Completed	4	2	28	#5006

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

[grover-trilinos]

Test results for issue #5006 as of 2021-05-09

Tests with issue trackers Failed: twif=2

Detailed test results: (click to expand)

Tests with issue trackers Failed: twif=2

Site	Build Name	Test Name	Status	Details	Consec­utive Non-pass Days	Non-pass Last 30 Days	Pass Last 30 Days	Issue Tracker
mutrino	Trilinos-atdm-ats1-knl_­intel-19.0.4_­mpich-7.7.15_­openmp_­static_­opt	Ifpack2_­MTSGS_­belos_­MPI_­1	Failed	Completed (Failed)	4	9	18	#5006
cee-rhel7	Trilinos-atdm-cee-rhel7_­intel-19.0.3_­mpich2-3.2_­openmp_­static_­opt	Ifpack2_­MTSGS_­belos_­MPI_­1	Failed	Completed (Failed)	1	2	28	#5006

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

[grover-trilinos]

Test results for issue #5006 as of 2021-05-18

Tests with issue trackers Passed: twip=1

Detailed test results: (click to expand)

Tests with issue trackers Passed: twip=1

Site	Build Name	Test Name	Status	Details	Consec­utive Pass Days	Non-pass Last 30 Days	Pass Last 30 Days	Issue Tracker
cee-rhel7	Trilinos-atdm-cee-rhel7_­intel-19.0.3_­mpich2-3.2_­openmp_­static_­opt	Ifpack2_­MTSGS_­belos_­MPI_­1	Passed	Completed	9	2	28	#5006

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

[grover-trilinos]

Test results for issue #5006 as of 2021-05-23

Tests with issue trackers Passed: twip=1

Detailed test results: (click to expand)

Tests with issue trackers Passed: twip=1

Site	Build Name	Test Name	Status	Details	Consec­utive Pass Days	Non-pass Last 30 Days	Pass Last 30 Days	Issue Tracker
cee-rhel7	Trilinos-atdm-cee-rhel7_­intel-19.0.3_­mpich2-3.2_­openmp_­static_­opt	Ifpack2_­MTSGS_­belos_­MPI_­1	Passed	Completed	14	2	28	#5006

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

[grover-trilinos]

Test results for issue #5006 as of 2021-05-30

Tests with issue trackers Passed: twip=1

Detailed test results: (click to expand)

Tests with issue trackers Passed: twip=1

Site	Build Name	Test Name	Status	Details	Consec­utive Pass Days	Non-pass Last 30 Days	Pass Last 30 Days	Issue Tracker
cee-rhel7	Trilinos-atdm-cee-rhel7_­intel-19.0.3_­mpich2-3.2_­openmp_­static_­opt	Ifpack2_­MTSGS_­belos_­MPI_­1	Passed	Completed	21	1	29	#5006

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

[grover-trilinos]

Test results for issue #5006 as of 2021-06-06

Tests with issue trackers Passed: twip=1

Detailed test results: (click to expand)

Tests with issue trackers Passed: twip=1

Site	Build Name	Test Name	Status	Details	Consec­utive Pass Days	Non-pass Last 30 Days	Pass Last 30 Days	Issue Tracker
cee-rhel7	Trilinos-atdm-cee-rhel7_­intel-19.0.3_­mpich2-3.2_­openmp_­static_­opt	Ifpack2_­MTSGS_­belos_­MPI_­1	Passed	Completed	1	3	27	#5006

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

[grover-trilinos]

Test results for issue #5006 as of 2021-06-20

Tests with issue trackers Passed: twip=1

Detailed test results: (click to expand)

Tests with issue trackers Passed: twip=1

Site	Build Name	Test Name	Status	Details	Consec­utive Pass Days	Non-pass Last 30 Days	Pass Last 30 Days	Issue Tracker
cee-rhel7	Trilinos-atdm-cee-rhel7_­intel-19.0.3_­mpich2-3.2_­openmp_­static_­opt	Ifpack2_­MTSGS_­belos_­MPI_­1	Passed	Completed	9	3	23	#5006

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

[grover-trilinos]

Test results for issue #5006 as of 2021-06-27

Tests with issue trackers Passed: twip=1

Detailed test results: (click to expand)

Tests with issue trackers Passed: twip=1

Site	Build Name	Test Name	Status	Details	Consec­utive Pass Days	Non-pass Last 30 Days	Pass Last 30 Days	Issue Tracker
cee-rhel7	Trilinos-atdm-cee-rhel7_­intel-19.0.3_­mpich2-3.2_­openmp_­static_­opt	Ifpack2_­MTSGS_­belos_­MPI_­1	Passed	Completed	16	3	23	#5006

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

[grover-trilinos]

Test results for issue #5006 as of 2021-07-04

Tests with issue trackers Passed: twip=1

Detailed test results: (click to expand)

Tests with issue trackers Passed: twip=1

Site	Build Name	Test Name	Status	Details	Consec­utive Pass Days	Non-pass Last 30 Days	Pass Last 30 Days	Issue Tracker
cee-rhel7	Trilinos-atdm-cee-rhel7_­intel-19.0.3_­mpich2-3.2_­openmp_­static_­opt	Ifpack2_­MTSGS_­belos_­MPI_­1	Passed	Completed	23	2	24	#5006

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

[grover-trilinos]

Test results for issue #5006 as of 2021-07-11

Tests with issue trackers Passed: twip=1

Detailed test results: (click to expand)

Tests with issue trackers Passed: twip=1

Site	Build Name	Test Name	Status	Details	Consec­utive Pass Days	Non-pass Last 30 Days	Pass Last 30 Days	Issue Tracker
cee-rhel7	Trilinos-atdm-cee-rhel7_­intel-19.0.3_­mpich2-3.2_­openmp_­static_­opt	Ifpack2_­MTSGS_­belos_­MPI_­1	Passed	Completed	26	0	26	#5006

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

[grover-trilinos]

Test results for issue #5006 as of 2021-07-18

Tests with issue trackers Passed: twip=1

Detailed test results: (click to expand)

Tests with issue trackers Passed: twip=1

Site	Build Name	Test Name	Status	Details	Consec­utive Pass Days	Non-pass Last 30 Days	Pass Last 30 Days	Issue Tracker
cee-rhel7	Trilinos-atdm-cee-rhel7_­intel-19.0.3_­mpich2-3.2_­openmp_­static_­opt	Ifpack2_­MTSGS_­belos_­MPI_­1	Passed	Completed	1	1	29	#5006

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

[grover-trilinos]

Test results for issue #5006 as of 2021-07-25

Tests with issue trackers Failed: twif=2

Detailed test results: (click to expand)

Tests with issue trackers Failed: twif=2

Site	Build Name	Test Name	Status	Details	Consec­utive Non-pass Days	Non-pass Last 30 Days	Pass Last 30 Days	Issue Tracker
mutrino	Trilinos-atdm-ats1-knl_­intel-19.0.4_­mpich-7.7.15_­openmp_­static_­opt	Ifpack2_­MTSGS_­belos_­MPI_­1	Failed	Completed (Failed)	1	3	1	#5006
cee-rhel7	Trilinos-atdm-cee-rhel7_­intel-19.0.3_­mpich2-3.2_­openmp_­static_­opt	Ifpack2_­MTSGS_­belos_­MPI_­1	Failed	Completed (Failed)	1	4	26	#5006

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

[grover-trilinos]

Test results for issue #5006 as of 2021-08-01

Tests with issue trackers Passed: twip=2

Detailed test results: (click to expand)

Tests with issue trackers Passed: twip=2

Site	Build Name	Test Name	Status	Details	Consec­utive Pass Days	Non-pass Last 30 Days	Pass Last 30 Days	Issue Tracker
mutrino	Trilinos-atdm-ats1-knl_­intel-19.0.4_­mpich-7.7.15_­openmp_­static_­opt	Ifpack2_­MTSGS_­belos_­MPI_­1	Passed	Completed	3	5	6	#5006
cee-rhel7	Trilinos-atdm-cee-rhel7_­intel-19.0.3_­mpich2-3.2_­openmp_­static_­opt	Ifpack2_­MTSGS_­belos_­MPI_­1	Passed	Completed	1	5	25	#5006

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

[grover-trilinos]

Test results for issue #5006 as of 2021-08-08

Tests with issue trackers Passed: twip=1
Tests with issue trackers Failed: twif=1

Detailed test results: (click to expand)

Tests with issue trackers Passed: twip=1

Site	Build Name	Test Name	Status	Details	Consec­utive Pass Days	Non-pass Last 30 Days	Pass Last 30 Days	Issue Tracker
mutrino	Trilinos-atdm-ats1-knl_­intel-19.0.4_­mpich-7.7.15_­openmp_­static_­opt	Ifpack2_­MTSGS_­belos_­MPI_­1	Passed	Completed	1	9	9	#5006

Tests with issue trackers Failed: twif=1

Site	Build Name	Test Name	Status	Details	Consec­utive Non-pass Days	Non-pass Last 30 Days	Pass Last 30 Days	Issue Tracker
cee-rhel7	Trilinos-atdm-cee-rhel7_­intel-19.0.3_­mpich2-3.2_­openmp_­static_­opt	Ifpack2_­MTSGS_­belos_­MPI_­1	Failed	Completed (Failed)	1	7	23	#5006

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

[grover-trilinos]

Test results for issue #5006 as of 2021-08-08

Tests with issue trackers Passed: twip=1
Tests with issue trackers Failed: twif=1

Detailed test results: (click to expand)

Tests with issue trackers Passed: twip=1

Site	Build Name	Test Name	Status	Details	Consec­utive Pass Days	Non-pass Last 30 Days	Pass Last 30 Days	Issue Tracker
mutrino	Trilinos-atdm-ats1-knl_­intel-19.0.4_­mpich-7.7.15_­openmp_­static_­opt	Ifpack2_­MTSGS_­belos_­MPI_­1	Passed	Completed	1	9	9	#5006

Tests with issue trackers Failed: twif=1

Site	Build Name	Test Name	Status	Details	Consec­utive Non-pass Days	Non-pass Last 30 Days	Pass Last 30 Days	Issue Tracker
cee-rhel7	Trilinos-atdm-cee-rhel7_­intel-19.0.3_­mpich2-3.2_­openmp_­static_­opt	Ifpack2_­MTSGS_­belos_­MPI_­1	Failed	Completed (Failed)	1	7	23	#5006

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

[grover-trilinos]

Test results for issue #5006 as of 2021-08-08

Tests with issue trackers Passed: twip=1
Tests with issue trackers Failed: twif=1

Detailed test results: (click to expand)

Tests with issue trackers Passed: twip=1

Site	Build Name	Test Name	Status	Details	Consec­utive Pass Days	Non-pass Last 30 Days	Pass Last 30 Days	Issue Tracker
mutrino	Trilinos-atdm-ats1-knl_­intel-19.0.4_­mpich-7.7.15_­openmp_­static_­opt	Ifpack2_­MTSGS_­belos_­MPI_­1	Passed	Completed	1	9	9	#5006

Tests with issue trackers Failed: twif=1

Site	Build Name	Test Name	Status	Details	Consec­utive Non-pass Days	Non-pass Last 30 Days	Pass Last 30 Days	Issue Tracker
cee-rhel7	Trilinos-atdm-cee-rhel7_­intel-19.0.3_­mpich2-3.2_­openmp_­static_­opt	Ifpack2_­MTSGS_­belos_­MPI_­1	Failed	Completed (Failed)	1	7	23	#5006

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

[grover-trilinos]

Test results for issue #5006 as of 2021-08-29

Tests with issue trackers Passed: twip=2

Detailed test results: (click to expand)

Tests with issue trackers Passed: twip=2

Site	Build Name	Test Name	Status	Details	Consec­utive Pass Days	Non-pass Last 30 Days	Pass Last 30 Days	Issue Tracker
mutrino	Trilinos-atdm-ats1-knl_­intel-19.0.4_­mpich-7.7.15_­openmp_­static_­opt	Ifpack2_­MTSGS_­belos_­MPI_­1	Passed	Completed	1	18	12	#5006
cee-rhel7	Trilinos-atdm-cee-rhel7_­intel-19.0.3_­mpich2-3.2_­openmp_­static_­opt	Ifpack2_­MTSGS_­belos_­MPI_­1	Passed	Completed	1	10	20	#5006

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

[grover-trilinos]

Test results for issue #5006 as of 2021-09-05

Tests with issue trackers Passed: twip=1
Tests with issue trackers Failed: twif=1

Detailed test results: (click to expand)

Tests with issue trackers Passed: twip=1

Site	Build Name	Test Name	Status	Details	Consec­utive Pass Days	Non-pass Last 30 Days	Pass Last 30 Days	Issue Tracker
cee-rhel7	Trilinos-atdm-cee-rhel7_­intel-19.0.3_­mpich2-3.2_­openmp_­static_­opt	Ifpack2_­MTSGS_­belos_­MPI_­1	Passed	Completed	1	9	20	#5006

Tests with issue trackers Failed: twif=1

Site	Build Name	Test Name	Status	Details	Consec­utive Non-pass Days	Non-pass Last 30 Days	Pass Last 30 Days	Issue Tracker
mutrino	Trilinos-atdm-ats1-knl_­intel-19.0.4_­mpich-7.7.15_­openmp_­static_­opt	Ifpack2_­MTSGS_­belos_­MPI_­1	Failed	Completed (Failed)	3	20	10	#5006

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

[grover-trilinos]

Test results for issue #5006 as of 2021-09-12

Tests with issue trackers Failed: twif=2

Detailed test results: (click to expand)

Tests with issue trackers Failed: twif=2

Site	Build Name	Test Name	Status	Details	Consec­utive Non-pass Days	Non-pass Last 30 Days	Pass Last 30 Days	Issue Tracker
mutrino	Trilinos-atdm-ats1-knl_­intel-19.0.4_­mpich-7.7.15_­openmp_­static_­opt	Ifpack2_­MTSGS_­belos_­MPI_­1	Failed	Completed (Failed)	1	18	12	#5006
cee-rhel7	Trilinos-atdm-cee-rhel7_­intel-19.0.3_­mpich2-3.2_­openmp_­static_­opt	Ifpack2_­MTSGS_­belos_­MPI_­1	Failed	Completed (Failed)	1	8	21	#5006

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

[grover-trilinos]

Test results for issue #5006 as of 2021-10-03

Tests with issue trackers Passed: twip=1

Detailed test results: (click to expand)

Tests with issue trackers Passed: twip=1

Site	Build Name	Test Name	Status	Details	Consec­utive Pass Days	Non-pass Last 30 Days	Pass Last 30 Days	Issue Tracker
cee-rhel7	Trilinos-atdm-cee-rhel7_­intel-19.0.3_­mpich2-3.2_­openmp_­static_­opt	Ifpack2_­MTSGS_­belos_­MPI_­1	Passed	Completed	6	7	23	#5006

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

[grover-trilinos]

Test results for issue #5006 as of 2021-10-03

Tests with issue trackers Passed: twip=1

Detailed test results: (click to expand)

Tests with issue trackers Passed: twip=1

Site	Build Name	Test Name	Status	Details	Consec­utive Pass Days	Non-pass Last 30 Days	Pass Last 30 Days	Issue Tracker
cee-rhel7	Trilinos-atdm-cee-rhel7_­intel-19.0.3_­mpich2-3.2_­openmp_­static_­opt	Ifpack2_­MTSGS_­belos_­MPI_­1	Passed	Completed	6	7	23	#5006

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

[github-actions]

This issue has had no activity for 365 days and is marked for closure. It will be closed after an additional 30 days of inactivity.
If you would like to keep this issue open please add a comment and/or remove the MARKED_FOR_CLOSURE label.
If this issue should be kept open even with no activity beyond the time limits you can add the label DO_NOT_AUTOCLOSE.
If it is ok for this issue to be closed, feel free to go ahead and close it. Please do not add any comments or change any labels or otherwise touch this issue unless your intention is to reset the inactivity counter for an additional year.

[github-actions]

This issue was closed due to inactivity for 395 days.