To deal with system issues, automatically retry tests

[billsacks]

This is a low-priority enhancement request; I just want to record some thoughts on it for the future.

When running a test suite, we often run into problems where a small number of tests failed due to system issues. It slows down the testing/tagging process to have to go back and manually rerun these tests. It would be nice if cime's testing facilities had an option to automatically retry tests N times before giving up.

@ekluzek raised this at a CSEG meeting June 12, 2017. @jedwards4b felt this could cause too much confusion. But I was just thinking: A way around the confusion problem would be for the test system to create a totally new version of the test, with a new test id (say, by appending "try2" to the testid) (I realize that may not be trivial to implement, though). @mnlevy1981 suggests having a flag to --create-test (like --num-tries – which is my name, not his). This would default to 1, but you could set it to something like 3 if you want

[gold2718]

It seems to me that this would only be useful (in speeding the process) if you always used an N > 1. My experience is that these failures happen a minority of the time. I would like a button to push when I look at failures and suspect a system problem. How about something like the --use-existing switch but called --retry which would create new version(s) of the failed test(s).

[jedwards4b]

A switch to create_test that would refer to an existing test set and only run new tests that had a failed status in the existing test suite seems like a worthy option.

[ekluzek]

@gold2718 our experience with using the aux_clm test list is that these failures happen all the time. I have to rerun tests either on cheyenne or yellowstone (see #1793) whenever I run the test list.

[billsacks]

It seems to me that this would only be useful (in speeding the process) if you always used an N > 1

Personally, I would always do this. I was just recording the earlier conversation, where it sounded like some people wouldn't want it.

I would like a button to push when I look at failures and suspect a system problem. How about something like the --use-existing switch but called --retry which would create new version(s) of the failed test(s).

This would be a good incremental step towards what I want. Long-term, I still think it would be good to have an option to do this automatically. The problem is this: I often kick off CLM test suites overnight with the intention of making a CLM tag first thing in the morning. (The test suites currently take about 4-6 hours to turn around.) When there are system issues like this, it has these consequences:

1. I spend more of my time checking on test results and running tests. While this doesn't take too long, it creates more context switches in my day, which causes productivity to suffer.

2. It lengthens the time from start of testing until when a CLM tag can be made. This is especially problematic when there are a few CLM tags queued up.

That said, I'd be happy with your solution as an incremental step, since it would presumably be easier to implement, and would be a good stepping stone to this longer-term solution.