Create automated tool to update ATDM Trilinos GitHub issues with current status of tests attached to those issues

[bartlettroscoe]

CC: @fryeguy52, @trilinos/framework

Description

As part of the work to implement a Python tool to pull down, analyze and the summarize CDash build and test data in #2933, we realized that we had all of the information that would be needed to update Trilinos GitHub issues about the status of tests associated with Trilinos GitHub issues. The idea would be to add a GitHub issue comment containing tables showing the current status of the tests related to a GitHub Issue. An example of what this could look like is shown in #3579 (comment) and #3833 (comment) where I just manually copied and pasted the HTML-formatted tables and rows for those issues right into the the GitHub comments.

For #3579 on 11/14/2018, the results for 11/13/2018 might look like:

---

Test results for #3579 as of testing day 2018-11-13

Tests with issue trackers Passed: twip=2
Tests with issue trackers Missing: twim=3

NOTE: With all of the tests associated with this issue passing or missing (i.e. disabled), might this Issue be addressed and perhaps be closed?

Detailed test results: (click to expand)

Tests with issue trackers Passed: twip=2

Site	Build Name	Test Name	Status	Details	Consec­utive Days Pass	Nopass last 30 Days	Pass last 30 Days	Tracker
hansen	Trilinos-atdm-hansen-shiller-cuda-8.0-opt	PanzerAdaptersSTK_­CurlLaplacianExample-ConvTest-Quad-Order-4	Passed	Completed	4	34	4	#3579
white	Trilinos-atdm-white-ride-cuda-9.2-opt	PanzerAdaptersSTK_­CurlLaplacianExample-ConvTest-Quad-Order-4	Passed	Completed	2	26	3	#3579

Tests with issue trackers Missing: twim=3

Site	Build Name	Test Name	Status	Details	Consec­utive Days Missing	Nopass last 30 Days	Pass last 30 Days	Tracker
hansen	Trilinos-atdm-hansen-shiller-cuda-8.0-debug	PanzerAdaptersSTK_­CurlLaplacianExample-ConvTest-Quad-Order-4	Missing	Missing	4	26	0	#3579
hansen	Trilinos-atdm-hansen-shiller-cuda-9.0-debug	PanzerAdaptersSTK_­CurlLaplacianExample-ConvTest-Quad-Order-4	Missing	Missing	4	25	0	#3579
white	Trilinos-atdm-white-ride-cuda-9.2-debug	PanzerAdaptersSTK_­CurlLaplacianExample-ConvTest-Quad-Order-4	Missing	Missing	4	25	0	#3579

---

So the idea is is that once that comment was added, the Trilinos developer responsible for the github issue could add a comment stating that this was not a randomly failing test so having this test pass or be disabled indicated that the issue could be resolved and then close the issue. No need to look at CDash directly, add in new CDash links, etc. Just comment and close. That would save a lot of human time.

So we would add these comments in the following cases:

• Once a week just as a reminder of the current status of the tests related to this issue (so that Trilinos developers would not forget about the issue).

• When one of the associated tests changed status (e..g went from passing to failing or failings to passing). (But not for frequent randomly failing tests or that would create a lot of spam updates).

• When all of associated tests were were all passing or missing for X (e.g. 2) consecutive days, like shown above. (But not for any randomly failing tests or that would create a lot of spam.)

• When all of the associated tests are all passing for the full acquired test history (e.g. 30 days) or are missing for frequent randomly failing tests. (But not for rare randomly failing tests.)

This should make it so that people don't need to manually check on the status of the associated tests for a GitHub issue. They could just let an automated system (that will be created in this Story) update the GitHub issue when something worth noting has occurred and when the issue might be closed.

Also, this tool could check that if all of the tests are passing and the GitHub issue was closed, then it could automatically remove the entries for that issue from a list of *.csv files. And, if it detects that tests are not all passing and the GitHub issue is closed, then it could automatically re-open the issue and provide a summary of the test results.

Related

• https://sems-atlassian-son.sandia.gov/jira/browse/SEDD-58: @rmmilewi is using this JIRA issue to drive this?

Tasks

1. Update the cdash_analyze_and_report.py tool to write detailed test information and history for all tests with issue trackers being monitored by the tool. [Done]
2. Experiment with how to robustly add comments to a GitHub issue. (e.g. look into PyGitHub?) [Done]
3. Write beginning Python code with unit tests and a Python tool called something like update_github_issues_for_tests_status.py that will update GitHub issues as described above given the output from the cdash_analyze_and_report.py tool described above ... See grover_update_trilinos_github_issues_with_test_status.sh [Done]
4. Provide first implementation that just posts the list of passing, missing and failing tests for each open Trilinos GitHub issue once a week (Minimum Viable Product) ... See below [Done]
5. Extend cdash_analyze_and_report.py to provide the fail_frequency field and put in output data-structure for each issue.
6. Extend CDashQueryAnalyzeReport.py to provide pass/fail criteria for each issue and provide suggestions for when to close an issue based on if tests passing for X days matching fail_frequency criteria. (needs more analysis)
7. Write a tool to remove entries from the *.csv file for Issues that are closed and have met the passing criteria based on fail_frequency logic (see above).
8. Extend Grover to write comments to Trilinos GitHub issues that are closed but fail the passing criteria based on 'fail_frequency'.

[bartlettroscoe]

@fryeguy52, as we discussed yesterday, this would be a fun thing to start working and it would have a large impact in making sure that people follow up on ATDM Trilinos github issues and require less of our own personal time to do so.

[bartlettroscoe]

@fryeguy52, other just just PyGitHub, some other things to consider include:

• https://stackoverflow.com/questions/10625190/most-suitable-python-library-for-github-api-v3
• https://github3py.readthedocs.io/en/master/

[bartlettroscoe]

@fryeguy52,

FYI: based on our conversation today, I just added the scope:

Also, this tool could check that if all of the tests are passing and the GitHub issue was closed, then it could automatically remove the entries for that issue from a list of *.csv files. And, if it detects that tests are not all passing and the GitHub issue is closed, then it could automatically re-open the issue and provide a summary of the test results.

That would make it so that we would never need to manually re-open github issues that should not have been closed and it would make it so that we would not have to manually remove entires for the *.csv files for closed issues.

[bartlettroscoe]

FYI: Joe Frye mentioned that Aaron L. has some code that pulls down GitHub Issue data and sends out summary emails. We need to ask Aaron to look at his scripts once we can get down to this.

[bartlettroscoe]

CC: @rmmilewi

[bartlettroscoe]

@rmmilewi, take a look at:

• https://snl-wiki.sandia.gov/display/CoodinatedDevOpsATDM/ATDM+Builds+of+Trilinos

to understand the entire ATDM Trilinos build process and where this fits in and especially:

• https://snl-wiki.sandia.gov/display/CoodinatedDevOpsATDM/Triaging+and+Addressing+ATDM+Trilinos+Failures

[bartlettroscoe]

@rmmilewi, as discussed at our meeting just now, I will add the option --write-test-data-to-file=<file> to the cdash_analyze_and_report.py tool and I will add a REAMD.md file for how to run the tool on the ATDM Trilinos builds to generate the data your tool/bot will work with.

[bartlettroscoe]

@rmmilewi, I added the cdash_analyze_and_report.py option --write-test-data-to-file=<file> and updated the TrilinosATDMStatus scripts to call that option. If you follow the instructions at:

• https://gitlab-ex.sandia.gov/atdm/TrilinosATDMStatus/-/blob/master/README.md

you can get setup locally to run the driver script as:

$ ../TrilinosATDMStatus/trilinos_atdm_builds_status_driver.sh --date=yesterday \
  --email-from-address= --send-email-to=${USER}@sandia.gov

That will produce the files:

$ ls -w 1 *TestData.py
promotedAtdmTrilinosTestData.py
specializedAtdmTrilinosCleanupTestData.py

You read these in Python as:

>>> testData = []
>>> testData.extend(eval(open('promotedAtdmTrilinosTestData.py', 'r').read()))
>>> testData.extend(eval(open('specializedAtdmTrilinosCleanupTestData.py', 'r').read()))
>>> print(len(testData))
167

You want to group these by the 'issue_tracker_url' field the test dicts and then you want to create an HTML string table using the function createCDashTestHtmlTableStr() shown at:

• https://github.com/TriBITSPub/TriBITS/blob/master/tribits/ci_support/CDashQueryAnalyzeReport.py#L2098

or perhaps the function createHtmlTableStr() shown at:

• https://github.com/TriBITSPub/TriBITS/blob/master/tribits/ci_support/CDashQueryAnalyzeReport.py#L1920

Let me know if you have questions. Otherwise, I can mock up how this might look. It is not much code to create the markdown and HTML that would go into the GitHub Issue comment.

[bartlettroscoe]

@rmmilewil, I think I want to change these file formats over to proper JSON. When I do that at some point, I will let you know. It should be pretty easy to adjust.

[rmmilewi]

@bartlettroscoe So, I'm sorting out the design and assessing the requirements based on the notes I took, the audio recording of our session, and what's described here on this issue. Could you provide me with a example copy of the data that you want the tool to read in? One that you generated, looked over, and confirmed was valid and as-expected? I know I can fetch this data myself, but I want another human in the loop on this. 😄

[bartlettroscoe]

I know I can fetch this data myself, but I want another human in the loop on this.

@rmmilewi, I will be in the loop as much as needed but it should be trivial to generate the data yourself as provided in the instructions at:

• https://gitlab-ex.sandia.gov/atdm/TrilinosATDMStatus/-/blob/master/README.md

However, here is the file promotedAtdmTrilinosTestData.py generated from the command:

$ ../TrilinosATDMStatus/trilinos_promoted_atdm_builds_status.sh --date=2020-03-06

That is only partial data from today but it gives tests in all of the different categories 'twiof', 'twip', 'twim' and 'twif' in the associated summary HTML email text
promotedAtdmTrilinosBuilds.html.

Please make sure that you can run that driver on your local machine and generate this same data as well. Again, it should be trivial if you have Python 2.7+.

Just download those files, remove the .txt extension and you should be able to use them right away.

Have a look at that data and let me me know if you have any questions about it. See the basic instructions above.

[rmmilewi]

@bartlettroscoe It's trivial, I know, and I'll be generating plenty of data myself, but it's an important step in the process of stakeholder engagement for me that I have some agreed-upon, canonical data that captures the phenomena of interest. Same goes for any other data you think I may need (like that CSV file you showed me). In this case, either I can generate it myself and send it to you, or we can use the data you just generated, so long as I get a confirmation from you.

Speaking of requirements, does cdash_analyze_and_report.py require Python 2? I just wanted to make sure.

[bartlettroscoe]

Same goes for any other data you think I may need (like that CSV file you showed me).

@rmmilewi, with my current thinking the the tool and process being written in this issue should never need to look at those CVS files. All of the data needed should be present in the Python dicts for each of these tests. If there is missing data, then we need to add it.

In this case, either I can generate it myself and send it to you, or we can use the data you just generated, so long as I get a confirmation from you.

The best way to do this is so "Hey Ross, what testing day looks to be a good day for running my tool?". That is is just a string YYYY-MM-DD passed in the --date field.

Speaking of requirements, does cdash_analyze_and_report.py require Python 2? I just wanted to make sure.

It is only currently being tested with Python 2. Making sure it works with Python 3 is not a lot of work. There is an overlap of Python 2 and 3 that allows a Python program to work with both. (I need to figure out how to add Python 3 testing to the Travis CI testing for TriBITS.)

[rmmilewi]

With my current thinking the the tool and process being written in this issue should never need to look at those CVS files. All of the data needed should be present in the Python dicts for each of these tests. If there is missing data, then we need to add it.

@bartlettroscoe Ah, gotcha! Good to know!

The best way to do this is so "Hey Ross, what testing day looks to be a good day for running my tool?". That is is just a string YYYY-MM-DD passed in the --date field.

Not a bad idea. It might be worthwhile to keep me in the loop for any noteworthy test/bug-related events. It would be interesting to build up a dataset of that sort.

It is only currently being tested with Python 2. Making sure it works with Python 3 is not a lot of work. There is an overlap of Python 2 and 3 that allows a Python program to work with both. (I need to figure out how to add Python 3 testing to the Travis CI testing for TriBITS.)

Cool! From the looks of it, I can interact with your tools through your scripts, so I'm not concerned about any compatibility issues between your code and mine. For my part in this I'm leaning towards making sure my code is Python-3-compatible. It's possible that elements of this codebase could be reused in the future, and seeing as Python 2 has reached end of life, I expect Python 3 to be the norm for future development. Miranda and I reached the same conclusion about the Traffic code we've been developing. Of course, I'm open to any ideas or preferences you might have.

[bartlettroscoe]

I can interact with your tools through your scripts, so I'm not concerned about any compatibility issues between your code and mine.

@rmmilewi, not entirely. The code the generates the HTML tables needs to be reused. Please don't write that stuff again. That will just make more code to maintain.

It's possible that elements of this codebase could be reused in the future, and seeing as Python 2 has reached end of life, I expect Python 3 to be the norm for future development.

The reality is that as long as Python 2.7 is the default Python on SNL CEO RHEL7 systems, our Python tools will need to support Pythong 2.7 for many years to come.

[bartlettroscoe]

CC: @rmmilewi

I just merged the TriBITS PR TriBITSPub/TriBITS#322 to the TriBITS 'master' branch that created a new class IssueTrackerTestsStatusReporter. I plugged this into the 'grover' code in the MR:

• https://gitlab-ex.sandia.gov/rmilewi/grover/-/merge_requests/2

I manually posted what the content would look like as posted by 'grover' to a few Trilinos GitHub Issues:

• MueLu_ConvergenceTpetra_MPI tests failing in build Trilinos-atdm-mutrino-intel-opt-openmp-KNL starting before 2018-11-13 #3862 (comment)
• Test ROL_NonlinearProblemTest_MPI_4 timing out in ATDM waterman CUDA 9.2 builds starting 2019-11-14 #6329 (comment)

There are a few issues that need to be fixed in 'grover' before we can deploy the first version. This is outlined in:

• https://gitlab-ex.sandia.gov/rmilewi/grover/-/issues/3

The only one that needs to get fixed before we can deploy and initial version of this tool is to refactor the code to not copy the test dicts into a different data-structure and then copy them back. The data was being copied back incorrectly in some cases and it does not support the addition of new fields. Once that is fixed, I think we can deploy the first version that just updates the open issues once a week.

[bartlettroscoe]

CC: @rmmilewi

A problem with this approach of just embedding the tables in the GitHub Issue comments as you can see in:

• MueLu_ConvergenceTpetra_MPI tests failing in build Trilinos-atdm-mutrino-intel-opt-openmp-KNL starting before 2018-11-13 #3862 (comment)
• Test ROL_NonlinearProblemTest_MPI_4 timing out in ATDM waterman CUDA 9.2 builds starting 2019-11-14 #6329 (comment)

is that the test summary tables are hard to read. First, GitHub Issue comments are very narrow and you can't see all of the columns of the table and makes for very tall rows. Second, the colors are not shown which makes the tables harder to read.

Therefore, it would be good to find an easy way for developers to view these tables in their full HTML spender. One approach might be to attach the HTML as a file to the GitHub Issue comment. I showed how this might look in:

• MueLu_ConvergenceTpetra_MPI tests failing in build Trilinos-atdm-mutrino-intel-opt-openmp-KNL starting before 2018-11-13 #3862 (comment)

However, you can't directly attach and *.html file and instead have to attach an *.html.txt file. Then the developer needs to download that file, change the extension from *.html.txt to *.html and then open that file in a browser. So this is not very nice and I don't think many developers will bother to do that.

Another approach would be to post the HTML to website and then view it from there. We could do that with a GitHub pages site maintained with with a GitHub repo, for example. Or, we could just post it to the trilinos.org site. Just to show what that might look like, I manually posted:

• https://tribits.org/trilinos_issue_test_status/issues/3862/2020-06-05.html

and put in a link to this from:

• MueLu_ConvergenceTpetra_MPI tests failing in build Trilinos-atdm-mutrino-intel-opt-openmp-KNL starting before 2018-11-13 #3862 (comment)

That looks much better and is much more readable. I think we need to do something like this to make this information more readable and useful for developers.

[bartlettroscoe]

Below I log some details of how I tested this with 'grover' and the subtasks that I did in the last week.

Details: (click to expand)

(6/6/2020)

I have this working with grover for the branch in MR:

• https://gitlab-ex.sandia.gov/rmilewi/grover/-/merge_requests/2

and the TriBITS PR:

• Add IssueTrackerTestsStatusReporter and related refactorings (trilinos./Trilinos#3887) TriBITSPub/TriBITS#322

I tested this on 'crf450' with:

$ cd /home/rabartl/Trilinos.base/grover/

$ cat load-env.sh
module load sems-env
module load sems-python/2.7.9
export https_proxy=https://wwwproxy.sandia.gov:80
export TRILINOS_ATDM_STATUS_DIR=$HOME/Trilinos.base/TrilinosATDMStatus
export GROVER_TRIBITS_DIR=$HOME/Trilinos.base/TrilinosATDMStatus/TriBITS
export PYTHONPATH=$PYTHONPATH:${GROVER_TRIBITS_DIR}/tribits/ci_support

$ load-env.sh

$ cat runMockGrover.sh
#!/bin/bash

python -m grover.main.main updateissues &> grover.out

grep "\(<body>\|Test results for issue\|Tests with issue trackers\)" grover.out \
  | grep -v "<h3>.*Tests with issue trackers" \
  | grep -v "^Tests" &> grover.summary.out

$ ./runMockGrover.sh

That provided the summary output file grover.summary.out:

<body>
<h2>Test results for issue #5892 as of YYYY-MM-DD (ToDo: fix grover!, see rmilewi/grover#3)</h2>
<font color="green">Tests with issue trackers Passed: twip=1</font><br>
<body>
<h2>Test results for issue #6790 as of YYYY-MM-DD (ToDo: fix grover!, see rmilewi/grover#3)</h2>
<font color="green">Tests with issue trackers Passed: twip=1</font><br>
<body>
<h2>Test results for issue #5002 as of YYYY-MM-DD (ToDo: fix grover!, see rmilewi/grover#3)</h2>
<font color="green">Tests with issue trackers Passed: twip=2</font><br>
<body>
<h2>Test results for issue #6796 as of YYYY-MM-DD (ToDo: fix grover!, see rmilewi/grover#3)</h2>
<font color="green">Tests with issue trackers Passed: twip=5</font><br>
<font color="gray">Tests with issue trackers Missing: twim=3</font><br>
<body>
<h2>Test results for issue #5006 as of YYYY-MM-DD (ToDo: fix grover!, see rmilewi/grover#3)</h2>
<font color="green">Tests with issue trackers Passed: twip=1</font><br>
<body>
<h2>Test results for issue #6799 as of YYYY-MM-DD (ToDo: fix grover!, see rmilewi/grover#3)</h2>
<font color="green">Tests with issue trackers Passed: twip=8</font><br>
<body>
<h2>Test results for issue #6801 as of YYYY-MM-DD (ToDo: fix grover!, see rmilewi/grover#3)</h2>
<font color="green">Tests with issue trackers Passed: twip=3</font><br>
<body>
<h2>Test results for issue #6804 as of YYYY-MM-DD (ToDo: fix grover!, see rmilewi/grover#3)</h2>
<font color="green">Tests with issue trackers Passed: twip=1</font><br>
<body>
<h2>Test results for issue #6805 as of YYYY-MM-DD (ToDo: fix grover!, see rmilewi/grover#3)</h2>
<font color="green">Tests with issue trackers Passed: twip=4</font><br>
<body>
<h2>Test results for issue #3862 as of YYYY-MM-DD (ToDo: fix grover!, see rmilewi/grover#3)</h2>
<font color="red">Tests with issue trackers Failed: twif=2</font><br>
<body>
<h2>Test results for issue #3863 as of YYYY-MM-DD (ToDo: fix grover!, see rmilewi/grover#3)</h2>
<font color="red">Tests with issue trackers Failed: twif=1</font><br>
<body>
<h2>Test results for issue #6553 as of YYYY-MM-DD (ToDo: fix grover!, see rmilewi/grover#3)</h2>
<font color="red">Tests with issue trackers Failed: twif=2</font><br>
<body>
<h2>Test results for issue #6554 as of YYYY-MM-DD (ToDo: fix grover!, see rmilewi/grover#3)</h2>
<font color="green">Tests with issue trackers Passed: twip=1</font><br>
<body>
<h2>Test results for issue #6051 as of YYYY-MM-DD (ToDo: fix grover!, see rmilewi/grover#3)</h2>
<font color="green">Tests with issue trackers Passed: twip=1</font><br>
<body>
<h2>Test results for issue #5545 as of YYYY-MM-DD (ToDo: fix grover!, see rmilewi/grover#3)</h2>
<font color="red">Tests with issue trackers Failed: twif=2</font><br>
<body>
<h2>Test results for issue #7089 as of YYYY-MM-DD (ToDo: fix grover!, see rmilewi/grover#3)</h2>
<font color="green">Tests with issue trackers Passed: twip=8</font><br>
<font color="gray">Tests with issue trackers Missing: twim=2</font><br>
<body>
<h2>Test results for issue #7090 as of YYYY-MM-DD (ToDo: fix grover!, see rmilewi/grover#3)</h2>
<font color="gray">Tests with issue trackers Missing: twim=3</font><br>
<body>
<h2>Test results for issue #6070 as of YYYY-MM-DD (ToDo: fix grover!, see rmilewi/grover#3)</h2>
<font color="green">Tests with issue trackers Passed: twip=1</font><br>
<body>
<h2>Test results for issue #6455 as of YYYY-MM-DD (ToDo: fix grover!, see rmilewi/grover#3)</h2>
<font color="green">Tests with issue trackers Passed: twip=6</font><br>
<body>
<h2>Test results for issue #6329 as of YYYY-MM-DD (ToDo: fix grover!, see rmilewi/grover#3)</h2>
<font color="gray">Tests with issue trackers Missing: twim=6</font><br>
<body>
<h2>Test results for issue #6333 as of YYYY-MM-DD (ToDo: fix grover!, see rmilewi/grover#3)</h2>
<font color="green">Tests with issue trackers Passed: twip=1</font><br>
<body>
<h2>Test results for issue #5310 as of YYYY-MM-DD (ToDo: fix grover!, see rmilewi/grover#3)</h2>
<font color="green">Tests with issue trackers Passed: twip=2</font><br>
<body>
<h2>Test results for issue #6216 as of YYYY-MM-DD (ToDo: fix grover!, see rmilewi/grover#3)</h2>
<font color="green">Tests with issue trackers Passed: twip=1</font><br>
<font color="red">Tests with issue trackers Failed: twif=1</font><br>
<body>
<h2>Test results for issue #6991 as of YYYY-MM-DD (ToDo: fix grover!, see rmilewi/grover#3)</h2>
<font color="green">Tests with issue trackers Passed: twip=6</font><br>
<body>
<h2>Test results for issue #7251 as of YYYY-MM-DD (ToDo: fix grover!, see rmilewi/grover#3)</h2>
<font color="green">Tests with issue trackers Passed: twip=2</font><br>
<body>
<h2>Test results for issue #6882 as of YYYY-MM-DD (ToDo: fix grover!, see rmilewi/grover#3)</h2>
<font color="green">Tests with issue trackers Passed: twip=27</font><br>
<font color="gray">Tests with issue trackers Missing: twim=9</font><br>
<body>
<h2>Test results for issue #6246 as of YYYY-MM-DD (ToDo: fix grover!, see rmilewi/grover#3)</h2>
<font color="green">Tests with issue trackers Passed: twip=6</font><br>
<body>
<h2>Test results for issue #6126 as of YYYY-MM-DD (ToDo: fix grover!, see rmilewi/grover#3)</h2>
<font color="red">Tests with issue trackers Failed: twif=1</font><br>
<body>
<h2>Test results for issue #5872 as of YYYY-MM-DD (ToDo: fix grover!, see rmilewi/grover#3)</h2>
<font color="green">Tests with issue trackers Passed: twip=4</font><br>

I manually posted what the content would look like to a few Trilinos GitHub Issues:

• MueLu_ConvergenceTpetra_MPI tests failing in build Trilinos-atdm-mutrino-intel-opt-openmp-KNL starting before 2018-11-13 #3862 (comment)
• Test ROL_NonlinearProblemTest_MPI_4 timing out in ATDM waterman CUDA 9.2 builds starting 2019-11-14 #6329 (comment)

Some stuff in grover needs to be fixed and described in:

• https://gitlab-ex.sandia.gov/rmilewi/grover/-/issues/3

Completed tasks: (click to expand)

• Create TriBITS branch 'tril-3887-test-summary-tables-refactor' [Done]

• Look over grover code some to see how it is calling TriBITS ... See the function TriBITSCommentFormatter() in the file grover/grover/core/interactor.py [Done]

• Look over organization of code to see how to to provide functions to produce test results by table [Done]

• Change name of testSetXXX to testsetXXX [Done]

• Refactor createCDashTestHtmlTableStr() to get 'testHistoryDays' out of the test dict field 'test_history_num_days' [Done]

• Move var 'testSortOrder' from cdash_analyze_and_report.py to a function 'getDefaultTestsSortOrder()' in CDashQueryAnalyzeReport.py and use it as the default argument in class constructor 'TestsetGetDataAnayzeReporter' [Done]

• Factor out function createOverallSummaryLine() that takes in an 'overallVars' object and builds the summary line with 'PASSED|FAILED (....)' [Done]

• Factor out a class 'Testsetinfo' that will hold data {'testsetType', 'testTypeDescr', 'testTypeCountAncryNum', 'colorTestset'} and provide the function 'getStandardTestsetInfo(testsetType)' to return the standard info for the test table types 'twoif', 'twoinr', 'twip', 'twim', 'twif', and 'twinr' and refactor code to use this [Done]

• Create a new class 'AddTestHistoryStrategy' that will take in 'inOptions' and 'testHistoryCacheDir' and provide a function 'getTestHistory()' and refactor the class 'TestsetGetDataAnayzeReporter' to use it [Done]

• Move the class 'TestsetGetDataAnayzeReporter' from cdash_analyze_and_report.py to CDashQueryAnalyzeReport.py rename it 'TestsetReporter' to but leave the class 'AddTestHistoryStrategy' in cdash_analyze_and_report.py [Done]

• Move the class 'OverallVars' from cdash_analyze_and_report.py to CDashQueryAnalyzeReport.py and rename it 'CDashReportData' so it can be reused there [Done]

• Create function binTestDictsByIssueTracker(testsLOD) => (testDictsByIssueTracker, testsWithoutIssueTrackersLOD) that will create a dictionary where the keys are the string name of the issue tracker and the values are the sublists of test dicts for that issue tracker. (Any test dicts without an issue tracker will collected into a reminder list testsWithoutIssueTrackersLOD.) (must write unit tests for this) [Done]

• Create function getTestsetAcroFromTestDict(testDict) => that returns the standard test-set acronym 'twoif', 'twoinr', 'twip', 'twim', 'twif', 'twinr' given the input test dict [Done]

• Clean up some raw html being used in the cdash_analyze_and_report.py [Done]

• Create function binTestDictsByTestsetAcro(testsLOD) => testsByTestsetAcro for the standard test sets 'twoif', 'twoinr', 'twip', 'twim', 'twif', 'twinr' (just use a dict where each key is a stanadrd acronym type) [Done]

• Change name of class TestsetReporter to SingleTestsetReporter [Done]

• Create class TestsetsReporter that takes CDashReportData object and has function report(testsLOD) => (testsStatusHtmlReportStr) that takes in a testLOD and seprates them out into sublists 'twip, 'twim', etc. and adds the HTML text to display them (must write some unit tests for this):

  • Create failing tests for empyt list of test dicts [Done]
  • Get some pre-generated *.py data from cdash_analyze_and_report.py output to set up first test case [Done]
  • Create faling test for just 'twif' and 'twinr' [Done]
  • Test out generated HTML in GitHub issue comment using copy-and-paste [Done]
  • Fix up formatting to look okay on GitHub [Done]
  • Create test for all supported types [Done]

• Create new class IssueTrackerTestsStatusReporter that will use TestsetsReporter to create a custom message and will handle all future logic for informing the developer about what the status of their tsets for their issue:

  • Copy test for TestsetsReporter and use it to create test for IssueTrackerTestsStatusReporter [Done]
  • Remove the top table <style>...</style> block [Done]
  • Add function getIssueTrackerAndAssertAllSame(testsLOD) that will assert that all tests in testsLOD have a non-empty 'issue_tracker' field and that the fields 'issue_tracker' and 'issue_tracker_url' are all identical (add unit tests for this alone and through the class) [Done]
  • Add field 'cdash_testing_day' to test dicts that get written to file and then extract out to provide the testing date in IssueTrackerTestsStatusReporter [Done]

• Update grover/grover/core/interactor.py to use the new class IssueTrackerTestsStatusReporter:

  • Create grover branch 'tril-3887-test-summary-tables-refactor' [Done]
  • Put in new class IssueTrackerTestsStatusReporter [Done]
  • Manually run grover to see the generated output [Done]
  • Run the grover unit tests and update if any fail ... I am not going to fix this. I will let Reed fix these tests [Done]

• Wrap detailed test tables is a "details" HTML block [Done]

• Rename the test 'test_twip_twim' to 'test_bm_1_twoif_12_twip_2_twim_2_twif_5' to avoid confusion [Done]

• Create new test 'twip_2_twim_2' for cdash_analyze_and_report.py to correctly check 'twip' and 'twim' that should not fail the glboal pass:

  • Create test that matches existing behavior that incorrectly says 'FAILED' and returns 1 instead of 0 [Done]
  • Change the test to what it should expect and will fail [Done]
  • Add var 'existanceTriggersGlobalFail' to class TestsetInfo for if existing tests in this category should trigger global fail (only 'twip' and 'twim' tests should not trigger a global failure) [Done]

• Rename 'TestsetInfo' to 'TestsetTypeInfo' and 'testsetInfo' to 'testsetTypeIfo' [Done]

• Remove tests for class TestsetsReporter (and add comment that it gets tested through the IssueTrackerTestStatusReporter class tests [Done]

• Clean up commits on TriBITS branch 'tril-3887-test-summary-tables-refactor' and force push [Done]

• Review changes on GitHub PR [Done]

• Address few issues after my review [Done]

• Run cdash_anayze_and_report.py drivers for real to make sure they still work [Done]

• Merge the branch 'tril-3887-test-summary-tables-refactor' to 'master' with checkin-test.py script [Done]

• Add comment to Trilinos GitHub Create automated tool to update ATDM Trilinos GitHub issues with current status of tests attached to those issues #3887 [Done]

• Add comment to https://gitlab-ex.sandia.gov/rmilewi/grover/-/issues/3 [Done]

• Add todo to Trilinos GitHub Create automated tool to update ATDM Trilinos GitHub issues with current status of tests attached to those issues #3887 to consider publishing the issue-by-issue HTML reports to some website and then put in a link to it from the commit to that page. Or, consider just attaching the formatted file with as issue-XXXX-test-status-YYYY-MM-DD.html.txt [Done]

[bartlettroscoe]

FYI: Meet with @rmmilewi yesterday to go over the current status of Grover towards the minimum viable product. There are the todos:

• Use updated code with correct issue info being submitted (i.e. make sure that changes that I made are being used in all steps below)
• Fix formatting problems with beginning <html> block.
• Update testing mode for Grover to create a new comment for each issue tracker just as it would in production mode.
• Do a manual running of the Grover to submit status (as @rmmilewi's GitHub account) for all of the current tracked tests to their GitHub issues
• Set up a new GitHub entity account that will be run using an internal Jenkins job
• Set up a new jenkins job that will submit to GitHub but initially create a new dummy Trilinos GitHub issue and use --dummyIssueId=<issueid> to submit the status of all of the tracked tests as new comments to the dummy Trilinos GitHub issue. (That will both validate that the GitHub authentication is working correctly and it will show, again, what the comments look like.)

[jhux2]

@bartlettroscoe @rmmilewi Thanks for setting this up -- it looks to be really useful. Just wanted to mention a small typo: the Grover message is missing the word "are" just after "there":

Grover saw that there tests being tracked on CDash that are associated with this open issue.

[bartlettroscoe]

Just wanted to mention a small typo: the Grover message is missing the word "are" just after "there"

@jhux2, thanks for the catch! Should be fixed going forward.

[bartlettroscoe]

We now have finally deployed the first "Minimum Viable Product" that adds a single comment per open ATDM Trilinos GitHub issue as described in ATDV-365. The Jenkins project:

• https://jenkins-son.sandia.gov/search/?q=grover_trilinosatdmstatus_update_trilinos_issues

is set up to run once a week at 7 AM on Mondays to post these comments. I sent out the following email to the Trilinos developers yesterday, and then I manually ran that above Jenkins project which resulting in 11 comments posted and the log output showed:

16:47:35 [main.py] issue #6126,             success=False,            message:The issue (#6126) is not open.                 Right now, Grover avoids publishing results to closed issues.
16:47:35 [main.py] issue #6246,             success=False,            message:The issue (#6246) is not open.                 Right now, Grover avoids publishing results to closed issues.
16:47:35 [main.py] issue #3862,             success=False,            message:The issue (#3862) is not open.                 Right now, Grover avoids publishing results to closed issues.
16:47:35 [main.py] issue #6882,             success=False,            message:The issue (#6882) is not open.                 Right now, Grover avoids publishing results to closed issues.
16:47:35 [main.py] issue #6540,             success=True,            message:None
16:47:35 [main.py] issue #6799,             success=True,            message:None
16:47:35 [main.py] issue #7778,             success=True,            message:None
16:47:35 [main.py] issue #3863,             success=True,            message:None
16:47:35 [main.py] issue #6790,             success=True,            message:None
16:47:35 [main.py] issue #6991,             success=False,            message:The issue (#6991) is not open.                 Right now, Grover avoids publishing results to closed issues.
16:47:35 [main.py] issue #7690,             success=False,            message:The issue (#7690) is not open.                 Right now, Grover avoids publishing results to closed issues.
16:47:35 [main.py] issue #6009,             success=True,            message:None
16:47:35 [main.py] issue #6216,             success=True,            message:None
16:47:35 [main.py] issue #6553,             success=True,            message:None
16:47:35 [main.py] issue #6455,             success=True,            message:None
16:47:35 [main.py] issue #5006,             success=True,            message:None
16:47:35 [main.py] issue #7089,             success=True,            message:None

and posted the comments:

• SEACAS CGNS tests failing in ATDM Trilinos cts1 intel-18.0.2 openmpi-2.0.3 builds starting 2020-01-03 #6540 (comment)
• KokkosCore_UnitTest_Cuda_MPI_1 failing in ATDM Trilinos 'waterman', 'ats2'/'vortex', 'ride', 'sems-rhel7' CUDA 'opt' builds starting 2020-02-04  #6799 (comment)
• Build errors in Sacado_GTestSuite.exe for 'arm-20.1' on 'stria' and 'intel-18.0.0.20170811' on 'tlcc2' starting 2020-08-04 #7778 (comment)
• Stratimikos_test_single_belos_thyra_solver_driver_nos1_nrhs8_MPI_1 failing in build Trilinos-atdm-mutrino-intel-opt-openmp-KNL starting before 2018-11-13 #3863 (comment)
• Test NOX_Tpetra_1DFEM_MPI_4 random failure showing "Concurrent modification of host and device views in DualView" starting 2020-02-09  #6790 (comment)
• Tempus tests timing out in ATDM debug builds starting testing day 2019-09-24 #6009 (comment)
• New test ROL_example_PDE-OPT_helmholtz_example_02_MPI_1 added 2019-10-16 timing out in clang and cuda builds staring before 2019-11-02 #6216 (comment)
• ROL tests failing in new ATDM Trilinos 'cts1' and 'ats2' builds starting 2020-01-03 #6553 (comment)
• Teko_testdriver_tpetra_MPI_1 failing in ATDM Power/GPU CUDA builds on 'ride' and 'waterman' starting 2019-12-13 #6455 (comment)
• Ifpack2_MTSGS_belos_MPI_1 randomly failing in build Trilinos-atdm-cee-rhel6_intel-18.0.2_mpich2-3.2_openmp_static_opt starting before 2019-04-24 #5006 (comment)
• SEACASIoss_pamgen_to_unstructured_cgns faining with diffs in all CUDA, clang, gnu, and intel ATDM Trilinos builds where it is run starting 2020-03-27 #7089 (comment)

The next step is to update the comments so they give suggestions on if the issue can be closed since the issues are addressed.

[bartlettroscoe]

Related to:

• SEPW-215

[github-actions]

This issue has had no activity for 365 days and is marked for closure. It will be closed after an additional 30 days of inactivity.
If you would like to keep this issue open please add a comment and/or remove the MARKED_FOR_CLOSURE label.
If this issue should be kept open even with no activity beyond the time limits you can add the label DO_NOT_AUTOCLOSE.
If it is ok for this issue to be closed, feel free to go ahead and close it. Please do not add any comments or change any labels or otherwise touch this issue unless your intention is to reset the inactivity counter for an additional year.