Apply TCPairs, TCStat, CyclonePlotter, GridStat to EMC 2020 Data

[fentoad72]

Replace italics below with details for this issue.

Describe the Task

Learn how to use TCPairs, TCStat, CyclonePlotter, and GridStat in the context of cyclones, using existing sample data from GitHub

Time Estimate

Estimate the amount of work required here.
3 calendar days of work spread out over December.
Future work will either extend this task or create another one

Sub-Issues

Consider breaking the task down into sub-issues.

• Add a checkbox for each sub-issue here.

Relevant Deadlines

• John O will provide more data probably in January.*

Funding Source

Define the source of funding and account keys here or state NONE.

Define the Metadata

Assignee

• Select engineer(s) or no engineer required
• Select scientist(s) or no scientist required

Labels

• Select component(s)
• Select priority
• Select requestor(s)

Projects and Milestone

• Review projects and select relevant Repository and Organization ones or add "alert:NEED PROJECT ASSIGNMENT" label
• Select milestone to next major version milestone or "Future Versions"

Define Related Issue(s)

Consider the impact to the other METplus components.

• METplus, MET, METdatadb, METviewer, METexpress, METcalcpy, METplotpy

Task Checklist

See the METplus Workflow for details.

• Complete the issue definition above, including the Time Estimate and Funding Source.
• Fork this repository or create a branch of develop.
  Branch name: feature_<Issue Number>_<Description>
• Complete the development and test your changes.
• Add/update log messages for easier debugging.
• Add/update unit tests.
• Add/update documentation.
• Push local changes to GitHub.
• Submit a pull request to merge into develop.
  Pull request: feature <Issue Number> <Description>
• Define the pull request metadata, as permissions allow.
  Select: Reviewer(s), Project(s), Milestone, and Linked issues
• Iterate until the reviewer(s) accept and merge your changes.
• Delete your fork or branch.
• Close this issue.

[fentoad72]

I've run through the wrappers using the canned data under /d1/projects/METplus/METplus_data successfully.

Then I wrote a python program to look at the 2020 verification data and extract the 000 hour forecast files into an analyses file. Not sure if the Python program is part of this issue or should be made into a new issue.

[JohnHalleyGotway]

Todd, When deciding whether or not to create a GitHub issue, I'd recommend asking a simple question. Do you anticipate that this work will add to or change any of the files in the repository? If so, please document the work as an issue. If not, please don't create an issue. For example, this issue, learning about existing tools, probably shouldn't be a GitHub issue at all. If you want to keep track of your work in GitHub, but that work won't change the repo, you can create a project card instead. See the "*+Add Cards*" button in the top-right corner: https://github.com/dtcenter/METplus/projects/25 Thanks, John

…

On Tue, Dec 29, 2020 at 2:03 PM Todd Arbetter ***@***.***> wrote: I've run through the wrappers using the canned data under /d1/projects/METplus/METplus_data successfully. Then I wrote a python program to look at the 2020 verification data and extract the 000 hour forecast files into an analyses file. Not sure if the Python program is part of this issue or should be made into a new issue. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#728 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFA4HKCUGP4676LRLVRSSM3SXI73PANCNFSM4UVO3JWA> .

[fentoad72]

I've gotten a sample storm through TCPairs, CyclonePlotter, and TCStat.

Next is to update repo as FeatureRelative cases may have changed

Then try to get the sample storm through the FeatureRelative cases

[j-opatz]

Reopening this to create a new PR that addresses a few issues with the embedded script

[j-opatz]

With the plotting error corrected and Python script updated to place the A and B deck in separate directories (to avoid reading in both decks as A decks and B decks), a new issue occurred where the resulting image is no longer comparable to Guang Ping's website imagery.
After a discussion with George, the central problem with the use case seems to be how TCPairs is being utilized. The tool is best suited for single storm files, not entire seasons or all storms in one file. This is even stated in the MET User's Guide. Currently the use case pushes one Adeck and one Bdeck that contains all storms with a valid warning time of 2020100700 (~430 storms). Combined with the inclusion of the unique name identifier column, TCPairs does its best to match up the two files but expectedly, encounters many issues.
The tested path will be twofold: the first is to eliminate the extra column, bringing the file in line with ATCF format. The second is to utilize the embedded Python script to create a separate A and B deck file for each storm, with a unique file name to identify the pair for TCPairs. This is much closer to the intended use of TCPairs and should result in more meaningful output.

[j-opatz]

Including here the shell script currently being run on Guang Ping's website.
verify_pair_glbl.sh.txt

[JohnHalleyGotway]

John O, Please be aware that you do NOT need to remove that extra column. The existing code does actually handle it. Although I suspect this may need to be updated in the user's guide. Take a look at this line: https://github.com/dtcenter/MET/blob/acf97fc54d22178993e13ed9e2acd81f6f52a9ae/met/src/libcode/vx_tc_util/atcf_line_base.cc#L174 All input TC data goes through this "ATCFLineBase::read_line()" function. It checks the contents of the 4th column (where TechniqueNumberOffset = 3 but is 0-based) and calls the "string_to_atcflinetype()" function. So the contents of that column determine how the line is parsed. For example, if it's in YYYYMMDDHH format, it's a genesis line type: https://github.com/dtcenter/MET/blob/acf97fc54d22178993e13ed9e2acd81f6f52a9ae/met/src/libcode/vx_tc_util/atcf_line_base.cc#L474 And that's exactly what's happening with Guang Ping's data with that extra column. So instead of changing the data to fit the code, I'd recommend that we change code as needed to fit the data. We just need to define exactly what it is that needs to be changed. Thanks, John HG

…

On Wed, Apr 7, 2021 at 12:55 PM j-opatz ***@***.***> wrote: Including here the shell script currently being run on Guang Ping's website. verify_pair_glbl.sh.txt <https://github.com/dtcenter/METplus/files/6273882/verify_pair_glbl.sh.txt> — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#728 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFA4HKHZDGUAF3DGDMN3J4TTHSTCNANCNFSM4UVO3JWA> .

[j-opatz]

Success: yesterday a working group with @JohnHalleyGotway and @georgemccabe had the breakthrough for this use case. The solution was in simplifying what was the actual end-goal; that is, to produce a picture comparable to Guang Ping's. Because extra TCs lack an authentic analysis track, the Adecks that were pulled from the source file needed to serve as their own analysis tracks, essentially creating a 1:1 track plot. At this time we aren't concerned with an analysis file; that's already done for us, inside the modified source file. This use case is a multi-step plotting example, pure and simple.
Future work includes allowing METplus to create these modified source files and plotting via METplotpy, possibly including the option to filter storms by their forecast length (to even better match the website output), but that work should reside in a separate use case.