Feature 348 dmap reformat

[bikegeek]

Pull Request Testing

• Describe testing already performed for these changes:
  

• manually compiled docs and verified no warnings, errors and content is as expected

• ran all the METreformat tests for DMAP and verified all tests pass

• generated simple scatter, line, and contour plots using METplotpy using DMAP output

verified that current tests are passing as a result of updates:

• updated the ECNT tests and data (now using latest ensemble stat output)

• fixed any affected tests

• Recommend testing for the reviewer(s) to perform, including the location of input datasets, and any additional instructions:
  

For @jprestop:

• verify all tests pass
• evaluate documentation for clarity of instructions

For @CPKalb:

• verify all tests pass
• reformat DMAP output for generating a line plot
• reformat DMAP output for generating a scatter plot
• generate one or both plot types using the appropriate reformatted data:
  - verify line plot can be generated (limited data, so not much to plot)
  - verify scatter plot can be generated (limited data, not much to plot)

Reformatting

Run the code on 'seneca':
METdataio (feature_348_dmap_reformat)
METcalcpy (develop)
METplotpy (feature_23_scatter) - not yet merged, refactored for Land Project

conda environment used
(permissions on conda and env are set up for anyone to use)
/d1/personal/mwin/miniconda3/condabin/conda activate /d1/personal/mwin/miniconda3/envs/mp_analysis_env

setting up PYTHONPATH

(also sets $DATAIO_BASE, $CALCPY_BASE, and $PLOTPY_BASE env vars)

• cd /d1/projects/NRL_DMAP
• switch to bash shell
• source setup_env.bash

Location:
/d1/projects/NRL_DMAP/

Reformat the DMAP linetype data for scatter plot and line plot:

For scatter plot format:
python $DATAIO_BASE/METreformat/write_stat_ascii.py $DATAIO_BASE/METreformat/reformat_dmap_for_scatter.yaml

File called reformat_dmap_for_scatter.data is created in the $DATAIO_BASE/METreformat/output directory

For line plot format:
**python $DATAIO_BASE/METreformat/write_stat_ascii.py $DATAIO_BASE/METreformat/reformat_dmap_for_line.yaml**

File called reformat_dmap_for_line.scatter is created in the $DATAIO_BASE/METreformat/output directory

Generate line or scatter plot to verify that reformatted files can be used by METplotpy

line plot:

python $PLOTPY_BASE/metplotpy/plots/line/line.py $PLOTPY_BASE/metplotpy/plots/line/custom_line.yaml

Creates the line_from_dmap.png file in $PLOTPY_BASE/metplotpy/plots/line

scatter plot:

python $PLOTPY_BASE/metplotpy/plots/scatter/scatter.py $PLOTPY_BASE/metplotpy/plots/scatter/dmap_scatter.yaml

Creates the scatter_dmap.png file in $PLOTPY_BASE/metplotpy/plots/scatter

• Do these changes include sufficient documentation updates, ensuring that no errors or warnings exist in the build of the documentation? [Yes]

• Do these changes include sufficient testing updates? [NA]

• Will this PR result in changes to the test suite? [No]
  
  If yes, describe the new output and/or changes to the existing output:
  

• Do these changes introduce new SonarQube findings? [No]
  
  If yes, please describe:

• Please complete this pull request review by earliest convenience.
  

Pull Request Checklist

See the METplus Workflow for details.

• Add any new Python packages to the METplus Components Python Requirements table.
• Review the source issue metadata (required labels, projects, and milestone).
• Complete the PR definition above.
• Ensure the PR title matches the feature or bugfix branch name.
• Define the PR metadata, as permissions allow.
  Select: Reviewer(s) and Development issue
  Select: Milestone as the version that will include these changes
  Select: Coordinated METplus-X.Y Support project for bugfix releases or METdataio-X.Y.Z Development project for official releases
• After submitting the PR, select the ⚙️ icon in the Development section of the right hand sidebar. Search for the issue that this PR will close and select it, if it is not already selected.
• After the PR is approved, merge your changes. If permissions do not allow this, request that the reviewer do the merge.
• Close the linked issue and delete your feature or bugfix branch from GitHub.

[jprestop]

@bikegeek Thanks for getting that cleaned up! I approve this request with regard to the documentation. Thanks for all of your work on this task!

[CPKalb]

Do you have examples of what the output graphics should look like? I was able to generate the images but want to make sure I'm getting the expected results.

[bikegeek]

on seneca /d1/projects/NRL_DMAP/METplotpy/metplotpy/plots/line/mw_output and /d1/projects/NRL_DMAP/METplotpy/metplotpy/plots/scatter/mw_output

…

On Thu, Dec 12, 2024 at 3:48 PM Christina Kalb ***@***.***> wrote: Do you have examples of what the output graphics should look like? I was able to generate the images but want to make sure I'm getting the expected results. — Reply to this email directly, view it on GitHub <#354 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA4UJHUDLY6DJTJTKIQ3G432FIHDJAVCNFSM6AAAAABTJZNS46VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNBQGE3DGNZQGA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[CPKalb]

Ran the reformatter and plots, and was able to successfully generate the expected results.

[jprestop]

@bikegeek I see this issue was merged into a main_v3.1 branch. Should that have been the main_v3.0 branch and the develop branch? I'm confused because I think we shouldn't have a main_v3.1 branch yet. Maybe I do not fully understand the process?

[bikegeek]

I merged into the 3_1 branch because this is NOT going into the coordinated release.

…

On Fri, Dec 13, 2024 at 9:08 AM Julie Prestopnik ***@***.***> wrote: @bikegeek <https://github.com/bikegeek> I see this issue was merged into a main_v3.1 branch. Should that have been the main_v3.0 branch and the develop branch? I'm confused because I think we shouldn't have a main_v3.1 branch yet. Maybe I do not fully understand the process? — Reply to this email directly, view it on GitHub <#354 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA4UJHT75YLNI2UHBEFHKYL2FMA6ZAVCNFSM6AAAAABTJZNS46VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNBRG43TMNBRG4> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[jprestop]

@bikegeek I believe there should only be a develop branch at this time and not a main_v3.1 branch.

According to the release instructions, main_v3.1 should be created only when we're creating the 3.1.0-rc1 release and not prior to. These instructions are all new since this last 3.0.0-rc1 release and I know it's a bit confusing.

I believe we need to merge main_v3.1 into develop and delete main_v3.1 now.

[bikegeek]

But that will put the DMAP changes into the coordinated release, which is new code. I thought we weren't supposed to introduce new code after release candidates?

…

On Fri, Dec 13, 2024 at 9:41 AM Julie Prestopnik ***@***.***> wrote: @bikegeek <https://github.com/bikegeek> I believe there should only be a develop branch at this time and not a main_v3.1 branch. According to the release instructions <https://metplus.readthedocs.io/en/develop/Release_Guide/met_development.html#create-release-reference-branch>, main_v3.1 should be created only when we're creating the 3.1.0-rc1 release and not prior to. These instructions are all new since this last 3.0.0-rc1 release and I know it's a bit confusing. I believe we need to merge main_v3.1 into develop and delete main_v3.1 now. — Reply to this email directly, view it on GitHub <#354 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA4UJHWGPIV7ZHFCAT3WRID2FME2DAVCNFSM6AAAAABTJZNS46VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNBRHAZTKOBZG4> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[jprestop]

@bikegeek Thanks for the question. The main_v3.0 branch is the branch that will become the coordinated release. The current develop branch is intended for development for v3.1.0. Anything that you want to go into v3.1.0 should go into develop. Anything you want in BOTH the coordinated release AND 3.1.0 should go into both main_v3.0 and develop. Hopefully, this feature branch feature_348_dmap_reformat was created off of develop. It sounds like you do not want these changes in the 3.0.0 coordinated release, but into the 3.1.0 coordinated release, so they should be merged into develop. I'll schedule a quick call for now for you, me, @JohnHalleyGotway, and @georgemccabe to ensure we're all on the same page and work through any questions. It's a bit confusing having switched our methods in the last rc1 release.