Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cleanup job for GEFS #2919

Merged
merged 16 commits into from
Sep 14, 2024
Merged

Cleanup job for GEFS #2919

merged 16 commits into from
Sep 14, 2024

Conversation

AntonMFernando-NOAA
Copy link
Contributor

@AntonMFernando-NOAA AntonMFernando-NOAA commented Sep 13, 2024 •
edited
Loading

Description

As forecast ensemble jobs are added to the global workflow, this PR ensures the output is being cleaned up properly once it is no longer needed.
Resolves #833

Type of change

  • A cleanup job that properly removes old ensemble data once it is no longer needed.
  • Add the cleanup job to the rocoto mesh.

Change characteristics

  • Is this a breaking change (a change in existing functionality)? NO
  • Does this change require a documentation update? NO
  • Does this change require an update to any of the following submodules? NO

How has this been tested?

  • CI test on HERCULES

Checklist

  • Any dependent changes have been merged and published
  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • My changes generate no new warnings
  • New and existing tests pass with my changes
  • I have made corresponding changes to the documentation if necessary

@AntonMFernando-NOAA AntonMFernando-NOAA marked this pull request as ready for review September 13, 2024 19:27
aerorahul
aerorahul reviewed Sep 13, 2024
View reviewed changes
Copy link
Contributor

@aerorahul aerorahul left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good.
just a couple of comments.

parm/config/gefs/config.cleanup Outdated Show resolved Hide resolved
parm/config/gefs/config.cleanup Outdated Show resolved Hide resolved
WalterKolczynski-NOAA
WalterKolczynski-NOAA requested changes Sep 13, 2024
View reviewed changes
Copy link
Contributor

@WalterKolczynski-NOAA WalterKolczynski-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We will run this through the normal CI tests when it is complete, but to test it you'll need to run several cycles to test the cleanup bit properly. We don't keep a bunch of GEFS initial conditions around, so work with the ensemble team to set that up.

workflow/rocoto/gefs_tasks.py Outdated Show resolved Hide resolved
WalterKolczynski-NOAA
WalterKolczynski-NOAA approved these changes Sep 14, 2024
View reviewed changes
Copy link
Contributor

@WalterKolczynski-NOAA WalterKolczynski-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Conditionally approved pending CI and manual completion of a longer GEFS test.

@aerorahul aerorahul merged commit b7f71e0 into NOAA-EMC:develop Sep 14, 2024
5 checks passed
DavidHuber-NOAA added a commit to DavidHuber-NOAA/global-workflow that referenced this pull request Sep 16, 2024
@DavidHuber-NOAA
Merge remote-tracking branch 'origin/develop' into feature/parallel_metp
91c0ffe 
* origin/develop:
  Update config.resources for bufr sounding job postsnd (NOAA-EMC#2917)
  Cleanup job for GEFS (NOAA-EMC#2919)
  Build GDASApp and unset memory in Gaea-C5 xml files (NOAA-EMC#2912)
  add 1 deg ocean/ice info to parm/config/gfs/config.resources (NOAA-EMC#2922)
  Support gefs C48 on Azure (NOAA-EMC#2881)
  Disable native grid writes for non-JEDI experiments; update C384 compression options (NOAA-EMC#2914)
DavidHuber-NOAA added a commit to DavidHuber-NOAA/global-workflow that referenced this pull request Sep 16, 2024
@DavidHuber-NOAA
Merge remote-tracking branch 'emc/develop' into fix/c768_hera
ec0803d 
* emc/develop:
  Update config.resources for bufr sounding job postsnd (NOAA-EMC#2917)
  Cleanup job for GEFS (NOAA-EMC#2919)
  Build GDASApp and unset memory in Gaea-C5 xml files (NOAA-EMC#2912)
  add 1 deg ocean/ice info to parm/config/gfs/config.resources (NOAA-EMC#2922)
  Support gefs C48 on Azure (NOAA-EMC#2881)
  Disable native grid writes for non-JEDI experiments; update C384 compression options (NOAA-EMC#2914)
  Add new UPP links to .gitignore (NOAA-EMC#2904)
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this pull request Sep 17, 2024
@KateFriedman-NOAA
Merge remote-tracking branch 'origin/develop' into feature/experiment…
3357a0d 
…al_obs_path

* origin/develop:
  Update global atmos upp job to use COMIN/COMOUT (NOAA-EMC#2867)
  Update config.resources for bufr sounding job postsnd (NOAA-EMC#2917)
  Cleanup job for GEFS (NOAA-EMC#2919)
  Build GDASApp and unset memory in Gaea-C5 xml files (NOAA-EMC#2912)
  add 1 deg ocean/ice info to parm/config/gfs/config.resources (NOAA-EMC#2922)
  Support gefs C48 on Azure (NOAA-EMC#2881)
  Disable native grid writes for non-JEDI experiments; update C384 compression options (NOAA-EMC#2914)
bbakernoaa pushed a commit to bbakernoaa/global-workflow that referenced this pull request Sep 19, 2024
@AntonMFernando-NOAA @bbakernoaa
Cleanup job for GEFS (NOAA-EMC#2919)
b22d7bc 
As forecast ensemble jobs are added to the global workflow, this PR
ensures the output is being cleaned up properly once it is no longer
needed.
  Resolves NOAA-EMC#833
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@WalterKolczynski-NOAA WalterKolczynski-NOAA WalterKolczynski-NOAA approved these changes

@aerorahul aerorahul aerorahul approved these changes

@DavidHuber-NOAA DavidHuber-NOAA Awaiting requested review from DavidHuber-NOAA DavidHuber-NOAA is a code owner

@EricSinsky-NOAA EricSinsky-NOAA Awaiting requested review from EricSinsky-NOAA

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Update cleanup job to scrub ensemble data
3 participants
@AntonMFernando-NOAA @aerorahul @WalterKolczynski-NOAA