Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Stage initial conditions for cycled and forecast-only via stage_ic job. #2475

Closed
aerorahul opened this issue Apr 9, 2024 · 2 comments · Fixed by #2651
Closed

Stage initial conditions for cycled and forecast-only via stage_ic job. #2475

aerorahul opened this issue Apr 9, 2024 · 2 comments · Fixed by #2651
Assignees
@KateFriedman-NOAA
Labels
feature New feature or request

Comments

@aerorahul
Copy link
Contributor

aerorahul commented Apr 9, 2024 •
edited by KateFriedman-NOAA
Loading

What new functionality do you need?

Presently, the cycled initial conditions are staged by setup_expt.py.
The initial conditions for forecast-only are staged by a job stage_ic.

This feature requests development of the stage_ic job to stage cycled initial conditions in addition to the ones for forecast-only.

What are the requirements for the new functionality?

The new stage_ic job should be able to handle both cycled and forecast-only modes as well as for gfs and gefs systems.
Extra points for making the job generic to allow users to provide (files) initial conditions of their choosing from arbitrary locations and structure.

Acceptance Criteria

Add the stage_ic job to the first half-cycle of the cycled mode.
@aerorahul aerorahul added feature New feature or request triage Issues that are triage labels Apr 9, 2024
@WalterKolczynski-NOAA WalterKolczynski-NOAA removed the triage Issues that are triage label Apr 10, 2024
@HenryRWinterbottom HenryRWinterbottom self-assigned this Apr 22, 2024
@aerorahul aerorahul mentioned this issue Apr 29, 2024
Closed
7 tasks
@WalterKolczynski-NOAA
Copy link
Contributor

@KateFriedman-NOAA See #2559 for changes relevant to this issue

@WalterKolczynski-NOAA
Copy link
Contributor

@KateFriedman-NOAA #2545 is also adding options to stage_ic you'll have to handle

KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue May 14, 2024
@KateFriedman-NOAA
Rename exglobal_stage script ahead of rewrite
25a959e 
- Change the extension of the exglobal_stage ex-script
from "sh" to "py".

Refs NOAA-EMC#2475
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue May 14, 2024
@KateFriedman-NOAA
Update for pythonization of stage script
9bf02ee 
- Update script extension for ex-script from "sh" to "py".
- Pull COM* variable declares up from ex-script.

Refs NOAA-EMC#2475
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue May 14, 2024
@KateFriedman-NOAA
Add PYTHONPATH to stage_ic.sh
8d2203d 
Refs NOAA-EMC#2475
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue May 14, 2024
@KateFriedman-NOAA
Remove IC staging from setup
ac9fe80 
Remove the functions and calls to set up
symlinks to ICs in ROTDIR.

Refs NOAA-EMC#2475
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue May 14, 2024
@KateFriedman-NOAA
Add initial changes to pythonize and yamlize staging
04d75e3 
- Add initial new yaml files for staging information
- Add new stage.py to python tasks.
- Add first draft pythonization of stage ex-script.

Much more work is still to be done.

Refs NOAA-EMC#2475
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue May 14, 2024
@KateFriedman-NOAA
Remove warm start check for ingesting fv3.yaml.j2
f6548e3 
Refs NOAA-EMC#2475
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue May 22, 2024
@KateFriedman-NOAA
Merge remote-tracking branch 'upstream/develop' into feature/issue_2475
7fc7b89 
Revert changes to workflow/setup_expt.py; will do in later task

Refs NOAA-EMC#2475

* upstream/develop:
  Sea-ice analysis insertion (NOAA-EMC#2584)
  Refactored archiving (NOAA-EMC#2491)
  Add remove RUNDIRS step in CI before creating experements (NOAA-EMC#2607)
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue May 22, 2024
@KateFriedman-NOAA
Split fv3 yaml into warm and cold
11df1af 
Also fix to set target and remove source

Refs NOAA-EMC#2475
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue May 22, 2024
@KateFriedman-NOAA
Fix other yamls to work
606c34a 
Add target, remove source, and update file info

Refs NOAA-EMC#2475
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue May 22, 2024
@KateFriedman-NOAA
Add COM variables exports in JGLOBAL_STAGE_IC
0176d2e 
Refs NOAA-EMC#2475
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue May 22, 2024
@KateFriedman-NOAA
Update fv3_cold yaml
4861ed1 
Update to use mkdir and copy instead of target and required

Refs NOAA-EMC#2475
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue May 22, 2024
@KateFriedman-NOAA
Update stage python scripts
b7a8c82 
Refs NOAA-EMC#2475
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue May 22, 2024
@KateFriedman-NOAA
Remove cycle_HH and cycle_YMD from fv3_cold yaml
5898ee2 
Refs NOAA-EMC#2475
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue May 23, 2024
@KateFriedman-NOAA
Remove unneeded funcions from stage.py
5e8f92f 
Refs NOAA-EMC#2475
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue May 23, 2024
@KateFriedman-NOAA
Refactor stage yamls
1cf0063 
Refs NOAA-EMC#2475
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue May 23, 2024
@KateFriedman-NOAA
Add MEMDIR array to JGLOBAL_STAGE_IC
9a39787 
Refs NOAA-EMC#2475
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue May 23, 2024
@KateFriedman-NOAA
Remove stage_ic job dependencies from setup
cd362fd 
Refs NOAA-EMC#2475
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue May 23, 2024
@KateFriedman-NOAA
Set MEMDIR as empty if not set
859d31d 
Refs NOAA-EMC#2475
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue May 23, 2024
@KateFriedman-NOAA
Add keys_gefs in exglobal_stage_ic.py
a805fa8 
If RUN=gefs add keys_gefs to keys.

Refs NOAA-EMC#2475
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue May 30, 2024
@KateFriedman-NOAA
Add USE_OCN_PERTURB_FILES to gfs config.base
719701b 
Set to .false. by default; needed for staging job

Refs NOAA-EMC#2475
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue May 30, 2024
@KateFriedman-NOAA
Stage job yaml updates
cf50f54 
- remove master yaml, no longer using
- update fv3_cold, ice, ocean, and wave yamls

Refs NOAA-EMC#2475
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue Aug 6, 2024
@KateFriedman-NOAA @WalterKolczynski-NOAA
Discontinue exporting rRUN
256b8e2 
Refs NOAA-EMC#2475

Co-authored-by: Walter Kolczynski - NOAA <[email protected]>
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue Aug 6, 2024
@KateFriedman-NOAA @WalterKolczynski-NOAA
Update cycle variables to use PDYcyc
da0e39c 
Refs NOAA-EMC#2475

Co-authored-by: Walter Kolczynski - NOAA <[email protected]>
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue Aug 6, 2024
@KateFriedman-NOAA @WalterKolczynski-NOAA
Add -x for COMOUT_ATMOS_ANALYSIS declare
0ee847f 
Refs NOAA-EMC#2475

Co-authored-by: Walter Kolczynski - NOAA <[email protected]>
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue Aug 6, 2024
@KateFriedman-NOAA
Update staging job declares
3fee34c 
- Remove exports and add "-x" to commands
- Add current_cycle to exports
- Move REPLAY_ICS if inside DO_OCN if

Refs NOAA-EMC#2475
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue Aug 6, 2024
@KateFriedman-NOAA @WalterKolczynski-NOAA
Update ENS check in config.stage_ic for gefs
8f096b7 
Refs NOAA-EMC#2475

Co-authored-by: Walter Kolczynski - NOAA <[email protected]>
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue Aug 6, 2024
@KateFriedman-NOAA @DavidHuber-NOAA
Remove PYTHONPATH export in stage_ic.sh
0527267 
Refs NOAA-EMC#2475

Co-authored-by: David Huber <[email protected]>
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue Aug 6, 2024
@KateFriedman-NOAA
Change non-exported variables to lowercase
00d0a8c 
Do so in stage configs

Refs NOAA-EMC#2475
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue Aug 6, 2024
@KateFriedman-NOAA
Add missing documentation to execute_stage function
e14e623 
Refs NOAA-EMC#2475
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue Aug 6, 2024
@KateFriedman-NOAA
Move IC variables from base to stage config
a0b9af0 
- Move ICSDIR and BASE_IC variables to config.stage_ic
- Update workflow/setup_expt.py to fill in templates
in config.stage_ic based on user input and host

Refs NOAA-EMC#2475
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue Aug 6, 2024
@KateFriedman-NOAA
Correct spelling mistake in config.com
bb54087 
- Change "decalre" to "declare"

Refs NOAA-EMC#2475
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue Aug 6, 2024
@KateFriedman-NOAA
Add comments in config.stage_ic
6f2b75a 
Short descriptions for ICSDIR and BASE_IC

Refs NOAA-EMC#2475
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue Aug 6, 2024
@KateFriedman-NOAA
MEMDIR updates in JGLOBAL_STAGE_IC
55e47c6 
- Remove MEMDIR export
- Add MEMDIR input for all declare commands that may need it;
if not gefs or enkfgdas then MEMDIR will be empty and COM won't include
- Remove incorrect comment about locally scoped variables

Refs NOAA-EMC#2475
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue Aug 12, 2024
@KateFriedman-NOAA
Add enkfgdas into stage_ic job checks
2913b58 
Refs NOAA-EMC#2475
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue Aug 12, 2024
@KateFriedman-NOAA
Refactor staging job for mem loop and COM declare
b74e696 
- Move rRUN, mem list, and COM declares inside python/yaml
- Move mem loop inside python
- Add path_exists checks to some initial conditions
- Create dicts for replace_tmpl COM declares

Refs NOAA-EMC#2475
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue Aug 12, 2024
@KateFriedman-NOAA
Resolve pynorms errors
9a0c642 
Refs NOAA-EMC#2475
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue Aug 12, 2024
@KateFriedman-NOAA
Some fixes and updates to conditions
863ecee 
- Replace all o_prefix with m_prefix and remove offset time
- Fix condition check for REPLAY_ICS (YES -> True)
- Add check against DO_JEDIOCNVAR for ice ICs
- Update model_start_date_current_cycle setting for MODE

Refs NOAA-EMC#2475
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue Aug 12, 2024
@KateFriedman-NOAA
Add ocninc.nc copy to stage.yaml.j2
7b3edf0 
- when DO_OCN=YES and DO_JEDIOCNVAR=YES

Refs NOAA-EMC#2475
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue Aug 12, 2024
@KateFriedman-NOAA
Add IC timestamps to ICSDIR in CI yamls
c7ec976 
Refs NOAA-EMC#2475
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue Aug 13, 2024
@KateFriedman-NOAA
Update AWSPW host BASE_IC path
ff1246b 
Refs NOAA-EMC#2475
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue Aug 13, 2024
@KateFriedman-NOAA
Consolidate staging settings
7ae9ea4 
- Move cycle, rRUN, model_start_date_current_cycle, and
member start/stop settings into new configure function
and out of JJOB.
- Address some reviewer feedback and make corrections.

Refs NOAA-EMC#2475
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue Aug 13, 2024
@KateFriedman-NOAA
Address pynorms errors
5c1f0c8 
Refs NOAA-EMC#2475
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue Aug 13, 2024
@KateFriedman-NOAA
Compress first[last]_mem logic and remove unused variable
c6bb969 
Refs NOAA-EMC#2475
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue Aug 13, 2024
@KateFriedman-NOAA
Remove unneeded lines and update mem setup
3022650 
- Check for "enkfgdas" or "gefs" when NMEM_ENS>=0
for first[last]_mem configuration

Refs NOAA-EMC#2475
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue Aug 15, 2024
@KateFriedman-NOAA
Separate stage yaml into component ones
54f8326 
- Create master yamls for gfs and gefs
- Create yamls for each component/group
- Update python to move all logic inside yamls
- Update yaml names in gfs/gefs configs

Refs NOAA-EMC#2475
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue Aug 15, 2024
@KateFriedman-NOAA
Remove enkfgdas from gdas/gfs stage dependency
a53c97e 
Refs NOAA-EMC#2475
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue Aug 19, 2024
@KateFriedman-NOAA
Adjust mem if-block and fix job list for JEDI
13dfad2 
- Incorporate reviewer feedback and adjust first[last_mem]
if-block in master_gfs.yaml.j2 to only check if RUN=enkfgdas
when setting ensemble mem values and not check for NMEM_ENS>0
- Update task list when DO_JEDIATMENS=YES to include the
stage_ic job in the hybrid_after_eupd_tasks group instead of
the hybrid_tasks. Now the enkfgdasstage_ic job shows up in
the xml, it was missing before.

Refs NOAA-EMC#2475
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue Aug 19, 2024
@KateFriedman-NOAA
Update analysis yaml for enkfgdas ratminc.nc
ab35ffc 
- Add ratminc.nc to the list of files in the analysis
staging yaml.
- Remove "gdas" condition for including analysis yaml.

Refs NOAA-EMC#2475
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@KateFriedman-NOAA KateFriedman-NOAA

Labels
feature New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Convert staging job to python and yaml KateFriedman-NOAA/global-workflow
4 participants
@aerorahul @WalterKolczynski-NOAA @KateFriedman-NOAA @HenryRWinterbottom