Add bash linting to CI. Cleanup .sh scripts a bit. Address .sh bugs. Adds -v Verbose option. (#2218) Remove nowarn Intel compiler flag #2225

BrianCurtis-NOAA · 2024-04-03T23:56:59Z

Commit Queue Requirements:

Fill out all sections of this template.
All sub component pull requests have been reviewed by their code managers.
Run the full Intel+GNU RT suite (compared to current baselines) on either Hera/Derecho/Hercules
Commit 'test_changes.list' from previous step

Description:

Add bash linting to CI:
- uses superlinter to check for consistent bash code writing
Cleans up .sh scripts to comply with superlinter
Cleans up .sh scripts to be more consistent, easier to read.
Add's -v verbose option if debugging outputs needed, otherwise simplifies rt.sh run echo's.
Addresses smaller bugs
- quota/timeout search logic adjusted.
- check for dirs existing (DISKNM, STMP, PTMP) before starting.
- adjustments/cleanup to ecflow/rocoto sections
- rt.sh will attempt to start ecflow, and only stop ecflow if it started from rt.sh.
- fix for issue where run_dir will not delete properly.
Remove -nowarn compiler flag from cmake/Intel.cmake
Explicitly disable two warnings:
- warning #5462: Global name too long.
- remark #7712: This variable has not been used.

Commit Message:

* UFSWM
- Add bash linting to CI:
  - uses superlinter to check for consistent bash code writing
- Cleans up .sh scripts to comply with superlinter
- Cleans up .sh scripts to be more consistent, easier to read.
- Add's -v verbose option if debugging outputs needed, otherwise simplifies rt.sh run echo's.
- Addresses smaller bugs
  - quota/timeout search logic adjusted.
  - check for dirs existing (DISKNM, STMP, PTMP) before starting.
  - adjustments/cleanup to ecflow/rocoto sections
  - rt.sh will attempt to start ecflow, and only stop ecflow if it started from rt.sh.
  - fix for issue where run_dir will not delete properly.
* FV3: Address compiler warnings
  * atmos_cubed_sphere: Address compiler warnings.

Priority:

Normal

Git Tracking

UFSWM:

Sub component Pull Requests:

FV3: Update dycore to remove compiler warnings NOAA-EMC/fv3atm#813
- atmos_cubed_sphere: Address compiler warnings for FV3 raised in an issue NOAA-GFDL/GFDL_atmos_cubed_sphere#324
stochastic_physics: Clean up compiler warnings NOAA-PSL/stochastic_physics#78

UFSWM Blocking Dependencies:

None

Changes

Regression Test Changes (Please commit test_changes.list):

No Baseline Changes.

Input data Changes:

None.

Library Changes/Upgrades:

No Updates

Testing Log:

…ture

…ated with the changes

…h log minimally verbose without -v option

… skip error checking with qstat

…ther-model into cleanup_scripts

zach1221 · 2024-04-16T20:17:34Z

Still working the derecho issue, fyi. Everything else should be done.

zach1221 · 2024-04-16T20:26:05Z

Ok jobs are running now on Derecho.

zach1221 · 2024-04-16T23:24:29Z

Ok we're ready to merge finally.

BrianCurtis-NOAA · 2024-04-17T10:45:03Z

Almost, I would like a little time to see if i can get Acorn finished.

BrianCurtis-NOAA · 2024-04-17T16:54:33Z

Skipping Acorn. Start merge.

zach1221 · 2024-04-17T17:44:42Z

@BrianCurtis-NOAA fv3atm merged. Hash: NOAA-EMC/fv3atm@979bcab

BrianCurtis-NOAA · 2024-04-17T17:50:01Z

I have the FV3 updated, but waiting on stochastic_physics before pushing. Who normally merges in stochastic_physics?

zach1221 · 2024-04-17T17:56:42Z

I have the FV3 updated, but waiting on stochastic_physics before pushing. Who normally merges in stochastic_physics?

Oh, Phil Pegion generally. I can leave a comment there.

zach1221 · 2024-04-17T18:18:48Z

Ok, @BrianCurtis-NOAA . We should be ready now, Stoch Physics is merged.

DeniseWorthen · 2024-04-17T19:33:12Z

@BrianCurtis-NOAA Shouldn't you link other bug reports so that they'll be closed ? 2087 maybe 2150 and also 2148?

DeniseWorthen · 2024-04-17T19:56:03Z

I've added the additional issues close. I'm not sure if #2150 is fixed, so I'll leave that open for now.

commit f234a3e Author: Ufuk Turunçoğlu <[email protected]> Date: Tue Apr 30 11:35:25 2024 -0600 Fix for land component model (ufs-community#2191) * UFSWM - fix fully coupled land component configuration * NOAHMP - get fixed information from surface file commit 04bbc15 Author: jiandewang <[email protected]> Date: Thu Apr 25 14:52:00 2024 -0400 update MOM6 to its main repo. 20240401 commit (ufs-community#2241) * UFSWM - * MOM6 - update MOM6 to its main repo. 20240401 commit (NCAR-candidate-20240319) commit b6c576d Author: Daniel Sarmiento <[email protected]> Date: Tue Apr 23 12:24:22 2024 -0400 Merged global namelist (ufs-community#2173) * UFSWM - global_control.nml_IN has been added as the new regression test namelist template for all global regression tests. The namelist now uses pointers (i.e. @[abc]) for variables and default values have been added to the default_vars.sh script. A new section in default_vars.sh has been added (export_tiled) to account for tiled RTs that pulls the correct parameter files using the ATMRES variable. Regression tests have been modified to account for these changes. Tests that were not compatible with the GFSv17_p8 core have been disabled for now. They will be turned on as they are updated from GFSv16 to GFSv17. commit 5d2ca19 Author: WenMeng-NOAA <[email protected]> Date: Fri Apr 19 13:59:12 2024 -0400 Update upp submodule (ufs-community#2213) * UFSWM - Update inline post * FV3 - Update upp submodule for inline post commit 47c0099 Author: Brian Curtis <[email protected]> Date: Wed Apr 17 15:59:48 2024 -0400 Add bash linting to CI. Cleanup .sh scripts a bit. Address .sh bugs. Adds -v Verbose option. (ufs-community#2218) Remove nowarn Intel compiler flag (ufs-community#2225) * UFSWM - Add bash linting to CI: - uses superlinter to check for consistent bash code writing - Cleans up .sh scripts to comply with superlinter - Cleans up .sh scripts to be more consistent, easier to read. - Add's -v verbose option if debugging outputs needed, otherwise simplifies rt.sh run echo's. - Addresses smaller bugs - quota/timeout search logic adjusted. - check for dirs existing (DISKNM, STMP, PTMP) before starting. - adjustments/cleanup to ecflow/rocoto sections - rt.sh will attempt to start ecflow, and only stop ecflow if it started from rt.sh. - fix for issue where run_dir will not delete properly. * FV3: Address compiler warnings * atmos_cubed_sphere: Address compiler warnings. commit 4f32a4b Author: Rick Grubin <[email protected]> Date: Mon Apr 15 07:21:08 2024 -0600 Document ATMW / ATMAERO / HAFS WM configurations (ufs-community#2160) * UFSWM * doc/Userguide * source * conf.py * Configurations.rst * FAQ.rst * InputsOutputs.rst * Introduction.rst commit ac4445d Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon Apr 15 08:59:42 2024 -0400 Bump idna from 3.6 to 3.7 in /doc/UsersGuide (ufs-community#2234) *doc/UserGuide *requirements.txt - updates inda version from 3.6 to 3.7 commit 281b32f Author: Samuel Trahan (NOAA contractor) <[email protected]> Date: Mon Apr 15 08:38:01 2024 -0400 bug fixes: kchunk3d ignored, hailwat uninitialized in dycore, tile_num wrong for nests (ufs-community#2201) * UFSWM - None. * FV3 - Write component will use kchunk3d. Model init sends the right tile number to CCPP. * atmos_cubed_sphere - Initialize the hailwat variable. Pass global_tile index to model. commit 8a5f711 Author: Denise Worthen <[email protected]> Date: Thu Apr 11 13:32:26 2024 -0400 Add PIO namelist control for CICE (ufs-community#2145) Update to CICE-Consortium/CICE aca8357. Adds implementation of namelist PIO options for CICE commit 45c8b2a Author: JONG KIM <[email protected]> Date: Thu Apr 4 19:49:13 2024 -0400 Hotfix/cubed sphere hash fix: HAILCAST diagnostic code (units issue) (ufs-community#2223) cubed_sphere hash update: f060e85 for a bug- fix in the HAILCAST diagnostic code (units issue) commit 26e6db6 Author: Denise Worthen <[email protected]> Date: Wed Apr 3 19:57:08 2024 -0400 Enable cpl_scalars export from ATM and NoahMP for use by CMEPS (ufs-community#2175) * CMEPS - allow additional dimension in cpl_scalars for CSG and regional ATM domains for use in mediator history files * CMEPS - fix mapping mask for lnd->atm * FV3 - add export of cpl_scalars * NOAHMP - add export of cpl_scalars commit 1411b90 Author: Dusan Jovic <[email protected]> Date: Mon Apr 1 18:04:44 2024 -0400 Update module_write_netcdf to avoid hangs in RRFS runs (ufs-community#2193) * UFSWM - Update module_write_netcdf to avoid hangs in RRFS runs * FV3 - Update module_write_netcdf to avoid hangs in RRFS runs commit 87c27b9 Author: Matthew Masarik <[email protected]> Date: Fri Mar 29 15:23:42 2024 -0400 WW3 feature: Langmuir turbulence parameterization (ufs-community#2195) * WW3 - Langmuir turbulence parameterization commit c54e986 Author: Samuel Trahan (NOAA contractor) <[email protected]> Date: Wed Mar 27 16:11:03 2024 -0400 regression test system bug fixes, eliminate MOM6 warnings (ufs-community#2197), add xr_cnvcld flag to FV3 (ufs-community#2185) (ufs-community#2202) * UFSWM - atparse.bash: correctly handle input that doesn't end with an end-of-line character. Fix some bugs in Rocoto support and clean up rt.sh. * FV3 - namelist flag xr_cnvcld to control if suspended grid-mean convective cloud condensate should be included in cloud fraction and optical depth calculation in radiation in the GFS suite * ccpp - physics-level changes to implement new namelist variable * MOM6 - update MOM6 code to eliminate all compiler warnings

BrianCurtis-NOAA added 30 commits February 22, 2024 14:23

merge conflict through keeping upstream changes

02e289a

Address conflicts

59af98f

Add super linter GH action

bb77df4

Remove some directories from being linted, for now

1b3001c

not sure if this is the right fix, but making sure exclude path works OK

b6ece82

Maybe this is the way

44f2394

Let's only check shell for now, but keep all options available for fu…

7a29daa

…ture

updated to skip directories

409117f

updated to skip directories try 2

0dd460c

updated to skip directories try 3

af16d59

updated to skip directories try 4

e4edbc9

updated to skip directories try 5

d8f4680

cleanup rt.sh to pass shellcheck

0f17662

Update shell scripts to meet linter standards and address bugs associ…

978a86b

…ated with the changes

Bring in less hard coded values for ecflow_run

689a697

Address merge conflicts

c9b216b

ecflow counter was off, would error out but not report anything

4b2a3d9

Not test opnReqTest stuff yet, needs work

ce128f5

rt.sh only verbose on request, cleanup more linting issues, make rt.s…

b134a0d

…h log minimally verbose without -v option

Trying to update superlinter to ignore any opnReqTests stuff

4f1855f

Don't parse atparse.bash

ef4d7c6

Add linter rules path to superlinter

d2eb5d2

Trying something else for linter

9d61ef8

Maybe this time?

868e34e

Still not quite right, maybe this

8a91d7f

just guessing at this point

c3b49a7

just guessing at this point #2

d997293

Maybe just maybe?

f88c535

Ok this is working now, yay!

b8b92bc

RT Utils updates with set -e

050a450

BrianCurtis-NOAA and others added 6 commits April 16, 2024 12:00

Update to derecho pbs issues, pbs doesn't fail in a nice way, need to…

06d66ee

… skip error checking with qstat

Merge branch 'cleanup_scripts' of github.com:BrianCurtis-NOAA/ufs-wea…

d43ccd1

…ther-model into cleanup_scripts

Update opnReqTest export RTVERBOSE variable

9fd79a4

add control_p8 ORT logs: passed

c3606d7

add regional_control ORT logs: passed

5b03825

add cpld_control ORT logs: passed

1d53bb5

add derecho RT logs: passed

cddc583

BrianCurtis-NOAA mentioned this pull request Apr 17, 2024

Enable atmaq tests/compiles after updating DT_ATMOS to 120 (2min) #2240

Open

BrianCurtis-NOAA mentioned this pull request Apr 17, 2024

Clean up compiler warnings NOAA-PSL/stochastic_physics#78

Merged

Update FV3 hash and revert the FV3 .gitmodule

39021f1

Update stochastic_physics hash and revert its .gitmodules

22e6274

zach1221 requested review from jkbk2004, DeniseWorthen and FernandoAndrade-NOAA April 17, 2024 19:09

jkbk2004 approved these changes Apr 17, 2024

View reviewed changes

FernandoAndrade-NOAA approved these changes Apr 17, 2024

View reviewed changes

zach1221 merged commit 47c0099 into ufs-community:develop Apr 17, 2024
3 checks passed

grantfirl mentioned this pull request May 23, 2024

UFS-dev PR#184 NCAR/ufs-weather-model#132

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add bash linting to CI. Cleanup .sh scripts a bit. Address .sh bugs. Adds -v Verbose option. (#2218) Remove nowarn Intel compiler flag #2225

Add bash linting to CI. Cleanup .sh scripts a bit. Address .sh bugs. Adds -v Verbose option. (#2218) Remove nowarn Intel compiler flag #2225

BrianCurtis-NOAA commented Apr 3, 2024 •

edited by DeniseWorthen

Loading

zach1221 commented Apr 16, 2024

zach1221 commented Apr 16, 2024

zach1221 commented Apr 16, 2024

BrianCurtis-NOAA commented Apr 17, 2024

BrianCurtis-NOAA commented Apr 17, 2024

zach1221 commented Apr 17, 2024

BrianCurtis-NOAA commented Apr 17, 2024

zach1221 commented Apr 17, 2024

zach1221 commented Apr 17, 2024

DeniseWorthen commented Apr 17, 2024 •

edited

Loading

DeniseWorthen commented Apr 17, 2024

Add bash linting to CI. Cleanup .sh scripts a bit. Address .sh bugs. Adds -v Verbose option. (#2218) Remove nowarn Intel compiler flag #2225

Add bash linting to CI. Cleanup .sh scripts a bit. Address .sh bugs. Adds -v Verbose option. (#2218) Remove nowarn Intel compiler flag #2225

Conversation

BrianCurtis-NOAA commented Apr 3, 2024 • edited by DeniseWorthen Loading

Commit Queue Requirements:

Description:

Commit Message:

Priority:

Git Tracking

UFSWM:

Sub component Pull Requests:

UFSWM Blocking Dependencies:

Changes

Regression Test Changes (Please commit test_changes.list):

Input data Changes:

Library Changes/Upgrades:

Testing Log:

zach1221 commented Apr 16, 2024

zach1221 commented Apr 16, 2024

zach1221 commented Apr 16, 2024

BrianCurtis-NOAA commented Apr 17, 2024

BrianCurtis-NOAA commented Apr 17, 2024

zach1221 commented Apr 17, 2024

BrianCurtis-NOAA commented Apr 17, 2024

zach1221 commented Apr 17, 2024

zach1221 commented Apr 17, 2024

DeniseWorthen commented Apr 17, 2024 • edited Loading

DeniseWorthen commented Apr 17, 2024

BrianCurtis-NOAA commented Apr 3, 2024 •

edited by DeniseWorthen

Loading

DeniseWorthen commented Apr 17, 2024 •

edited

Loading