Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

updates for fix files for uGWD #346

Closed
JessicaMeixner-NOAA opened this issue Jun 16, 2021 · 13 comments · Fixed by #373
Closed

updates for fix files for uGWD #346

JessicaMeixner-NOAA opened this issue Jun 16, 2021 · 13 comments · Fixed by #373
Assignees
@WalterKolczynski-NOAA
Milestone
Prototype 7

Comments

@JessicaMeixner-NOAA
Copy link
Contributor

There are new fix files for uGWD here:
/scratch1/NCEPDEV/global/Fanglin.Yang/save/git/gfsv16_rrtmgp2/fix/fix_ugwd

Namelist changes are needed to use the uGWD which should be in the 35 day tests in the p7b branch:
https://github.com/ufs-community/ufs-weather-model/tree/release/P7b the code updates themselves should already be in the develop branch of ufs-weather-model (not sure which commit/PR those updates went in for an exact date).

FYI: @yangfanglin @KateFriedman-NOAA @junwang-noaa @WalterKolczynski-NOAA @jiandewang
@yangfanglin
Copy link
Contributor

yangfanglin commented Jun 16, 2021 via email

@KateFriedman-NOAA
Copy link
Member

The new fix_ugwd folder and files are now in both the dev fix and fix_NEW sets. The dev fix sets were updated on all WCOSSes, Hera, Orion, Jet, and RZDM. The fix_NEW set was updated where it is right now (Mars and Hera).

[emc.glopara@m71a2 fix]$ pwd
/gpfs/dell2/emc/modeling/noscrub/emc.glopara/git/fv3gfs/fix
[emc.glopara@m71a2 fix]$ ll fix_ugwd/
total 12
drwxr-sr-x 2 emc.glopara global 2048 Jun 13 04:07 C384
drwxr-sr-x 2 emc.glopara global 2048 Jun 13 04:07 C768
drwxr-sr-x 2 emc.glopara global 2048 Jun 13 04:07 C96
[emc.glopara@m71a2 fix]$ ll fix_ugwd/*
fix_ugwd/C384:
total 73728
-rw-r--r-- 1 emc.glopara global 5898708 Aug 24  2020 C384_oro_data_ls.tile1.nc
-rw-r--r-- 1 emc.glopara global 5898708 Aug 24  2020 C384_oro_data_ls.tile2.nc
-rw-r--r-- 1 emc.glopara global 5898708 Aug 24  2020 C384_oro_data_ls.tile3.nc
-rw-r--r-- 1 emc.glopara global 5898708 Aug 24  2020 C384_oro_data_ls.tile4.nc
-rw-r--r-- 1 emc.glopara global 5898708 Aug 24  2020 C384_oro_data_ls.tile5.nc
-rw-r--r-- 1 emc.glopara global 5898708 Aug 24  2020 C384_oro_data_ls.tile6.nc
-rw-r--r-- 1 emc.glopara global 5898708 Aug 24  2020 C384_oro_data_ss.tile1.nc
-rw-r--r-- 1 emc.glopara global 5898708 Aug 24  2020 C384_oro_data_ss.tile2.nc
-rw-r--r-- 1 emc.glopara global 5898708 Aug 24  2020 C384_oro_data_ss.tile3.nc
-rw-r--r-- 1 emc.glopara global 5898708 Aug 24  2020 C384_oro_data_ss.tile4.nc
-rw-r--r-- 1 emc.glopara global 5898708 Aug 24  2020 C384_oro_data_ss.tile5.nc
-rw-r--r-- 1 emc.glopara global 5898708 Aug 24  2020 C384_oro_data_ss.tile6.nc

fix_ugwd/C768:
total 282624
-rw-r--r-- 1 emc.glopara global 23593428 Sep 19  2019 C768_oro_data_ls.tile1.nc
-rw-r--r-- 1 emc.glopara global 23593428 Sep 19  2019 C768_oro_data_ls.tile2.nc
-rw-r--r-- 1 emc.glopara global 23593428 Sep 19  2019 C768_oro_data_ls.tile3.nc
-rw-r--r-- 1 emc.glopara global 23593428 Sep 19  2019 C768_oro_data_ls.tile4.nc
-rw-r--r-- 1 emc.glopara global 23593428 Sep 19  2019 C768_oro_data_ls.tile5.nc
-rw-r--r-- 1 emc.glopara global 23593428 Apr  3  2020 C768_oro_data_ls.tile6.nc
-rw-r--r-- 1 emc.glopara global 23593428 Sep 19  2019 C768_oro_data_ss.tile1.nc
-rw-r--r-- 1 emc.glopara global 23593428 Sep 19  2019 C768_oro_data_ss.tile2.nc
-rw-r--r-- 1 emc.glopara global 23593428 Sep 19  2019 C768_oro_data_ss.tile3.nc
-rw-r--r-- 1 emc.glopara global 23593428 Sep 19  2019 C768_oro_data_ss.tile4.nc
-rw-r--r-- 1 emc.glopara global 23593428 Sep 19  2019 C768_oro_data_ss.tile5.nc
-rw-r--r-- 1 emc.glopara global 23593428 Sep 19  2019 C768_oro_data_ss.tile6.nc

fix_ugwd/C96:
total 6144
-rw-r--r-- 1 emc.glopara global 369108 Dec 27  2018 C96_oro_data_ls.tile1.nc
-rw-r--r-- 1 emc.glopara global 369108 Dec 27  2018 C96_oro_data_ls.tile2.nc
-rw-r--r-- 1 emc.glopara global 369108 Dec 27  2018 C96_oro_data_ls.tile3.nc
-rw-r--r-- 1 emc.glopara global 369108 Dec 27  2018 C96_oro_data_ls.tile4.nc
-rw-r--r-- 1 emc.glopara global 369108 Dec 27  2018 C96_oro_data_ls.tile5.nc
-rw-r--r-- 1 emc.glopara global 369108 Dec 27  2018 C96_oro_data_ls.tile6.nc
-rw-r--r-- 1 emc.glopara global 369108 Dec 19  2018 C96_oro_data_ss.tile1.nc
-rw-r--r-- 1 emc.glopara global 369108 Dec 27  2018 C96_oro_data_ss.tile2.nc
-rw-r--r-- 1 emc.glopara global 369108 Dec 27  2018 C96_oro_data_ss.tile3.nc
-rw-r--r-- 1 emc.glopara global 369108 Dec 27  2018 C96_oro_data_ss.tile4.nc
-rw-r--r-- 1 emc.glopara global 369108 Dec 27  2018 C96_oro_data_ss.tile5.nc
-rw-r--r-- 1 emc.glopara global 369108 Dec 27  2018 C96_oro_data_ss.tile6.nc

@yangfanglin
Copy link
Contributor

yangfanglin commented Jun 17, 2021 via email

@KateFriedman-NOAA
Copy link
Member

@yangfanglin The additional uGWD datasets for the other resolutions are now in the dev fix and fix_NEW sets on supported platforms. Thanks!

@JessicaMeixner-NOAA JessicaMeixner-NOAA added this to the Prototype 7 milestone Jun 22, 2021
@WalterKolczynski-NOAA WalterKolczynski-NOAA self-assigned this Jul 7, 2021
@WalterKolczynski-NOAA
Copy link
Contributor

WalterKolczynski-NOAA commented Jul 21, 2021 •
edited
Loading

Can @SMoorthi-emc or @yangfanglin help with the following? I'm trying to update the forecast script and I want to make sure the namelist options being set are consistent for the different UGWD options. (NB: I'm currently using knob_ugwp_version = -1 as a setting for UGWP off)

  • Do some of the v1 options need to also be set in v0?
  • Should the do_gsl_drag* options be determined independent of UGWD?
  • Is lsm=2 actually RUC?
  • Is the &cires_ugwp_nml section of the namelist ignored if do_ugwp is false?
  • Are there any other consistency issues?
  • Why are there so many namelist options that could just be determined by other namelist options and eliminate the possibility of a mismatch?
	case $knob_ugwp_version in;
		-1)	# Non-UGWD
			gwd_opt="1"
			do_ugwp=".false."
			do_ugwp_v0=".false."
			do_ugwp_v1=".false."
			do_tofd=".true."
			;;
		0)	# UGWD v0
			gwd_opt="2"
			do_ugwp=".true."
			do_ugwp_v0=".true."
			do_ugwp_v1=".false."
			do_tofd=".true."
			;;
		1)	# UGWD v1
			gwd_opt="2"
			do_ugwp=".true."
			do_ugwp_v0=".false."
			do_ugwp_v1=".true."
			do_tofd=".false."

			do_gsl_drag_ls_bl=".true."
			do_gsl_drag_ss=".true."
			do_gsl_drag_tofd=".true."

			OROFIX_ugwd=${OROFIX_ugwd:-"${FIX_DIR}/fix_ugwd"}
			$NLN ${OROFIX_ugwd}/ugwp_limb_tau.nc $DATA/ugwp_limb_tau.nc
			for n in $(seq 1 $ntiles); do
				$NLN ${OROFIX_ugwd}/$CASE/${CASE}_oro_data_ls.tile${n}.nc $DATA/INPUT/oro_data_ls.tile${n}.nc
				$NLN ${OROFIX_ugwd}/$CASE/${CASE}_oro_data_ss.tile${n}.nc $DATA/INPUT/oro_data_ss.tile${n}.nc
			done

			knob_ugwp_dokdis=${knob_ugwp_dokdis:-2}
			knob_ugwp_ndx4lh=${knob_ugwp_ndx4lh:-4}
			lsm=${lsm:-"2"} #RUC LSM?
			;;
		*) # Unknown
			echo "FATAL: Unknown value of knob_ugwp_version provided: ${knob_ugwp_version}!"
			exit 1
	esac # $knob_ugwp_version

@yangfanglin
Copy link
Contributor

yangfanglin commented Jul 21, 2021 via email

@WalterKolczynski-NOAA
Copy link
Contributor

Walter, lsm=2 is for using NOAH-MP inclead of NOAH LSM.

Okay, then the CCPP documentation here is wrong, which says lsm=2 is RUC

To set up options to run the forecast model with uGWD.v1, lease refer to https://github.com/yangfanglin/global-workflow/blob/feature/gfsv17/parm/config/config.fcst#L69 and https://github.com/yangfanglin/global-workflow/blob/feature/gfsv17/scripts/exglobal_forecast.sh#L403 for reference
Please note a different SDF has to be used if one decides to run with the uGWD.v0. It is difficult and confusing to include all the options in the workflow to run with both uGWD.v0 and uGWD.v1. I'd suggest you only make the uGWD.v1 options available. I have discussed this matter with the developer(s). The model code will be cleaned up in the near future to only support uGWD.v1 options. -- Fanglin

What's a SDF?

When you say only support v1 options, does that include removing gwd_opt=1? If not, what other options do I need to set for consistency?

@yangfanglin
Copy link
Contributor

yangfanglin commented Jul 21, 2021 via email

@WalterKolczynski-NOAA
Copy link
Contributor

Okay, we current support different suites. What is going to happen if I run this with FV3_GFS_v16 (say, for aerosols) if I rip out all of the non-ugwd v1 code?

@SMoorthi-emc
Copy link
Contributor

SMoorthi-emc commented Jul 21, 2021 via email

@WalterKolczynski-NOAA
Copy link
Contributor

WalterKolczynski-NOAA commented Jul 22, 2021 •
edited
Loading

I think it is shortsighted to remove options in the model code. As I understood, UFS is not supposed to work for ONE set of physics. (for the record, I have never tried NOahMP nor the new gwd nor RRTMGP). And now all this python and rocoto mess. Do any of you know what the following cryptic error message means? "07/21/21 19:34:16 EDT :: c384_phyae.xml :: sbatch: error: QOSMaxWallDurationPerJobLimit sbatch: error: Batch job submission failed: Job violates accounting/QOS policy (job submit limit, user's size and/or time limits)" I see no such variable. Moorthi

QOSMaxWallDurationPerJobLimit is a scheduler (slurm) variable, not a UFS one. The error means a job is asking for a longer wallclock than the QOS allows. But please keep unrelated troubleshooting out of this issue (especially for things that aren't even global-workflow related).

As for the future of GWD options in UFS, that is also a bit outside of this issue. I just want to make sure the option(s) set by global-workflow are consistent (or at least aren't rejected at runtime).

@yangfanglin
Copy link
Contributor

yangfanglin commented Jul 22, 2021 via email

@yangfanglin
Copy link
Contributor

yangfanglin commented Jul 22, 2021 via email

WalterKolczynski-NOAA added a commit to WalterKolczynski-NOAA/global-workflow that referenced this issue Jul 24, 2021
@WalterKolczynski-NOAA
Update to tiled fix files, UGWD v1, NOAH-MP
3c56751 
Updates model to use tiled fix files. Fix directory is updated to the
fix_NEW location.

Adds the ability to use UGWD v1. Since this capability is tied to the
CCPP suite used, the sutie definition file is grepped to determine whether
UGWD is active. Otherwise, gwd_opt 1 is used. Either way, the appropriate
namelist settings are added to input.nml. For v1, the necessary fix files
are also linked to the run directory. If additional options are supported
in the future, there will need to be more sophisicated parsing.

Adds the ability to use Noah-MP. Like UGWD, this is dictated by the CCPP
suite used, so the suite definition file is grepped to determine whether
to use Noah (lsm=1) or Noah-MP (lsm=2).

Additional CCPP suites are added to allow for the new options.

The two non-fractional coupled cases are updated to use a CCPP suite using
both UGWD v1 and Noah-MP. The two aerosol cases are updated to a suite using
UGWD v1 (there does not appear to be an atm-only suite that has both).

There is also a minor change to the UFS build script to remove any existing
UFS build directory. This prevents problems when attempting to build a
different app after one has already been built.

Closes: NOAA-EMC#331, NOAA-EMC#346
Merged
@WalterKolczynski-NOAA WalterKolczynski-NOAA linked a pull request Jul 24, 2021 that will close this issue
Update to tiled fix files, UGWD v1, NOAH-MP #373
Merged
WalterKolczynski-NOAA added a commit to WalterKolczynski-NOAA/global-workflow that referenced this issue Jul 27, 2021
@WalterKolczynski-NOAA
Ensure CCPP_SUITE is set and suite file exists
a079e38 
Previously there was no guarantee that CCPP_SUITE was set before it
was used in forecast_postdet.sh. The suite how has a default value if
it is not set. A default later in the execution chain is removed, as
CCPP_SUITE is now guaranteed to be set beforehand. The default is also
updated to FV3_GFS_v16.

There is also now a check to ensure the suite file is present inside
the UFS directory before trying to determine settings from it.

Refs: NOAA-EMC#346
WalterKolczynski-NOAA added a commit that referenced this issue Jul 27, 2021
@WalterKolczynski-NOAA
Update to tiled fix files, UGWD v1, NOAH-MP (#373)
dbc28c4 
* Update to tiled fix files, UGWD v1, NOAH-MP

Updates model to use tiled fix files. Fix directory is updated to the
fix_NEW location.

Adds the ability to use UGWD v1. Since this capability is tied to the
CCPP suite used, the sutie definition file is grepped to determine whether
UGWD is active. Otherwise, gwd_opt 1 is used. Either way, the appropriate
namelist settings are added to input.nml. For v1, the necessary fix files
are also linked to the run directory. If additional options are supported
in the future, there will need to be more sophisicated parsing.

Adds the ability to use Noah-MP. Like UGWD, this is dictated by the CCPP
suite used, so the suite definition file is grepped to determine whether
to use Noah (lsm=1) or Noah-MP (lsm=2).

Additional CCPP suites are added to allow for the new options.

The two non-fractional coupled cases are updated to use a CCPP suite using
both UGWD v1 and Noah-MP. The two aerosol cases are updated to a suite using
UGWD v1 (there does not appear to be an atm-only suite that has both).

There is also a minor change to the UFS build script to remove any existing
UFS build directory. This prevents problems when attempting to build a
different app after one has already been built.

Closes: #331, #346
WalterKolczynski-NOAA added a commit that referenced this issue Jul 28, 2021
@WalterKolczynski-NOAA
Update to MERRA2 aerosol climatology (#380)
ef419a6 
* Update to tiled fix files, UGWD v1, NOAH-MP

Updates model to use tiled fix files. Fix directory is updated to the
fix_NEW location.

Adds the ability to use UGWD v1. Since this capability is tied to the
CCPP suite used, the sutie definition file is grepped to determine whether
UGWD is active. Otherwise, gwd_opt 1 is used. Either way, the appropriate
namelist settings are added to input.nml. For v1, the necessary fix files
are also linked to the run directory. If additional options are supported
in the future, there will need to be more sophisicated parsing.

Adds the ability to use Noah-MP. Like UGWD, this is dictated by the CCPP
suite used, so the suite definition file is grepped to determine whether
to use Noah (lsm=1) or Noah-MP (lsm=2).

Additional CCPP suites are added to allow for the new options.

The two non-fractional coupled cases are updated to use a CCPP suite using
both UGWD v1 and Noah-MP. The two aerosol cases are updated to a suite using
UGWD v1 (there does not appear to be an atm-only suite that has both).

There is also a minor change to the UFS build script to remove any existing
UFS build directory. This prevents problems when attempting to build a
different app after one has already been built.

Closes: #331, #346

* Ensure CCPP_SUITE is set and suite file exists

Previously there was no guarantee that CCPP_SUITE was set before it
was used in forecast_postdet.sh. The suite how has a default value if
it is not set. A default later in the execution chain is removed, as
CCPP_SUITE is now guaranteed to be set beforehand. The default is also
updated to FV3_GFS_v16.

There is also now a check to ensure the suite file is present inside
the UFS directory before trying to determine settings from it.

Refs: #346

* Update to MERRA2 aerosol climatology

Adds the capability to use MERRA2 aerosol climatology and makes it
the default.

An addition to the diag table is required. Rather than continue the
proliferation of diag tables to produce one specifically for coupled
and MERRA2, the existing diag table from the MERRA2 update in devleop
is pared down to just the additional fields introduced. This is then
appended to the main diag table if necessary.

Also corrected a related issue in the CROW forecast config. While
IAER and several other similar settings were already present in the
schema, the values were hard-coded in the forecast config instead of
using the value set. The script now correctly uses the values set by
the configuration system. The default for IAER is changed to 1011 for
MERRA2.

Refs: #379
lgannoaa pushed a commit to lgannoaa/global-workflow that referenced this issue Feb 9, 2022
@hsinmulin-NOAA
Code and parm modification for the RRFS-CMAQ chemistry calciulation (N…
6be6119 
…OAA-EMC#346)

* New modification for the RRFS-CMAQ PMTF (PM2.5)

New modification for the RRFS-CMAQ

* Modification to include necessary variables for the RRFS-CMAQ

Modification to include necessary variables for the RRFS-CMAQ

* Delete fv3lam_cmaq_post_avblflds.xml-org

* Delete fv3lam_cmaq.xml-org

* Delete fv3lam_cmaq.xml-pmtf_only

* Delete fv3lam_cmaq_post_avblflds.xml-new

* Delete makefile-cmaq

* Delete makefile-org

* Delete ALLOCATE_ALL.f-org

* Delete ALLOCATE_ALL.f-pmtf

* Delete DEALLOCATE.f-org

* Delete DEALLOCATE.f-pmtf

* Delete INITPOST_NETCDF.f-org

* Delete INITPOST_NETCDF.f-pmtf

* Delete MDLFLD.f-org

* Delete MDLFLD.f-pmtf

* Delete MDLFLD.f-pmtf-unit

* Delete RQSTFLD.F-org

* Delete RQSTFLD.F-pmtf

* Delete VRBLS3D_mod.f-org

* Delete VRBLS3D_mod.f-pmtf

* unified "post_avblflds.xml" related change

Update files according to the removal of "fv3lam_cmaq_post_avblflds.xml" to a unified "post_avblflds.xml"

* Delete fv3lam_cmaq_post_avblflds.xml

to an unified "post_avblflds.xml"

* remove variables do not in the GRIB2 TABLE

Update the file for variables used in the output and defined in GRIB2 TABLE

* Update codes to consistent with changes in parm

Remove variables that will not be used in UPP output.

* Remove O3MR

Remove O3MR that has been processed in the original code
The removal of O3MR in both codes also eliminate the duplicated "O3MR" in UPP output

* Change the allocation 

Add "aqfcmaq_on" as the itag for the conditional allocation for the input variables.  Only variables in the output are set in the main allocation, the rest are set as local in the "INITPOST_NETCDF" and will only be used when "aqfcmaq_on=.true."

* Initialize the AQF related variables to be "0." instead of "spval"

The purpose for the initialization is for run without itag "aqfcmaq_on=.true." (default is "aqfcmaq_on=.false.").  The value of PMTF & OZCON shown in the output will be "0" instead of misled value related to the calculation from "spval".  O3MR is the tracer also in FV3GFS, the process is outside the "aqfcmaq_on" itag control.

* renew file

Reload the file to eliminate the "block & spacce" (-b) difference from the code diff

* Further limit the allociation of AQF UPP output

Further limit the allocation of AQF UPP output variables only exist when itag "aqfcmaq_on=.true." (default .false.).  With itag "aqfcmaq=.true." , the UPP output will contains the allocated variables (currently, OZCON, PMTF).  Without the itag (default .false.), the UPP output will return to original non-CMAQ version (no OZCON & PMTF)

* Remove O3MR

Remove O3mr that has been coded in the original code as "O3" from related codes

* change makefile 

comment out the RRFS-CMAQ related process currently for it is not in the RRFS prallel run

* New NT file for chemistry only

file contains only chemical species that have been added to the source code "initpost_netcdf".

* Revert makefile to original

exclude the generation of RRFS-CMAQ control file related changes until parallel setup.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@WalterKolczynski-NOAA WalterKolczynski-NOAA

Labels
None yet
Projects
None yet
Milestone
Prototype 7
Development

Successfully merging a pull request may close this issue.

Update to tiled fix files, UGWD v1, NOAH-MP WalterKolczynski-NOAA/global-workflow
5 participants
@JessicaMeixner-NOAA @WalterKolczynski-NOAA @KateFriedman-NOAA @yangfanglin @SMoorthi-emc