Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add fire blending to GOCART #2883

Open
wants to merge 91 commits into
base: develop
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
91 commits
Select commit Hold shift + click to select a range
c8e6344
add bones to the fire blending in the prep_emissions job
bbakernoaa Sep 13, 2024
4d81cb8
pycodestyle
bbakernoaa Sep 13, 2024
5f86abd
pycodestyle fixes
bbakernoaa Sep 13, 2024
62dbf31
fix prep_emission jjob
bbakernoaa Sep 13, 2024
ae3f599
Merge branch 'develop' into feature/fire_blending
bbakernoaa Sep 13, 2024
16e21bd
Merge branch 'develop' into feature/fire_blending
aerorahul Sep 14, 2024
b22d7bc
Cleanup job for GEFS (#2919)
AntonMFernando-NOAA Sep 14, 2024
9965857
Update config.resources for bufr sounding job postsnd (#2917)
BoCui-NOAA Sep 16, 2024
5b57604
Update global atmos upp job to use COMIN/COMOUT (#2867)
mingshichen-noaa Sep 16, 2024
1f95b2b
Update to obsproc/v1.2.0 and prepobs/v1.1.0 (#2903)
KateFriedman-NOAA Sep 18, 2024
eb8bc78
Merge remote-tracking branch 'remotes/upstream/develop' into feature/…
bbakernoaa Sep 19, 2024
9a9a874
Merge branch 'develop' into feature/fire_blending
aerorahul Sep 25, 2024
3b87649
Update parm/prep/aero_emissions.yaml
bbakernoaa Oct 7, 2024
21d7bad
Update parm/prep/aero_emissions.yaml
bbakernoaa Oct 7, 2024
58e7280
Update parm/prep/aero_emissions.yaml
bbakernoaa Oct 7, 2024
f74f471
Update parm/prep/aero_emissions.yaml
bbakernoaa Oct 7, 2024
945508f
Update ush/python/pygfs/task/aero_emissions.py
bbakernoaa Oct 15, 2024
eb53925
add coarsen_scale to yaml and use in aero_emission.py
bbakernoaa Oct 15, 2024
55d7ada
remove debugging print statements
bbakernoaa Oct 15, 2024
4aec698
Update parm/prep/aero_emissions.yaml
bbakernoaa Oct 15, 2024
2d8155d
add init doc block
bbakernoaa Oct 15, 2024
389f9da
remove comment
bbakernoaa Oct 15, 2024
2875b6e
move syncing to initialize
bbakernoaa Oct 15, 2024
404131d
Merge remote-tracking branch 'refs/remotes/origin/feature/fire_blendi…
bbakernoaa Oct 15, 2024
ca87ed9
add loger info for this and error trapping
bbakernoaa Oct 15, 2024
9ff202f
add more changes
bbakernoaa Oct 15, 2024
55d5892
more changes, docstrings, error checking etc
bbakernoaa Oct 16, 2024
f853e4f
pycodestyle changes
bbakernoaa Oct 16, 2024
4dfba87
add blended emission option
bbakernoaa Oct 16, 2024
dc5c8bc
copy blending from COMOUT if AERO_EMIS_FIRE == blending
bbakernoaa Oct 16, 2024
dbc63e8
Update ush/forecast_postdet.sh
bbakernoaa Oct 17, 2024
a43e0c6
Update ush/forecast_postdet.sh
bbakernoaa Oct 17, 2024
6836bfc
Update ush/python/pygfs/task/aero_emissions.py
bbakernoaa Oct 17, 2024
4e30517
Update ush/python/pygfs/task/aero_emissions.py
bbakernoaa Oct 17, 2024
d0b20b4
changing resources
bbakernoaa Oct 17, 2024
1fa7df1
remove starthour
bbakernoaa Oct 17, 2024
d1dad35
Merge branch 'develop' into feature/fire_blending
bbakernoaa Oct 17, 2024
4a696b3
fix pycodestyle
bbakernoaa Oct 17, 2024
8d9a124
complete passing vars from yaml
bbakernoaa Oct 17, 2024
42da2da
pycodestyle again
bbakernoaa Oct 17, 2024
9753ddc
fix passing matching vars arrays from yaml file
bbakernoaa Oct 17, 2024
84edf69
more changes to fix issues
bbakernoaa Oct 17, 2024
9583817
Update ush/python/pygfs/task/aero_emissions.py
bbakernoaa Oct 17, 2024
ec4f462
Update ush/python/pygfs/task/aero_emissions.py
bbakernoaa Oct 17, 2024
65572fa
Update ush/python/pygfs/task/aero_emissions.py
bbakernoaa Oct 17, 2024
ffb8664
add changes to copy the data for each fire case and add a method that…
bbakernoaa Oct 17, 2024
7a1efd1
Update ush/python/pygfs/task/aero_emissions.py
bbakernoaa Oct 17, 2024
b628f29
last suggetions?
bbakernoaa Oct 17, 2024
3c170db
Merge branch 'develop' into feature/fire_blending
bbakernoaa Oct 17, 2024
d44c5b9
add with command when opening files
bbakernoaa Oct 18, 2024
5d2f493
Merge remote-tracking branch 'refs/remotes/origin/feature/fire_blendi…
bbakernoaa Oct 18, 2024
f204898
Update scripts/exglobal_prep_emissions.py
bbakernoaa Oct 18, 2024
e7bf48a
Update ush/python/pygfs/task/aero_emissions.py
bbakernoaa Oct 18, 2024
eab4554
Update ush/python/pygfs/task/aero_emissions.py
bbakernoaa Oct 18, 2024
71a0547
Merge branch 'develop' into feature/fire_blending
bbakernoaa Oct 22, 2024
d6ce0ab
Update aero_emissions.py
bbakernoaa Oct 22, 2024
313cb62
Update ush/python/pygfs/task/aero_emissions.py
bbakernoaa Oct 30, 2024
438c421
Update ush/python/pygfs/task/aero_emissions.py
bbakernoaa Oct 30, 2024
8a471fa
Update ush/python/pygfs/task/aero_emissions.py
bbakernoaa Oct 30, 2024
1df78dd
Merge branch 'develop' into feature/fire_blending
bbakernoaa Oct 31, 2024
4e660a5
addressing many issues
bbakernoaa Oct 31, 2024
f4a31bd
Adding n_persist to the yaml file and populating that through
bbakernoaa Oct 31, 2024
c379b5c
Merge branch 'develop' into feature/fire_blending
bbakernoaa Oct 31, 2024
82a2805
Merge branch 'develop' into feature/fire_blending
bbakernoaa Nov 4, 2024
6d12fb3
Update ush/python/pygfs/task/aero_emissions.py
bbakernoaa Nov 12, 2024
934c95f
Update ush/python/pygfs/task/aero_emissions.py
bbakernoaa Nov 12, 2024
46838fd
Update ush/python/pygfs/task/aero_emissions.py
bbakernoaa Nov 12, 2024
d6e8fc1
Update ush/python/pygfs/task/aero_emissions.py
bbakernoaa Nov 12, 2024
5039957
Update ush/python/pygfs/task/aero_emissions.py
bbakernoaa Nov 12, 2024
5124844
Use current_cycle
zmoon Nov 21, 2024
9b86ea7
Seems like this should be underscore
zmoon Nov 21, 2024
5cdd6a6
`config_dict`
zmoon Nov 21, 2024
4ced2d2
Break long line
zmoon Nov 21, 2024
d8b6d29
Just use the args
zmoon Nov 21, 2024
511629c
Add missing inits
zmoon Nov 21, 2024
6ecf5c6
Remove some unused imports and sort
zmoon Nov 21, 2024
98ab6e6
Address unused vars
zmoon Nov 21, 2024
7aae5e9
Seems like these are really assuming lists of str
zmoon Nov 21, 2024
1b56d63
Rename not needed
zmoon Nov 21, 2024
7ea3b68
Replace input/output var lists with mapping
zmoon Nov 21, 2024
5ce4047
Some spelling/typing/docstring fixes
zmoon Nov 21, 2024
0670aae
Merge remote-tracking branch 'origin/develop' into feature/fire_blending
zmoon Nov 21, 2024
8b3718f
Does nothing
zmoon Nov 25, 2024
3227263
Log before raise
zmoon Nov 25, 2024
991b348
Fix HFED guard in init
zmoon Nov 26, 2024
ca3bc1a
Updates following convo with Barry
zmoon Nov 26, 2024
7df5b5e
Dict of files from yaml
zmoon Nov 26, 2024
bc9e8f9
Use new files dict
zmoon Nov 26, 2024
d047882
FATAL ERROR
zmoon Dec 3, 2024
8fabb61
Merge remote-tracking branch 'origin/develop' into feature/fire_blending
zmoon Dec 3, 2024
46fdc0b
`f` -> `filepath`
zmoon Dec 3, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion ci/scripts/utils/publish_logs.py
Original file line number Diff line number Diff line change
Expand Up @@ -91,7 +91,7 @@ def upload_logs_to_repo(args, emcbot_gh, emcbot_ci_url):
file_path_in_repo = f"{repo_path}/{path_header}/" + str(os.path.basename(file.name))
emcbot_gh.repo.create_file(file_path_in_repo, "Adding error log file", file_content, branch="error_logs")

file_url = f"{emcbot_ci_url.rsplit('.',1)[0]}/tree/{repo_branch}/{repo_path}/{path_header}"
file_url = f"{emcbot_ci_url.rsplit('.', 1)[0]}/tree/{repo_branch}/{repo_path}/{path_header}"
print(file_url)


Expand Down
8 changes: 8 additions & 0 deletions jobs/JGLOBAL_PREP_EMISSIONS
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,8 @@ source "${HOMEgfs}/ush/jjob_header.sh" -e "prep_emissions" -c "base prep_emissio
##############################################
# Generate COM variables from templates
# TODO: Add necessary COMIN, COMOUT variables for this job
YMD="${PDY}" HH="${cyc}" declare_from_tmpl -rx \
COMOUT_CHEM_HISTORY:COM_CHEM_HISTORY_TMPL

###############################################################
# Run relevant script
Expand All @@ -32,4 +34,10 @@ if [[ -e "${pgmout}" ]] ; then
cat "${pgmout}"
fi

##########################################
# Remove the Temporary working directory
##########################################
cd "${DATAROOT}" || exit 1
[[ ${KEEPDATA} = "NO" ]] && rm -rf "${DATA}"

exit 0
27 changes: 27 additions & 0 deletions parm/config/gefs/config.prep_emissions
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,37 @@

########## config.prep_emissions ##########
# aerosol emissions preprocessing specific
# TODO: this is duplicated in the config.aero file. Should use a common function
case ${machine} in
"HERA")
AERO_INPUTS_DIR="/scratch1/NCEPDEV/global/glopara/data/gocart_emissions"
;;
"ORION" | "HERCULES")
AERO_INPUTS_DIR="/work2/noaa/global/wkolczyn/noscrub/global-workflow/gocart_emissions"
;;
"S4")
AERO_INPUTS_DIR="/data/prod/glopara/gocart_emissions"
;;
"WCOSS2")
AERO_INPUTS_DIR="/lfs/h2/emc/global/noscrub/emc.global/data/gocart_emissions"
;;
"GAEA")
AERO_INPUTS_DIR="/gpfs/f5/epic/proj-shared/global/glopara/data/gocart_emissions"
;;
"JET")
AERO_INPUTS_DIR="/lfs4/HFIP/hfv3gfs/glopara/data/gocart_emissions"
;;
*)
echo "FATAL ERROR: Machine ${machine} unsupported for aerosols"
exit 2
;;
esac
export AERO_INPUTS_DIR
bbakernoaa marked this conversation as resolved.
Show resolved Hide resolved

echo "BEGIN: config.prep_emissions"

# Get task specific resources
export PREP_EMISSION_CONFIG="${PARMgfs}/prep/aero_emissions.yaml"
source "${EXPDIR}/config.resources" prep_emissions

echo "END: config.prep_emissions"
2 changes: 1 addition & 1 deletion parm/config/gefs/config.resources
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,7 @@ case ${step} in
export ntasks=1
export threads_per_task=1
export tasks_per_node=$(( max_tasks_per_node / threads_per_task ))
export memory="1GB"
export memory="20GB"
;;

"fcst" | "efcs")
Expand Down
2 changes: 1 addition & 1 deletion parm/config/gfs/config.aero
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ export AERO_INPUTS_DIR=@AERO_INPUTS_DIR@

export AERO_DIAG_TABLE="${PARMgfs}/ufs/fv3/diag_table.aero"
export AERO_FIELD_TABLE="${PARMgfs}/ufs/fv3/field_table.aero"
# Biomass burning emission dataset. Choose from: gbbepx, qfed, none
# Biomass burning emission dataset. Choose from: gbbepx, qfed, hfed, blended, none
export AERO_EMIS_FIRE="qfed"
# Directory containing GOCART configuration files
export AERO_CONFIG_DIR="${PARMgfs}/ufs/gocart"
Expand Down
2 changes: 1 addition & 1 deletion parm/config/gfs/config.resources
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ if (( $# != 1 )); then

echo "Must specify an input task argument to set resource variables!"
echo "argument can be any one of the following:"
echo "stage_ic aerosol_init"
echo "stage_ic aerosol_init prep_emissions"
echo "prep prepatmiodaobs"
echo "atmanlinit atmanlvar atmanlfv3inc atmanlfinal"
echo "atmensanlinit atmensanlobs atmensanlsol atmensanlletkf atmensanlfv3inc atmensanlfinal"
Expand Down
48 changes: 48 additions & 0 deletions parm/prep/aero_emissions.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
aero_emissions:
config:
debug: False
ratio: 0.95 # weighting ratio for obs
emistype: 'QFED' # emission type {'QFED', 'GBBEPx', 'HFED'}
climfile_str: 'GBBEPx-all01GRID_v4r0_climMean' # climate file base string used for glob later
bbakernoaa marked this conversation as resolved.
Show resolved Hide resolved
gbbepx_version: 'v4r0' # GBBEPx version
qfed_version: '006' # QFED version
species: ['so2', 'oc', 'bc'] # species to be used
historical: False # set to true to just use true data for the given day
coarsen_scale: 150 # scale for coarsen function to generate weights
output_var_map: # for GBBEPx
'bc': 'BC'
'ch4': 'CH4'
'co': 'CO'
'co2': 'CO2'
'nh3': 'NH3'
'no': 'NOx'
'oc': 'OC'
'pm25': 'PM2.5'
'so2': 'SO2'
n_persist: 5
data_in:
mkdir:
- "{{ DATA }}"
- "{{ COMOUT_CHEM_HISTORY}}"
qfed: # copy qfed
copy:
- ["{{ AERO_INPUTS_DIR }}/nexus/QFED/{{ current_cycle | strftime('%Y/%m') }}/qfed2.emis_oc.006.{{ current_cycle | to_YMD }}.nc4", "{{ DATA }}/"]
- ["{{ AERO_INPUTS_DIR }}/nexus/QFED/{{ current_cycle | strftime('%Y/%m') }}/qfed2.emis_so2.006.{{ current_cycle | to_YMD }}.nc4", "{{ DATA }}/"]
- ["{{ AERO_INPUTS_DIR }}/nexus/QFED/{{ current_cycle | strftime('%Y/%m') }}/qfed2.emis_bc.006.{{ current_cycle | to_YMD }}.nc4", "{{ DATA }}/"]
hfed: # copy hfed
copy:
- ["{{ AERO_INPUTS_DIR }}/nexus/HFED/{{ current_cycle | strftime('%Y/M%m') }}/hfed.emis_oc.x576_y361.{{ current_cycle | strftime('%Y%j') }}.nc4", "{{ DATA }}/"]
- ["{{ AERO_INPUTS_DIR }}/nexus/HFED/{{ current_cycle | strftime('%Y/M%m') }}/hfed.emis_bc.x576_y361.{{ current_cycle | strftime('%Y%j') }}.nc4", "{{ DATA }}/"]
- ["{{ AERO_INPUTS_DIR }}/nexus/HFED/{{ current_cycle | strftime('%Y/M%m') }}/hfed.emis_so2.x576_y361.{{ current_cycle | strftime('%Y%j') }}.nc4", "{{ DATA }}/"]
gbbepx:
copy:
- ["{{ AERO_INPUTS_DIR }}/nexus/GBBEPx/v4/GBBEPx_all01GRID_v4r0_climMean_{{ current_cycle | strftime('%m%d') }}.nc", "{{ DATA }}/"] # copy climo files
climo: # copy climatology
copy:
- ["{{ AERO_INPUTS_DIR }}/nexus/GBBEPx/v4/climMean/GBBEPx-all01GRID_v4r0_climMean_{{ current_cycle | strftime('%m%d') }}.nc", "{{ DATA }}/"] # copy climo files
{% for fdate in forecast_dates %}
- ["{{ AERO_INPUTS_DIR }}/nexus/GBBEPx/v4/climMean/GBBEPx-all01GRID_v4r0_climMean_{{ fdate | strftime('%m%d') }}.nc", "{{ DATA }}/"] # copy climo files
{% endfor %}
data_out:
copy:
- ["{{ DATA }}/{{ RUN }}_blended_emissions.{{ current_cycle | to_YMD }}.nc", "{{ COMOUT_CHEM_HISTORY }}/{{ RUN }}.{{ current_cycle | to_YMD }}.{{ RUN }}_blended_emissions.nc"]
8 changes: 8 additions & 0 deletions parm/ufs/gocart/ExtData.blended
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
#====== BIOMASS BURNING EMISSIONS =======================================

# Blended Emissions
#--------------------------------------------------------------------------------------------------------------------------------
SU_BIOMASS NA N Y %y4-%m2-%d2t12:00:00 none 0.7778 SO2 gefs.%y4%m2%d2.gefs_blended_emissions.nc
OC_BIOMASS NA N Y %y4-%m2-%d2t12:00:00 none 0.7778 OC gefs.%y4%m2%d2.gefs_blended_emissions.nc
BC_BIOMASS NA N Y %y4-%m2-%d2t12:00:00 none 0.7778 BC gefs.%y4%m2%d2.gefs_blended_emissions.nc
# EMI_NH3_BB NA N Y %y4-%m2-%d2t12:00:00 none 0.7778 NH3 gefs.%y4%m2%d2.gefs_blended_emissions.nc
8 changes: 8 additions & 0 deletions parm/ufs/gocart/ExtData.hfed
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
#====== BIOMASS BURNING EMISSIONS =======================================

# HFED
#--------------------------------------------------------------------------------------------------------------------------------
SU_BIOMASS NA N Y %y4-%m2-%d2t12:00:00 none 0.7778 biomass ExtData/nexus/HFED/Y1994/M%m2/hfed.emis_so2.x576_y361.%y4%m2.nc4
OC_BIOMASS NA N Y %y4-%m2-%d2t12:00:00 none 0.7778 biomass ExtData/nexus/HFED/Y1994/M%m2/hfed.emis_oc.x576_y361.%y4%m2.nc4
BC_BIOMASS NA N Y %y4-%m2-%d2t12:00:00 none 0.7778 biomass ExtData/nexus/HFED/Y1994/M%m2/hfed.emis_bc.x576_y361.%y4%m2.nc4
# EMI_NH3_BB NA N Y %y4-%m2-%d2t12:00:00 none 0.7778 biomass ExtData/nexus/HFED/Y1994/M%m2/hfed.emis_nh3.x576_y361.%y4%m2.nc4
aerorahul marked this conversation as resolved.
Show resolved Hide resolved
22 changes: 16 additions & 6 deletions scripts/exglobal_prep_emissions.py
Original file line number Diff line number Diff line change
@@ -1,10 +1,20 @@
#!/usr/bin/env python3
# exglobal_prep_emissions.py
# This script creates a emissions object
# which perform the pre-processing for aerosol emissions
"""
This script initializes a logger, reads configuration from the environment, and performs emissions pre-processing tasks using the AerosolEmissions class.

The script does the following:
1. Initializes a root logger with the specified logging level and colored log output.
2. Reads configuration from the environment and converts it into a Python dictionary.
3. Instantiates an AerosolEmissions object with the configuration.
4. Retrieves specific keys from the emissions task configuration and stores them in a dictionary.
5. Sets the 'emistype' attribute in the configuration dictionary based on the 'emistype' value in the emissions configuration.
6. Initializes, configures, runs, and finalizes the emissions task using the provided parameters.

Note: Make sure to have the necessary dependencies (wxflow, pygfs) installed to run this script successfully.
"""
import os

from wxflow import Logger, cast_strdict_as_dtypedict
from wxflow import Logger, AttrDict, cast_strdict_as_dtypedict
from pygfs import AerosolEmissions


Expand All @@ -19,7 +29,7 @@

# Instantiate the emissions pre-processing task
emissions = AerosolEmissions(config)

emissions.initialize()
emissions.configure()
emissions.execute(emissions.task_config.DATA, emissions.task_config.APRUN)
emissions.run()
emissions.finalize()
2 changes: 1 addition & 1 deletion scripts/exglobal_stage_ic.py
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ def main():
for key in keys:
# Make sure OCNRES is three digits
if key == "OCNRES":
stage.task_config.OCNRES = f"{stage.task_config.OCNRES :03d}"
stage.task_config.OCNRES = f"{stage.task_config.OCNRES:03d}"
stage_dict[key] = stage.task_config[key]

# Also import all COM* directory and template variables
Expand Down
2 changes: 1 addition & 1 deletion ush/compare_f90nml.py
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,7 @@ def _print_diffs(diff_dict: Dict) -> None:
for kk in diff_dict[path].keys():
items = diff_dict[path][kk]
print(
f"{kk:>{max_len+2}} : {' | '.join(map(str, diff_dict[path][kk]))}")
f"{kk:>{max_len + 2}} : {' | '.join(map(str, diff_dict[path][kk]))}")
aerorahul marked this conversation as resolved.
Show resolved Hide resolved

_print_diffs(result)

Expand Down
5 changes: 5 additions & 0 deletions ush/forecast_postdet.sh
Original file line number Diff line number Diff line change
Expand Up @@ -683,6 +683,11 @@ GOCART_rc() {
[[ ${status} -ne 0 ]] && exit "${status}"
fi

# Link blended emissions if AERO_EMIS_FIRE is 'blended'
if [[ "${AERO_EMIS_FIRE}" == "blended" && "${RUN}" == "gefs" ]]; then
${NCP} "${COMOUT_CHEM_HISTORY}/${RUN}.${current_cycle:0:8}.${RUN}_blended_emissions.nc" "${DATA}"
fi

source "${USHgfs}/parsing_namelists_GOCART.sh"
GOCART_namelists
}
Expand Down
Loading