Skip to content

Commit

Permalink
Marine obs reorganization and more ... (#876)
Browse files Browse the repository at this point in the history
Things done:
- Assumes that the obs will be organized like something like this from
now on:
```console
└── gdas.20180415
    └── 12
        └── ocean
            ├── adt
            │   ├── rads_adt_3a_2018105.nc
            │   └── rads_adt_3b_2018105.nc
            ├── icec
            │   ├── AMSR2-SEAICE-NH_v2r2_GW1_s201804150835180_e201804151014170_c201804151052280.nc
            │   ├── AMSR2-SEAICE-NH_v2r2_GW1_s201804151014190_e201804151150170_c201804151309570.nc
            │   ├── AMSR2-SEAICE-SH_v2r2_GW1_s201804150835180_e201804151014170_c201804151052280.nc
            │   └── AMSR2-SEAICE-SH_v2r2_GW1_s201804151014190_e201804151150170_c201804151309570.nc
            ├── sss
            └── sst
                ├── 20180415095000-STAR-L3U_GHRSST-SSTsubskin-AVHRRF_MB-ACSPO_V2.80-v02.0-fv01.0.nc
                ├── 20180415114000-STAR-L3U_GHRSST-SSTsubskin-AVHRRF_MB-ACSPO_V2.80-v02.0-fv01.0.nc
                └── 20180415132000-STAR-L3U_GHRSST-SSTsubskin-AVHRRF_MB-ACSPO_V2.80-v02.0-fv01.0.nc
```
- Use of the `*` wildcard instead of `?` to simplify the obs file
selection
- Addition of binning for the sst retrieval
- The `soca` # needs to be more recent.
- This wasn't planned as part of this work, but the bump localization
was having issues, so I replaced it with `diffusion`

### Issues addressed
- fixes #869 
- fixes #874
  • Loading branch information
guillaumevernieres authored Jan 26, 2024
1 parent 9ba6bb9 commit 5549447
Show file tree
Hide file tree
Showing 16 changed files with 53 additions and 92 deletions.
1 change: 1 addition & 0 deletions .gitmodules
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,7 @@
[submodule "sorc/crtm"]
path = sorc/crtm
url = https://github.com/jcsda/crtm.git
branch = release/crtm_jedi_v2.4.1
[submodule "sorc/fms"]
path = sorc/fms
url = https://github.com/jcsda/fms.git
Expand Down
25 changes: 17 additions & 8 deletions parm/soca/berror/saber_blocks.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -74,15 +74,24 @@ components:
localization:
localization method: SABER
saber central block:
saber block name: BUMP_NICAS
saber block name: EXPLICIT_DIFFUSION
active variables: [tocn, socn, ssh]
geometry:
mom6_input_nml: mom_input.nml
fields metadata: ./fields_metadata.yaml
group mapping:
- name: ocean
variables: [tocn, socn, socn, ssh]
- name: ice
variables: [cicen]
read:
io:
data directory: bump
drivers:
multivariate strategy: duplicated
read local nicas: true
model:
do not cross mask boundaries: false
groups:
- name: ocean
horizontal:
filename: hz_ocean.nc
- name: ice
horizontal:
filename: hz_ice.nc

weight:
read_from_file: 3
Expand Down
2 changes: 1 addition & 1 deletion parm/soca/obs/obs_list.yaml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
observers:
- !INC ${OBS_YAML_DIR}/adt_rads_all.yaml
- !INC ${OBS_YAML_DIR}/sst_metopb_l3u.yaml
#- !INC ${OBS_YAML_DIR}/sst_metopb_l3u.yaml
- !INC ${OBS_YAML_DIR}/icec_amsr2_north.yaml
- !INC ${OBS_YAML_DIR}/icec_amsr2_south.yaml
27 changes: 15 additions & 12 deletions parm/soca/obsprep/obsprep_config.yaml
Original file line number Diff line number Diff line change
@@ -1,41 +1,44 @@
observations:
- obs space:
name: sss_smap
dmpdir subdir: SSS
dmpdir regex: SMAP_L2B_SSS_NRT_?????_[AD]_????????T??????.h5
dmpdir subdir: ocean/sss
dmpdir regex: SMAP_L2B_SSS_NRT_*.h5
provider: SMAP
output file: sss_smap.ioda.nc
- obs space:
name: sss_smos
provider: SMOS
dmpdir subdir: SSS
dmpdir subdir: ocean/sss
output file: sss_smos.ioda.nc
dmpdir regex: SM_OPER_MIR_OSUDP2_????????T??????_????????T??????_700_001_1.nc
dmpdir regex: SM_OPER_MIR_OSUDP2_*.nc
- obs space:
name: adt_rads_all
dmpdir subdir: ADT
dmpdir regex: rads_adt_??_???????.nc
dmpdir subdir: ocean/adt
dmpdir regex: rads_adt_*.nc
provider: RADS
output file: adt_rads_all.nc4
- obs space:
name: icec_amsr2_north
provider: AMSR2
dmpdir subdir: icec
dmpdir subdir: ocean/icec
output file: icec_amsr2_north.ioda.nc
dmpdir regex: AMSR2-SEAICE-NH_v2r2_GW1_s???????????????_e???????????????_c???????????????.nc
dmpdir regex: AMSR2-SEAICE-NH_v2r2_GW1_s*.nc
- obs space:
name: icec_amsr2_south
provider: AMSR2
dmpdir subdir: icec
dmpdir subdir: ocean/icec
output file: icec_amsr2_south.ioda.nc
dmpdir regex: AMSR2-SEAICE-SH_v2r2_GW1_s???????????????_e???????????????_c???????????????.nc
dmpdir regex: AMSR2-SEAICE-SH_v2r2_GW1_s*.nc
- obs space:
name: sst_metopb_l3u
provider: GHRSST
dmpdir subdir: 'sst'
dmpdir subdir: 'ocean/sst'
output file: sst_metopb_l3u.ioda.nc
dmpdir regex: '??????????????-STAR-L3U_GHRSST-SSTsubskin-AVHRRF_MB-ACSPO_V2.80-v02.0-fv01.0.nc'
dmpdir regex: '*-L3U_GHRSST-SSTsubskin-AVHRRF_MB-ACSPO_V*.nc'
bounds:
units: C
min: -3.0
max: 50.0
binning:
stride: 5
min number of obs: 10
34 changes: 1 addition & 33 deletions scripts/exgdas_global_marine_analysis_bmat.sh
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
# - generates the DA grid
# - computes diagonal of B based on the background if a std. dev. file
# was not staged. TODO: Remove this option in the future
# - creates the bump correlation operators
# - initialize the loacalization and correlation operator
#
# $Id$
#
Expand Down Expand Up @@ -110,38 +110,6 @@ if [ $err -gt 0 ]; then
exit $err
fi

################################################################################
# Correlation and Localization operators
shopt -s nullglob
files=(./bump/*.nc)
echo $files
if [ ${#files[@]} -gt 0 ]; then
echo "BUMP/NICAS correlation and localization already staged, skipping BUMP initialization"
set +x
if [ $VERBOSE = "YES" ]; then
echo $(date) EXITING $0 with return code $err >&2
fi
exit $err # Exit early, we're done with B
shopt -u nullglob
fi

################################################################################
# Set localization scales for the hybrid en. var.
$APRUN_OCNANAL $JEDI_BIN/soca_setcorscales.x soca_setlocscales.yaml
export err=$?; err_chk
if [ $err -gt 0 ]; then
exit $err
fi

################################################################################
# Compute convolution coefs for L
clean_yaml soca_bump_loc.yaml
$APRUN_OCNANAL $JEDI_BIN/soca_error_covariance_toolbox.x soca_bump_loc.yaml
export err=$?; err_chk
if [ $err -gt 0 ]; then
exit $err
fi

################################################################################
set +x
if [ $VERBOSE = "YES" ]; then
Expand Down
18 changes: 0 additions & 18 deletions scripts/exgdas_global_marine_analysis_prep.py
Original file line number Diff line number Diff line change
Expand Up @@ -401,24 +401,6 @@ def find_clim_ens(input_date):
config = Template.substitute_structure(config, TemplateConstants.DOUBLE_CURLY_BRACES, envconfig.get)
config.save(diffu_vt_yaml)

################################################################################
# generate yaml for bump/nicas (used for correlation and/or localization)

logger.info(f"---------------- generate BUMP/NICAS localization yamls")
# localization bump yaml
bumpdir = 'bump'
ufsda.disk_utils.mkdir(os.path.join(anl_dir, bumpdir))
bump_yaml = os.path.join(anl_dir, 'soca_bump_loc.yaml')
bump_yaml_template = os.path.join(gdas_home,
'parm',
'soca',
'berror',
'soca_bump_loc.yaml')
config = YAMLFile(path=bump_yaml_template)
config = Template.substitute_structure(config, TemplateConstants.DOUBLE_CURLY_BRACES, envconfig.get)
config = Template.substitute_structure(config, TemplateConstants.DOLLAR_PARENTHESES, envconfig.get)
config.save(bump_yaml)

################################################################################
# generate yaml for soca_var

Expand Down
2 changes: 1 addition & 1 deletion sorc/iodaconv
2 changes: 1 addition & 1 deletion sorc/land-imsproc
2 changes: 1 addition & 1 deletion sorc/saber
Submodule saber updated 80 files
+12 −0 src/saber/blocks/SaberBlockParametersBase.h
+6 −4 src/saber/fastlam/CMakeLists.txt
+639 −271 src/saber/fastlam/FastLAM.cc
+22 −6 src/saber/fastlam/FastLAM.h
+4 −1 src/saber/fastlam/FastLAMParametersBase.h
+0 −2,237 src/saber/fastlam/Layer.cc
+0 −198 src/saber/fastlam/Layer.h
+1,056 −0 src/saber/fastlam/LayerBase.cc
+229 −0 src/saber/fastlam/LayerBase.h
+1,008 −0 src/saber/fastlam/LayerHalo.cc
+101 −0 src/saber/fastlam/LayerHalo.h
+1,220 −0 src/saber/fastlam/LayerRC.cc
+116 −0 src/saber/fastlam/LayerRC.h
+0 −38 src/saber/fastlam/Layers.cc
+0 −50 src/saber/fastlam/Layers.h
+42 −8 src/saber/oops/Utilities.cc
+22 −7 src/saber/oops/Utilities.h
+18 −41 src/saber/spectralb/GaussUVToGP.cc
+4 −2 src/saber/spectralb/GaussUVToGP.h
+21 −22 src/saber/spectralb/HydrostaticPressure.cc
+2 −2 src/saber/spectralb/HydrostaticPressure.h
+15 −6 src/saber/vader/AirTemperature.cc
+3 −1 src/saber/vader/AirTemperature.h
+15 −6 src/saber/vader/DryAirDensity.cc
+19 −0 src/saber/vader/DryAirDensity.h
+34 −14 src/saber/vader/GpToHp.cc
+3 −2 src/saber/vader/GpToHp.h
+16 −8 src/saber/vader/HpHexnerToPExner.cc
+22 −2 src/saber/vader/HpHexnerToPExner.h
+4 −4 src/saber/vader/HpToHexner.cc
+20 −6 src/saber/vader/HydroBal.cc
+27 −5 src/saber/vader/HydroBal.h
+20 −11 src/saber/vader/HydrostaticExner.cc
+32 −2 src/saber/vader/HydrostaticExner.h
+29 −8 src/saber/vader/MoistIncrOp.cc
+3 −1 src/saber/vader/MoistIncrOp.h
+72 −4 src/saber/vader/MoistIncrOpParameters.h
+20 −7 src/saber/vader/MoistureControl.cc
+29 −6 src/saber/vader/MoistureControl.h
+67 −0 src/saber/vader/PressureParameters.h
+15 −11 src/saber/vader/SuperMoistIncrOp.cc
+3 −3 src/saber/vader/SuperMoistIncrOp.h
+0 −0 test/testdeps/dirac_fastlam_7.txt
+3 −0 test/testdeps/dirac_fastlam_8.txt
+1 −0 test/testdeps/dirac_fastlam_9.txt
+0 −0 test/testdeps/dirac_spectralb_gauss_vader_4.txt
+0 −0 test/testdeps/randomization_fastlam.txt
+4 −0 test/testinput/compare_diagnostics_outer_vars.yaml
+2 −0 test/testinput/dirac_fastlam_1.yaml
+2 −0 test/testinput/dirac_fastlam_2.yaml
+2 −0 test/testinput/dirac_fastlam_3.yaml
+2 −0 test/testinput/dirac_fastlam_4.yaml
+2 −0 test/testinput/dirac_fastlam_5.yaml
+2 −0 test/testinput/dirac_fastlam_6.yaml
+121 −0 test/testinput/dirac_fastlam_7.yaml
+160 −0 test/testinput/dirac_fastlam_8.yaml
+143 −0 test/testinput/dirac_fastlam_9.yaml
+3 −13 test/testinput/dirac_spectralb_gauss_vader_1.yaml
+0 −10 test/testinput/dirac_spectralb_gauss_vader_2.yaml
+0 −10 test/testinput/dirac_spectralb_gauss_vader_3.yaml
+230 −0 test/testinput/dirac_spectralb_gauss_vader_4.yaml
+42 −0 test/testinput/randomization_fastlam.yaml
+0 −11 test/testinput/randomization_sqrtspectralb_2.yaml
+4 −0 test/testlist/saber_test_tier1-fastlam.txt
+2 −0 test/testlist/saber_test_tier1-spectralb-vader.txt
+2 −0 test/testref/dirac_fastlam_1.ref
+2 −0 test/testref/dirac_fastlam_2.ref
+7 −5 test/testref/dirac_fastlam_3.ref
+7 −5 test/testref/dirac_fastlam_4.ref
+7 −5 test/testref/dirac_fastlam_5.ref
+7 −5 test/testref/dirac_fastlam_6.ref
+75 −0 test/testref/dirac_fastlam_7.ref
+77 −0 test/testref/dirac_fastlam_8.ref
+72 −0 test/testref/dirac_fastlam_9.ref
+12 −31 test/testref/dirac_spectralb_gauss_vader_1.ref
+11 −27 test/testref/dirac_spectralb_gauss_vader_2.ref
+0 −24 test/testref/dirac_spectralb_gauss_vader_3.ref
+106 −0 test/testref/dirac_spectralb_gauss_vader_4.ref
+42 −0 test/testref/randomization_fastlam.ref
+8 −28 test/testref/randomization_sqrtspectralb_2.ref
2 changes: 1 addition & 1 deletion sorc/ufo
Submodule ufo updated 26 files
+2 −39 src/ufo/ObsBias.cc
+91 −2 src/ufo/ObsBiasIncrement.cc
+22 −2 src/ufo/ObsBiasIncrement.h
+4 −1 src/ufo/ObsBiasParameters.h
+1 −1 src/ufo/filters/ProfileAverageObsToModLevels.cc
+6 −0 src/ufo/filters/rttovonedvarcheck/RTTOVOneDVarCheckParameters.h
+10 −4 src/ufo/filters/rttovonedvarcheck/ufo_rttovonedvarcheck_mod.f90
+16 −0 src/ufo/filters/rttovonedvarcheck/ufo_rttovonedvarcheck_obs_mod.f90
+6 −0 src/ufo/filters/rttovonedvarcheck/ufo_rttovonedvarcheck_setup_mod.f90
+2 −0 src/ufo/operators/crtm/crtmParameters/ObsRadianceCRTMParameters.h
+52 −2 src/ufo/operators/crtm/ufo_crtm_utils_mod.F90
+3 −3 src/ufo/operators/rttov/ufo_radiancerttov_mod.F90
+53 −20 src/ufo/operators/rttov/ufo_radiancerttov_tlad_mod.F90
+1 −1 src/ufo/operators/rttov/ufo_radiancerttov_utils_mod.F90
+3 −0 src/ufo/utils/CMakeLists.txt
+52 −0 src/ufo/utils/SaveBiasCoeffs.cc
+28 −0 src/ufo/utils/SaveBiasCoeffs.h
+15 −0 test/mains/TestUfoObsBiasIncrement.cc
+1 −1 test/testinput/unit_tests/filters/atms_rttov_ops_qc_rttovonedvarcheck.yaml
+173 −3 test/testinput/unit_tests/filters/iasi_rttov_ops_qc_rttovonedvarcheck.yaml
+6 −1 test/testinput/unit_tests/filters/qc_profile_average_obs_to_mod_levels.yaml
+10 −0 test/testinput/unit_tests/operators/CMakeLists.txt
+9 −0 test/testinput/unit_tests/operators/atovs_rttov_scatt.yaml
+11 −0 test/testinput/unit_tests/predictors/CMakeLists.txt
+6 −0 test/testinput/unit_tests/predictors/bias_coeff.yaml
+102 −0 test/ufo/ObsBiasIncrement.h
18 changes: 9 additions & 9 deletions test/soca/gw/prepdata.sh
Original file line number Diff line number Diff line change
Expand Up @@ -19,23 +19,23 @@ project_source_dir=$1
cdldir=${project_source_dir}/test/soca/testdata

# GHRSST
cdl2nc4 sst/20180415114000-STAR-L3U_GHRSST-SSTsubskin-AVHRRF_MB-ACSPO_V2.80-v02.0-fv01.0.nc \
cdl2nc4 ocean/sst/20180415114000-STAR-L3U_GHRSST-SSTsubskin-AVHRRF_MB-ACSPO_V2.80-v02.0-fv01.0.nc \
${cdldir}/20180415114000-STAR-L3U_GHRSST-SSTsubskin-AVHRRF_MB-ACSPO_V2.80-v02.0-fv01.0.cdl
cdl2nc4 sst/20180415132000-STAR-L3U_GHRSST-SSTsubskin-AVHRRF_MB-ACSPO_V2.80-v02.0-fv01.0.nc \
cdl2nc4 ocean/sst/20180415132000-STAR-L3U_GHRSST-SSTsubskin-AVHRRF_MB-ACSPO_V2.80-v02.0-fv01.0.nc \
${cdldir}/20180415132000-STAR-L3U_GHRSST-SSTsubskin-AVHRRF_MB-ACSPO_V2.80-v02.0-fv01.0.cdl
cdl2nc4 sst/20180415095000-STAR-L3U_GHRSST-SSTsubskin-AVHRRF_MB-ACSPO_V2.80-v02.0-fv01.0.nc \
cdl2nc4 ocean/sst/20180415095000-STAR-L3U_GHRSST-SSTsubskin-AVHRRF_MB-ACSPO_V2.80-v02.0-fv01.0.nc \
${cdldir}/20180415095000-STAR-L3U_GHRSST-SSTsubskin-AVHRRF_MB-ACSPO_V2.80-v02.0-fv01.0.cdl

# Fake AMSR2 icec
cdl2nc4 icec/AMSR2-SEAICE-NH_v2r2_GW1_s201804150835180_e201804151014170_c201804151052280.nc \
cdl2nc4 ocean/icec/AMSR2-SEAICE-NH_v2r2_GW1_s201804150835180_e201804151014170_c201804151052280.nc \
${project_source_dir}/test/soca/testdata/icec_amsr2_north_1.cdl
cdl2nc4 icec/AMSR2-SEAICE-NH_v2r2_GW1_s201804151014190_e201804151150170_c201804151309570.nc \
cdl2nc4 ocean/icec/AMSR2-SEAICE-NH_v2r2_GW1_s201804151014190_e201804151150170_c201804151309570.nc \
${project_source_dir}/test/soca/testdata/icec_amsr2_north_2.cdl
cdl2nc4 icec/AMSR2-SEAICE-SH_v2r2_GW1_s201804150835180_e201804151014170_c201804151052280.nc \
cdl2nc4 ocean/icec/AMSR2-SEAICE-SH_v2r2_GW1_s201804150835180_e201804151014170_c201804151052280.nc \
${project_source_dir}/test/soca/testdata/icec_amsr2_south_1.cdl
cdl2nc4 icec/AMSR2-SEAICE-SH_v2r2_GW1_s201804151014190_e201804151150170_c201804151309570.nc \
cdl2nc4 ocean/icec/AMSR2-SEAICE-SH_v2r2_GW1_s201804151014190_e201804151150170_c201804151309570.nc \
${project_source_dir}/test/soca/testdata/icec_amsr2_south_2.cdl

# Fake RADS ADT
cdl2nc4 ADT/rads_adt_3a_2018105.nc ${project_source_dir}/test/soca/testdata/rads_adt_3a_2018105.cdl
cdl2nc4 ADT/rads_adt_3b_2018105.nc ${project_source_dir}/test/soca/testdata/rads_adt_3b_2018105.cdl
cdl2nc4 ocean/adt/rads_adt_3a_2018105.nc ${project_source_dir}/test/soca/testdata/rads_adt_3a_2018105.cdl
cdl2nc4 ocean/adt/rads_adt_3b_2018105.nc ${project_source_dir}/test/soca/testdata/rads_adt_3b_2018105.cdl
4 changes: 1 addition & 3 deletions test/soca/gw/setup_obsprep.sh
Original file line number Diff line number Diff line change
Expand Up @@ -11,9 +11,7 @@ mkdir -p ${test_dmpdir}

cd ${test_dmpdir}

mkdir SSS ADT icec sst
mkdir -p ocean/sss ocean/adt ocean/icec ocean/sst

${project_source_dir}/test/soca/gw/prepdata.sh ${project_source_dir}



0 comments on commit 5549447

Please sign in to comment.