Phase 6 Meteorology Model Run and Data Export/Import Progress #173

rburghol · 2022-03-25T15:43:03Z

Overview

Best Running Code

The following code can be used to add a most up to date version of met data to be used in CBP/HSPF models. It includes:

The ability to re-process the gridded data for a given set of years (specified by update_grid_years
To do a given date range (sdate and edate)
Creates land segments models for the specific model version if they do not exist
Runs QA routines to estimate varying attributes of the model meteorological data set for the given time period (see example here: om-model-info/6948507)
See also:
- check_met, check_prad and check_land for seeing if you DO need to run a model check_met, check_prad and check_land: New Helper Functions cbp_wsm#84
- Detailed comparison of NLDAS2 and NOAA gridded precip for detailed QA: https://github.com/HARPgroup/vahydro/issues/666

Data Acquisition and Processing Steps

Incorporate new meteorology data and QA Generate met2date (Meteorology to current date) model_meteorology#20
Run the land model for your basin of interest
Run the river model for your basin of interest (optional)

Notes:

You may need to create the directory /media/model/p6/out/land/[scenario]/eos when running a scenario for the first time, prior to running Rscript run/export/wdm_export_flow.R
first time you run slurm (cbp OneCommand.wsh) it will prompt you to run configure.csh,. You must run configure.csh from run_bhatt, so:

cd /opt/model/p6/vadeq
./configure.csh
# must have river order scripts
cbp bhatt_make_riv_seglist_order.csh JL2_6850_6890

Tasks

Based on process outlined in HARPgroup/cbp_wsm#59

York

# create seglist for YP3_6470_6690 
cbp basingen.csh CFBASE30Y20180615_vadeq 6470

# generate phase 6 meteorology WDMs
cd /opt/model/p6/vadeq
# Test in South Anna
# segs=`cbp get_landsegs YP3_6470_6690`
# do the whole basin 
segs=`cbp get_landsegs Y`
for i in $segs; do
  # convert grid CSVs into land segment CSVs
  a2l_one 1984010100 2020123123 /backup/meteorology/out/grid_met_csv /backup/meteorology/out/lseg_csv $i
  # update long term averages
  LongTermAvgRNMax /backup/meteorology/out/lseg_csv/1984010100-2020123123 /backup/meteorology/out/lseg_csv/RNMax 1 $i
  # finally, create a WDM for each land seg
  # this script reads the file /etc/hspf.config to get directories.
  wdm_pm_one $i 1984010100 2020123123 nldas2 harp2021 nldas1221 p20211221 
done

# Run them
# edit the run batch config
# set BASINS = YP3_6470_6690 
# set SUMALL_BASINS      = ( YP3_6470_6690 )
# copy the seglists from p5
cp ../../p53/p532c-sova/config/seglists/Y.* ./config/seglists/
nano config/control/script/CFBASE30Y20180615_vadeq.con
# run the model (only completes through runoff ATM)\
# Run south anna as a test
# cbp OneCommandWSM.csh CFBASE30Y20180615_vadeq  YP3_6470_6690 
# run the whole Pamunkey
cbp OneCommandWSM.csh CFBASE30Y20180615_vadeq  Y 

cd /opt/model/p6/vadeq
. hspf_config
for i in $segs; do
  # export to OM readable CSV
  Rscript run/export/wdm_export_flow.R CFBASE30Y20180615_vadeq $i 1984 2020 $CBP_EXPORT_DIR $CBP_ROOT
  # preload database tables into OM
  create_landseg_table.sh $i CFBASE30Y20180615_vadeq
done

James

cbp basingen.csh CFBASE30Y20180615_vadeq J

# generate phase 6 meteorology WDMs
cd /opt/model/p6/vadeq
# do the whole basin 
segs=`cbp get_landsegs J`
for i in $segs; do
  # convert grid CSVs into land segment CSVs
  a2l_one 1984010100 2020123123 /backup/meteorology/out/grid_met_csv /backup/meteorology/out/lseg_csv $i
  # update long term averages
  LongTermAvgRNMax /backup/meteorology/out/lseg_csv/1984010100-2020123123 /backup/meteorology/out/lseg_csv/RNMax 1 $i
  # finally, create a WDM for each land seg
  # this script reads the file /etc/hspf.config to get directories.
  wdm_pm_one $i 1984010100 2020123123 nldas2 harp2021 nldas1221 p20211221
done

# Run them
# edit the run batch config
# set BASINS = J
# set SUMALL_BASINS      = ( J )
# make seglists
cbp basingen.csh CFBASE30Y20180615_vadeq J
nano config/control/script/CFBASE30Y20180615_vadeq.con
# run the model (only completes through runoff ATM)
# run the whole James
cbp OneCommandWSM.csh CFBASE30Y20180615_vadeq  J

cd /opt/model/p6/vadeq
. hspf_config
for i in $segs; do
  # export to OM readable CSV
  Rscript run/export/wdm_export_flow.R CFBASE30Y20180615_vadeq $i 1984 2020 $CBP_EXPORT_DIR $CBP_ROOT
  # preload database tables into OM
  # note: we add arguments "p6 1" at end of cmd below to force overwrite since we have an earlier version of this table
  create_landseg_table.sh $i CFBASE30Y20180615_vadeq p6 1
done

Rappahannock


# copy the seglists from p5
cbp basingen.csh CFBASE30Y20180615_vadeq R
# generate phase 6 meteorology WDMs
cd /opt/model/p6/vadeq
# do the whole basin 
segs=`cbp get_landsegs R`
for i in $segs; do
  # convert grid CSVs into land segment CSVs
  a2l_one 1984010100 2020123123 /backup/meteorology/out/grid_met_csv /backup/meteorology/out/lseg_csv $i
  # update long term averages
  LongTermAvgRNMax /backup/meteorology/out/lseg_csv/1984010100-2020123123 /backup/meteorology/out/lseg_csv/RNMax 1 $i
  # finally, create a WDM for each land seg
  # this script reads the file /etc/hspf.config to get directories.
  wdm_pm_one $i 1984010100 2020123123 nldas2 harp2021 nldas1221 p20211221 
done

# Run them
# edit the run batch config
# set BASINS = R
# set SUMALL_BASINS      = ( R )
nano config/control/script/CFBASE30Y20180615_vadeq.con
# run the model (only completes through runoff ATM)\
# run the whole James
cbp OneCommandWSM.csh CFBASE30Y20180615_vadeq  R

cd /opt/model/p6/vadeq
. hspf_config
for i in $segs; do
  # export to OM readable CSV
  Rscript run/export/wdm_export_flow.R CFBASE30Y20180615_vadeq $i 1984 2020 $CBP_EXPORT_DIR $CBP_ROOT
  # preload database tables into OM
  create_landseg_table.sh $i CFBASE30Y20180615_vadeq
done

Potomac


# copy the seglists from p5
cbp basingen.csh CFBASE30Y20180615_vadeq P
# generate phase 6 meteorology WDMs
cd /opt/model/p6/vadeq
# do the whole basin 
segs=`cbp get_landsegs P`
for i in $segs; do
  # convert grid CSVs into land segment CSVs
  a2l_one 1984010100 2020123123 /backup/meteorology/out/grid_met_csv /backup/meteorology/out/lseg_csv $i
  # update long term averages
  LongTermAvgRNMax /backup/meteorology/out/lseg_csv/1984010100-2020123123 /backup/meteorology/out/lseg_csv/RNMax 1 $i
  # finally, create a WDM for each land seg
  # this script reads the file /etc/hspf.config to get directories.
  wdm_pm_one $i 1984010100 2020123123 nldas2 harp2021 nldas1221 p20211221
done

# Run them
# edit the run batch config
# set BASINS = P
# set SUMALL_BASINS      = ( P )
nano config/control/script/CFBASE30Y20180615_vadeq.con
# run the model (only completes through runoff ATM)\
# run the whole James
cbp OneCommandWSM.csh CFBASE30Y20180615_vadeq  P

cd /opt/model/p6/vadeq
. hspf_config
for i in $segs; do
  # export to OM readable CSV
  Rscript run/export/wdm_export_flow.R CFBASE30Y20180615_vadeq $i 1984 2020 $CBP_EXPORT_DIR $CBP_ROOT
  # preload database tables into OM
  create_landseg_table.sh $i CFBASE30Y20180615_vadeq
done

The text was updated successfully, but these errors were encountered:

jdkleiner · 2022-04-27T13:48:08Z

Shenandoah

# create seglist for PS5_4370_4150
cbp basingen.csh CFBASE30Y20180615_vadeq 4370

# generate phase 6 meteorology WDMs
cd /opt/model/p6/vadeq
# do that subwatershed
segs=`cbp get_landsegs PS5_4370_4150`
for i in $segs; do
  # convert grid CSVs into land segment CSVs
  a2l_one 1984010100 2020123123 /backup/meteorology/out/grid_met_csv /backup/meteorology/out/lseg_csv $i
  # update long term averages
  LongTermAvgRNMax /backup/meteorology/out/lseg_csv/1984010100-2020123123 /backup/meteorology/out/lseg_csv/RNMax 1 $i
  # finally, create a WDM for each land seg
  # this script reads the file /etc/hspf.config to get directories.
  wdm_pm_one $i 1984010100 2020123123 nldas2 harp2021 nldas1221 p20211221 
done

# Run them
# edit the run batch config
# set BASINS = PS5_4370_4150
# set SUMALL_BASINS      = ( PS5_4370_4150)
nano config/control/script/CFBASE30Y20180615_vadeq.con
# run the model (only completes through runoff ATM)\
# run that basin
cbp OneCommandWSM.csh CFBASE30Y20180615_vadeq  PS5_4370_4150 

cd /opt/model/p6/vadeq
. hspf_config
for i in $segs; do
  # export to OM readable CSV
  Rscript run/export/wdm_export_flow.R CFBASE30Y20180615_vadeq $i 1984 2020 $CBP_EXPORT_DIR $CBP_ROOT
  # preload database tables into OM
  create_landseg_table.sh $i CFBASE30Y20180615_vadeq
done

rburghol · 2022-08-31T14:51:05Z

Just do a quick update to create WDM files when the actual NLDAS dataset has already been processed fully. NOTE: we now run this from within the model version folder that we desire, and use wdm_pm_one instead of ./wdm_pm_one, which executes the globally available /usr/local/wdm_pm_one script.

# generate phase 6 meteorology WDMs
cd /opt/model/p6/vadeq/
# do that subwatershed
segs=`cbp get_landsegs J`
for i in $segs; do
  #only need to create a WDM for each land seg
  # this script reads the file /etc/hspf.config to get directories.
  wdm_pm_one $i 1984010100 2020123123 nldas2 harp2021 nldas1221 p20211221
done

rburghol · 2022-10-25T19:39:11Z

Update for Back Creek Drought. (Append must be mashed up for full year support until we get cloud cover sorted out - see HARPgroup/model_meteorology#13)

Step 1: Generate the full time period of available (trusted) data.

generate individual cell CSVs for individual years with all available data
update long term average RNMax for all data sets (I think it is shared)
Generate land segments CSVs for the areas of interest.

rm /backup/meteorology/out/lseg_csv/met2date 
ln -s /backup/meteorology/out/lseg_csv/1984010100-2022093023 /backup/meteorology/out/lseg_csv/met2date 

# generate full met time period (needed due to potential updates in RNMax)
sdate="1984010100"  
edate="2022093023"

segs="N51017 N51091"
cd /backup/meteorology
for i in $segs; do
  # handle the grid2land due to an error
  grid2land.sh 20210101 20211231 /backup/meteorology /backup/meteorology/out/grid_met_csv $i
  grid2land.sh 20220101 20220930 /backup/meteorology /backup/meteorology/out/grid_met_csv $i

  cd /opt/model/p6/vadeq
  # convert grid CSVs into land segment CSVs
  a2l_one $sdate $edate /backup/meteorology/out/grid_met_csv /backup/meteorology/out/lseg_csv $i
  # update long term averages
  LongTermAvgRNMax /backup/meteorology/out/lseg_csv/met2date /backup/meteorology/out/lseg_csv/RNMax 1 $i
done

rburghol · 2022-10-28T17:58:56Z

Create Mash-up and Run

Use the full available data with data from 2002 because wdm_insert_ALL requires a full calendar year (this can be fixed with wdm_insert_one but we must insure that the cloud cover is calculated properly, see Translate Cloud Cover CLDC calculation (DSN 1005) from C++ to R model_meteorology#13 )

cd /opt/model/p6/vadeq
. hspf_config

# create a drought scenario
cbp copy_scenario vadeq_2021 drought

sdate="1984010100"  
edate="2022123123"
met_scen="droughtmet"
prad_scen="droughtprad"
model_dataset="nldas2"
model_subset="drought"
segs=`cbp get_landsegs JU1_6290_6590`
for i in $segs; do
  # do the mashup top fill out the rest of the calendar year so wdm_insert_ALL won't break
  Rscript /opt/model/model_meteorology/R/mash.R $i 1984-01-01 2022-09-30 2002-10-01 2002-12-31 /backup/meteorology/out/lseg_csv/met2date /backup/meteorology/out/lseg_csv/mash
  wdm_pm_one $i $sdate $edate $model_dataset $model_subset $met_scen $prad_scen mash overwrite
done

# Change settings in 
# - river/drought.con land/drought.con
#   - TIME 1984 01 01 2022 12 31
#   - METEOROLOGY droughtmet 
#   - PRECIP ATMOS DEPOSITION droughtprad
# Then run
scenario="drought"
cd /opt/model/p6/vadeq
. hspf_config
MODEL_ROOT=/opt/model/p6/vadeq
META_MODEL_ROOT=/opt/model/meta_model
export MODEL_ROOT META_MODEL_ROOT
foreach i in $segs; do
  sbatch /opt/model/meta_model/run_model hspf_cbp6 $scenario $i auto land 
  # in order to send to OM do this:
  Rscript $MODEL_ROOT/run/export/wdm_export_flow.R $scenario $i 1984 2022 $CBP_EXPORT_DIR $MODEL_ROOT
done

Export for Use in OM

Export
Add a scenario to the target select list for "drought"

segs=`cbp get_landsegs JU1_6290_6590`
for i in $segs; do
  create_landseg_table.sh $i drought
done

rburghol · 2023-01-20T17:40:34Z

Most recent workflow includes QA and model property creation

# generate full met time period (needed due to potential updates in RNMax)
. hspf_config 
sdate="1984010100"  
edate="2022123123"
ftype="cbp6_landseg"
model_version="cbp-6.0"
segs="N51113"
cd /backup/meteorology
dataset=${sdate}-${edate}
update_grid_years="2021 2022"
for i in $segs; do
  for y in $update_grid_years; do
    spd="${y}010100"
    epd="${y}123123"
  ./grid2land.sh $spd $epd /backup/meteorology /backup/meteorology/out/grid_met_csv $i
  done

  cd /opt/model/p6/vadeq
  # convert grid CSVs into land segment CSVs for whole time period
  a2l_one $sdate $edate /backup/meteorology/out/grid_met_csv /backup/meteorology/out/lseg_csv $i
  wdm_pm_one $i $sdate $edate nldas2 harp2021 nldas1221 p20211221
  # update long term averages in model component
  cd /backup/meteorology/
  LongTermAvgRNMax /backup/meteorology/out/lseg_csv/${sdate}-${edate} /backup/meteorology/out/lseg_csv/RNMax 1 $i
  Rscript R/lseg_het.R $i $dataset
  Rscript R/lseg_hset.R $i $dataset
  Rscript R/Table_Rolling_Averages.R $i $dataset $ftype $model_version
  Rscript R/lseg_rolling_avg_graphs.R $i $dataset $ftype $model_version
  echo "View at: http://deq1.bse.vt.edu:81/met/out/lseg_csv/1984010100-2020123123/images/df.90day.precip_${i}.png"
  # Run QA
  Rscript R/lseg_qa_test_timeseries.R $i $dataset $ftype $model_version

done

rburghol · 2023-01-27T19:52:30Z

Update Shenandoah Headwaters through 12-31-2020 (tried trhough 2021, but need to update data)
Note: CFBASE30Y20180615_vadeq is currently the scenario used for the met 2 date in vahydro model


segs=`cbp get_landsegs PS2_5560_5100`
land_scenario="CFBASE30Y20180615_vadeq"
for i in $segs; do
  echo "sbatch /opt/model/meta_model/run_model hspf_cbp6 $land_scenario $i auto land "
  sbatch /opt/model/meta_model/run_model hspf_cbp6 $land_scenario $i auto land 
done

rburghol · 2023-02-13T21:15:00Z

Checking on swift creek. Repeated for Sandy River (also in Appomattox), (change to basin="JA1_7640_7280" and rerun)

cd /opt/model/p6/vadeq
. hspf_config
MODEL_ROOT=$CBP_ROOT
META_MODEL_ROOT="/opt/model/meta_model"
export MODEL_ROOT META_MODEL_ROOT
basin="JA0_7291_7290"
segs=`cbp get_landsegs $basin`
land_scenario="CFBASE30Y20180615_vadeq"
met_scenario=`cbp get_config $land_scenario river METEOROLOGY`
atdep_scenario=`cbp get_config $land_scenario river "PRECIP ATMOS DEPOSITION"`

cbp check_met $met_scenario $basin
cbp check_prad $atdep_scenario $basin

MET and PRAD are in place, but land has not been run.
Still, want to insure that we have the entire period to work with so, we repeat steps from above


# generate full met time period (needed due to potential updates in RNMax)
. hspf_config 
syear="1984"
eyear="2022"
sdate="${syear}010100"  
edate="${eyear}123123"
ftype="cbp6_landseg"
model_version="cbp-6.0"
segs=`cbp get_landsegs $basin`
cd /backup/meteorology
dataset=${sdate}-${edate}
update_grid_years="2021 2022"
for i in $segs; do
  for y in $update_grid_years; do
    spd="${y}010100"
    epd="${y}123123"
  ./grid2land.sh $spd $epd /backup/meteorology /backup/meteorology/out/grid_met_csv $i
  done

  cd /opt/model/p6/vadeq
  # convert grid CSVs into land segment CSVs for whole time period
  a2l_one $sdate $edate /backup/meteorology/out/grid_met_csv /backup/meteorology/out/lseg_csv $i
  # update long term averages in model component
  cd /backup/meteorology/
  LongTermAvgRNMax /backup/meteorology/out/lseg_csv/${sdate}-${edate} /backup/meteorology/out/lseg_csv/RNMax 1 $i
  Rscript R/lseg_het.R $i $dataset
  Rscript R/lseg_hset.R $i $dataset
  Rscript R/Table_Rolling_Averages.R $i $dataset $ftype $model_version
  Rscript R/lseg_rolling_avg_graphs.R $i $dataset $ftype $model_version
  echo "View at: http://deq1.bse.vt.edu:81/met/out/lseg_csv/1984010100-2020123123/images/df.90day.precip_${i}.png"
  # Run QA
  Rscript R/lseg_qa_test_timeseries.R $i $dataset $ftype $model_version
  # import the data into the WDM and model scenario 
  cd /opt/model/p6/vadeq
  wdm_pm_one $i $sdate $edate nldas2 harp2021 nldas1221 p20211221
done

Verify that all is good by looking at the model meteorology QA data:
- N51041 om-model-info/6972605
- N51145 om-model-info/6972621
Finally, we run the land models.

for i in $segs; do
  echo "sbatch /opt/model/meta_model/run_model hspf_cbp6 $land_scenario $i auto land "
  sbatch /opt/model/meta_model/run_model hspf_cbp6 $land_scenario $i auto land 
done

Then, update the OM model decks

cd /opt/model/p6/vadeq
. hspf_config
for i in $segs; do
  # export to OM readable CSV
  Rscript run/export/wdm_export_flow.R CFBASE30Y20180615_vadeq $i $syear $eyear $CBP_EXPORT_DIR $CBP_ROOT
  # preload database tables into OM
  create_landseg_table.sh $i CFBASE30Y20180615_vadeq p6 1
done

rburghol · 2023-04-17T17:14:37Z

Full Appomattox JA5_7480_0001

basin=JA5_7480_0001

rburghol · 2023-07-11T11:39:26Z

N54063 is missing, so: (note, did 1984-2020 as well as 1984-2022 so that all model runs can be completed)

had to manually define the cell config file from containment of ftype nldas2 at ows-watershed-dash-info/485536
file saved as /backup/meteorology/seg_maps/N54063.txt

cd /opt/model/p6/vadeq
. hspf_config
MODEL_ROOT=$CBP_ROOT
META_MODEL_ROOT="/opt/model/meta_model"
segs="N54063"
syear="1984"
eyear="2022"
sdate="${syear}010100"  
edate="${eyear}123123"
ftype="cbp6_landseg"
land_scenario="CFBASE30Y20180615_vadeq"
model_version="cbp-6.0"

cd /backup/meteorology
dataset=${sdate}-${edate}
update_grid_years="2021 2022"
for i in $segs; do
  for y in $update_grid_years; do
    spd="${y}010100"
    epd="${y}123123"
  ./grid2land.sh $spd $epd /backup/meteorology /backup/meteorology/out/grid_met_csv $i
  done
done
# summarize and convert into WDM
for i in $segs; do
  # convert grid CSVs into land segment CSVs for whole time period
  a2l_one $sdate $edate /backup/meteorology/out/grid_met_csv /backup/meteorology/out/lseg_csv $i
  wdm_pm_one $i $sdate $edate nldas2 harp2021 nldas1221 p20211221
  # update long term averages in model component
  cd /backup/meteorology/
  LongTermAvgRNMax /backup/meteorology/out/lseg_csv/${sdate}-${edate} /backup/meteorology/out/lseg_csv/RNMax 1 $i
  Rscript R/lseg_het.R $i $dataset
  Rscript R/lseg_hset.R $i $dataset
  Rscript R/Table_Rolling_Averages.R $i $dataset $ftype $model_version
  Rscript R/lseg_rolling_avg_graphs.R $i $dataset $ftype $model_version
  echo "View at: http://deq1.bse.vt.edu:81/met/out/lseg_csv/${sdate}-${edate}/images/df.90day.precip_${i}.png"
  # Run QA
  Rscript R/lseg_qa_test_timeseries.R $i $dataset $ftype $model_version

done

# run the land model
cd /opt/model/p6/vadeq
for i in $segs; do
  echo "sbatch /opt/model/meta_model/run_model hspf_cbp6 $land_scenario $i auto land "
  sbatch /opt/model/meta_model/run_model hspf_cbp6 $land_scenario $i auto land 
done
# Then, update the OM model decks
cd /opt/model/p6/vadeq
. hspf_config
for i in $segs; do
  # export to OM readable CSV
  Rscript run/export/wdm_export_flow.R CFBASE30Y20180615_vadeq $i $syear $eyear $CBP_EXPORT_DIR $CBP_ROOT
  # preload database tables into OM
  create_landseg_table.sh $i CFBASE30Y20180615_vadeq p6 1
done

rburghol · 2023-08-31T13:05:55Z

Rappahannock

Set Up Variables

. hspf_config
MODEL_ROOT=$CBP_ROOT
META_MODEL_ROOT="/opt/model/meta_model"
basin="RU5_6030_0001"
segs=`cbp get_landsegs $basin`
syear="1984"
eyear="2022"
sdate="${syear}010100"  
edate="${eyear}123123"
ftype="cbp6_landseg"
land_scenario="CFBASE30Y20180615_vadeq"
model_version="cbp-6.0"

Update Grids (if needed)

cd /backup/meteorology
dataset=${sdate}-${edate}
update_grid_years="2021 2022"
for i in $segs; do
  for y in $update_grid_years; do
    spd="${y}010100"
    epd="${y}123123"
  ./grid2land.sh $spd $epd /backup/meteorology /backup/meteorology/out/grid_met_csv $i
  done
done

summarize and convert into WDM

for i in $segs; do
  # convert grid CSVs into land segment CSVs for whole time period
  a2l_one $sdate $edate /backup/meteorology/out/grid_met_csv /backup/meteorology/out/lseg_csv $i
  wdm_pm_one $i $sdate $edate nldas2 harp2021 nldas1221 p20211221
  # update long term averages in model component
  cd /backup/meteorology/
  LongTermAvgRNMax /backup/meteorology/out/lseg_csv/${sdate}-${edate} /backup/meteorology/out/lseg_csv/RNMax 1 $i
  Rscript R/lseg_het.R $i $dataset
  Rscript R/lseg_hset.R $i $dataset
  Rscript R/Table_Rolling_Averages.R $i $dataset $ftype $model_version
  Rscript R/lseg_rolling_avg_graphs.R $i $dataset $ftype $model_version
  echo "View at: http://deq1.bse.vt.edu:81/met/out/lseg_csv/${sdate}-${edate}/images/df.90day.precip_${i}.png"
  # Run QA
  Rscript R/lseg_qa_test_timeseries.R $i $dataset $ftype $model_version
  # Run QA again, note the "1" as last argument, this is to save as a model scenario
  Rscript R/lseg_qa_test_timeseries.R $i $dataset $ftype $model_version 1

done

run the land model

cd /opt/model/p6/vadeq
for i in $segs; do
  echo "sbatch /opt/model/meta_model/run_model hspf_cbp6 $land_scenario $i auto land "
  sbatch /opt/model/meta_model/run_model hspf_cbp6 $land_scenario $i auto land 
done

Then, update the OM model decks

cd /opt/model/p6/vadeq
. hspf_config
for i in $segs; do
  # export to OM readable CSV
  # this step is NO LONGER NEEDED - it is completed in the meta-model analyze step
  # Just kept here for posterity
  # Rscript run/export/wdm_export_flow.R CFBASE30Y20180615_vadeq $i $syear $eyear $CBP_EXPORT_DIR $CBP_ROOT
  # preload database tables into OM
  create_landseg_table.sh $i CFBASE30Y20180615_vadeq p6 1
done

Update OM models with background

cd /opt/model/p6/vadeq
. hspf_config
for i in $segs; do
  # export to OM readable CSV
  sbatch ./run/export/omify_landseg.sh $i $syear $eyear CFBASE30Y20180615_vadeq
done

rburghol · 2023-09-01T14:14:58Z

. hspf_config
MODEL_ROOT=$CBP_ROOT
META_MODEL_ROOT="/opt/model/meta_model"
basin="RU5_6030_0001"
segs=`cbp get_landsegs $basin`
syear="1984"
eyear="2022"
sdate="${syear}010100"  
edate="${eyear}123123"
ftype="cbp6_landseg"
land_scenario="CFBASE30Y20180615_vadeq"
model_version="cbp-6.0"

cd /opt/model/p6/vadeq
for i in $segs; do
  echo "sbatch /opt/model/meta_model/run_model hspf_cbp6 $land_scenario $i auto land "
  sbatch /opt/model/meta_model/run_model hspf_cbp6 $land_scenario $i auto land 
done
cd /opt/model/p6/vadeq
. hspf_config

for i in $segs; do
  # export to OM readable CSV
  # this step is NO LONGER NEEDED - it is completed in the meta-model analyze step
  # Just kept here for posterity
  # Rscript run/export/wdm_export_flow.R CFBASE30Y20180615_vadeq $i $syear $eyear $CBP_EXPORT_DIR $CBP_ROOT
  # preload database tables into OM
  create_landseg_table.sh $i CFBASE30Y20180615_vadeq p6 1
done

rburghol · 2023-09-06T14:23:02Z

nohup ./make_met2date PM7_4820_0001 19840101 20230829 "2022 2023" & >0
see here for mashup in Potomac https://github.com/HARPgroup/vahydro/issues/606#issuecomment-1765085901

rburghol · 2023-09-06T17:11:46Z

bad data for N24031

`a2l_one 1984010101 2023082923 /backup/meteorology/out/grid_met_csv /backup/meteorology/out/lseg_csv N24031

rburghol mentioned this issue Mar 11, 2024

Data Model for NLDAS2 QA/Process Log Data HARPgroup/model_meteorology#31

Open

32 tasks

rburghol changed the title ~~Phase 6 Model Run and Data Export/Import Progress~~ Phase 6 Meteorology Model Run and Data Export/Import Progress Aug 30, 2022

rburghol mentioned this issue Sep 1, 2022

2022 Scenario details HARPgroup/cbp_wsm#70

Open

rburghol self-assigned this Oct 24, 2022

rburghol mentioned this issue Oct 27, 2022

wdm_insert_ALL should handle more than just year HARPgroup/model_meteorology#10

Closed

rburghol assigned jdkleiner Jan 20, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Phase 6 Meteorology Model Run and Data Export/Import Progress #173

Phase 6 Meteorology Model Run and Data Export/Import Progress #173

rburghol commented Mar 25, 2022 •

edited

Loading

jdkleiner commented Apr 27, 2022 •

edited by rburghol

Loading

rburghol commented Aug 31, 2022 •

edited

Loading

rburghol commented Oct 25, 2022 •

edited

Loading

rburghol commented Oct 28, 2022 •

edited

Loading

rburghol commented Jan 20, 2023 •

edited

Loading

rburghol commented Jan 27, 2023 •

edited

Loading

rburghol commented Feb 13, 2023 •

edited

Loading

rburghol commented Apr 17, 2023

rburghol commented Jul 11, 2023 •

edited

Loading

rburghol commented Aug 31, 2023 •

edited

Loading

rburghol commented Sep 1, 2023

rburghol commented Sep 6, 2023 •

edited

Loading

rburghol commented Sep 6, 2023

Phase 6 Meteorology Model Run and Data Export/Import Progress #173

Phase 6 Meteorology Model Run and Data Export/Import Progress #173

Comments

rburghol commented Mar 25, 2022 • edited Loading

Overview

Best Running Code

Data Acquisition and Processing Steps

Tasks

York

James

Rappahannock

Potomac

jdkleiner commented Apr 27, 2022 • edited by rburghol Loading

Shenandoah

rburghol commented Aug 31, 2022 • edited Loading

rburghol commented Oct 25, 2022 • edited Loading

rburghol commented Oct 28, 2022 • edited Loading

Create Mash-up and Run

Export for Use in OM

rburghol commented Jan 20, 2023 • edited Loading

rburghol commented Jan 27, 2023 • edited Loading

rburghol commented Feb 13, 2023 • edited Loading

rburghol commented Apr 17, 2023

rburghol commented Jul 11, 2023 • edited Loading

rburghol commented Aug 31, 2023 • edited Loading

Set Up Variables

Update Grids (if needed)

summarize and convert into WDM

run the land model

Then, update the OM model decks

Update OM models with background

rburghol commented Sep 1, 2023

rburghol commented Sep 6, 2023 • edited Loading

rburghol commented Sep 6, 2023

rburghol commented Mar 25, 2022 •

edited

Loading

jdkleiner commented Apr 27, 2022 •

edited by rburghol

Loading

rburghol commented Aug 31, 2022 •

edited

Loading

rburghol commented Oct 25, 2022 •

edited

Loading

rburghol commented Oct 28, 2022 •

edited

Loading

rburghol commented Jan 20, 2023 •

edited

Loading

rburghol commented Jan 27, 2023 •

edited

Loading

rburghol commented Feb 13, 2023 •

edited

Loading

rburghol commented Jul 11, 2023 •

edited

Loading

rburghol commented Aug 31, 2023 •

edited

Loading

rburghol commented Sep 6, 2023 •

edited

Loading