HSPF to hsp2 Test Cases #211

rburghol · 2022-06-13T20:30:03Z

Resources/Goals

h5dump - command line access to hdf5 databases
- h5dump --help lists command line option ("-- or - switches)
?h5read show help file in R
How do we find what is available in the hdf5?
- The numbers in the hdf5 don't really correspond to DSNs do they?
- What about hspf/hsp2 "internals"?
How does the contents of the h5 file correspond to the UCI: https://raw.githubusercontent.com/HARPgroup/HSPsquared/master/tests/test_cbp_river/OR1_7700_7980.uci

HSP2 River test case

River Seg: OR1_7700_7980 (Upper Cub Creek)
- create a directory in your home dir (terminal): mkdir cbp_river_test
- move into that directory: cd cbp_river_test
- Download Files for demo:
  - Repeat this example for all 4 files below: wget https://github.com/HARPgroup/HSPsquared/raw/master/tests/test_cbp_river/OR1_7700_7980.uci
  - Model config: https://github.com/HARPgroup/HSPsquared/raw/master/tests/test_cbp_river/OR1_7700_7980.uci
  - River outputs: https://github.com/HARPgroup/HSPsquared/raw/master/tests/test_cbp_river/OR1_7700_7980.wdm
  - Meteorological inputs (evap, temp, wind NOT precip): https://github.com/HARPgroup/HSPsquared/raw/master/tests/test_cbp_river/met_A51037.wdm
  - Precip and atmospheric deposition: https://github.com/HARPgroup/HSPsquared/raw/master/tests/test_cbp_river/prad_A51037.wdm
  - Water supply inputs: https://github.com/HARPgroup/HSPsquared/raw/master/tests/test_cbp_river/ps_sep_div_ams_p532sova_2021_OR1_7700_7980.wdm
- Use hsp2 to populate the model runtime "hdf5" database: hsp2 import_uci OR1_7700_7980.uci OR1_7700_7980.h5
- Run the model: hsp2 run OR1_7700_7980.h5
  - Alternative if you can't run the hsp2 yet:
    - Download already run h5 file at: http://deq1.bse.vt.edu:81/files/cbp/OR1_7700_7980.h5
- Run R analytics:

install.packages("BiocManager")
BiocManager::install("rhdf5")
library("rhdf5")
rchres_data = h5read("OR1_7700_7980.h5", "/RESULTS/RCHRES_R001/HYDR/table")
names(rchres_data )
quantile(rchres_data$ROVOL)

Export Via h5dump:

h5dump -d "/TIMESERIES/TS411/table" forA51800a.h5`

HSP2 Land Test Case

Get the files from here: https://github.com/HARPgroup/HSPsquared/tree/master/tests/test_cbp_land

hsp2 import_uci A51800.uci A51800.h5
hsp2 run A51800.h5
etc...

Miscellaneous

copy stream inflow here
- cp $CBP_ROOT/tmp/wdm/river/p532sova_2021/eos/OR1_7700_7980.wdm ./
Land segs:
- A51011
- A51037

The text was updated successfully, but these errors were encountered:

rburghol · 2022-06-13T20:47:29Z

Setting up a test case:

Find the files

head -n 20 OR1_7700_7980.uci

RUN

GLOBAL
  OR1_7700_7 riv | P5 | p532sova_2 | Club
  START       1984/01/01        END    2019/12/31
  RUN INTERP OUTPUT LEVEL    1    1
  RESUME     0 RUN     1 TSSFL    15 WDMSFL   16
END GLOBAL

FILES
<FILE>  <UN#>***<----FILE NAME------------------------------------------------->
WDM1       21   ../../../input/scenario/climate/met/nldas1221/met_A51037.wdm
WDM2       22   ../../../input/scenario/climate/prad/p20211221/prad_A51037.wdm
WDM3       23   ps_sep_div_ams_p532sova_2021_OR1_7700_7980.wdm
WDM4       24   OR1_7700_7980.wdm
MESSU      25   OR1_7700_7980.ech
           26   OR1_7700_7980.out
           31   ../../../output/pltgen/river/p532sova_2021/OR1_7700_7980.tau
END FILES


# get them
. hspf_config
fgrep ps_sep_div_ams_p532sova_2021_OR1_7700_7980.wdm /files.dir
/opt/model/p53/p532c-sova/tmp/wdm/river/p532sova_2021/eos/ps_sep_div_ams_p532sova_2021_OR1_7700_7980.wdm
cp /opt/model/p53/p532c-sova/tmp/wdm/river/p532sova_2021/eos/ps_sep_div_ams_p532sova_2021_OR1_7700_7980.wdm ./
cp $CBP_ROOT/input/scenario/climate/met/nldas1221/met_A51037.wdm ./
cp $CBP_ROOT/input/scenario/climate/prad/p20211221/prad_A51037.wdm ./

Get the river WDM. Does this have both the inputs and the outputs from the river, pre-loaded wit inputs from the land runoff?

# the river eos
fgrep OR1_7700_7980.wdm /files.dir
eos="/opt/model/p53/p532c-sova/tmp/wdm/river/p532sova_2021/eos/OR1_7700_7980.wdm"
# this river outputs
riv="/opt/model/p53/p532c-sova/tmp/wdm/river/p532sova_2021/stream/OR1_7700_7980.wdm"

cp $eos ./

Change units line
- From: RESUME 0 RUN UNITS 1
- To: RESUME 0 RUN 1 UNIT SYSTEM 1
  4.. Import the uci and wdm to h5 format.

hsp2 import_uci OR1_7700_7980.uci OR1_7700_7980.h5
DEBUG: Calling readUCI()
DEBUG: linkage net Empty DataFrame
Columns: []
Index: []
DEBUG: linkage sc None
DEBUG: Returned from readUCI
Calling readWDM( met_A51037.wdm )
1003 reading from wdm
1005 reading from wdm
1000 reading from wdm
1001 reading from wdm
1002 reading from wdm
1004 reading from wdm
Calling readWDM( prad_A51037.wdm )
2006 reading from wdm
2004 reading from wdm
2007 reading from wdm
2003 reading from wdm
2005 reading from wdm
2002 reading from wdm
2001 reading from wdm
2000 reading from wdm
Calling readWDM( ps_sep_div_ams_p532sova_2021_OR1_7700_7980.wdm )
3000 reading from wdm
3003 reading from wdm
3006 reading from wdm
3005 reading from wdm
3001 reading from wdm
3004 reading from wdm
3002 reading from wdm
3010 reading from wdm
3008 reading from wdm
3007 reading from wdm
3009 reading from wdm
3061 reading from wdm
3062 reading from wdm
3021 reading from wdm
3022 reading from wdm
3023 reading from wdm
3024 reading from wdm
Calling readWDM( OR1_7700_7980.wdm )
111 reading from wdm
23 reading from wdm
22 reading from wdm
21 reading from wdm
13 reading from wdm
12 reading from wdm
11 reading from wdm
31 reading from wdm
32 reading from wdm
33 reading from wdm
34 reading from wdm
35 reading from wdm
36 reading from wdm
41 reading from wdm
44 reading from wdm
43 reading from wdm
42 reading from wdm
45 reading from wdm
51 reading from wdm
52 reading from wdm
53 reading from wdm
112 reading from wdm
113 reading from wdm
121 reading from wdm
122 reading from wdm
211 reading from wdm
212 reading from wdm
213 reading from wdm
221 reading from wdm
222 reading from wdm
123 reading from wdm
223 reading from wdm
131 reading from wdm
231 reading from wdm
132 reading from wdm
232 reading from wdm
133 reading from wdm
233 reading from wdm
134 reading from wdm
234 reading from wdm
135 reading from wdm
235 reading from wdm
136 reading from wdm
236 reading from wdm
141 reading from wdm
241 reading from wdm
142 reading from wdm
242 reading from wdm
143 reading from wdm
243 reading from wdm
144 reading from wdm
244 reading from wdm
145 reading from wdm
245 reading from wdm
151 reading from wdm
251 reading from wdm
152 reading from wdm
252 reading from wdm
153 reading from wdm
253 reading from wdm
311 reading from wdm
411 reading from wdm
344 reading from wdm
312 reading from wdm
412 reading from wdm
313 reading from wdm
413 reading from wdm
321 reading from wdm
421 reading from wdm
322 reading from wdm
422 reading from wdm
453 reading from wdm
323 reading from wdm
423 reading from wdm
353 reading from wdm
331 reading from wdm
431 reading from wdm
452 reading from wdm
332 reading from wdm
432 reading from wdm
352 reading from wdm
333 reading from wdm
433 reading from wdm
451 reading from wdm
334 reading from wdm
434 reading from wdm
351 reading from wdm
335 reading from wdm
435 reading from wdm
445 reading from wdm
336 reading from wdm
436 reading from wdm
345 reading from wdm
341 reading from wdm
441 reading from wdm
444 reading from wdm
342 reading from wdm
442 reading from wdm
443 reading from wdm
343 reading from wdm

Run it

hsp2 run OR1_7700_7980.h5

rburghol · 2022-06-14T20:46:55Z

Could not run uci due to some issues with missing ADCALC-DATA block.
Made a branch called "honly" and did some tinkering
Show difference between branch and master

 git diff honly..master HSP2/main.py

Try to enable ADCALC.

can it run in p532?
enabled (uncommented) ADCALC line in config/control/river/p532sova_2021.con
regen uci: cbp run_rug.csh p532sova_2021 OR1_7700_7980
run model: cbp run_river.csh p532sova_2021 OR1_7700_7980
copy to folder cp /opt/model/p53/p532c-sova/tmp/uci/river/p532sova_2021/OR1_7700_7980.uci /home/rob/working/modeling/hsp2/test_cbp_river
All proceeds well in p532
hsp2 import_uci OR1_7700_7980.uci OR1_7700_7980.h5 works
hsp2 run OR1_7700_7980.h5
- Error:

  File "/opt/model/HSPsquared/HSP2/main.py", line 170, in main
    ui['FLAGS']['HTFG'] = flags['HTRCH']
KeyError: 'FLAGS'

repeat above process to enable HTRCH
created helper function patch_uci_head.sh
- Use: patch_uci_head.sh new_uci uci_with_correct_header
- Ex: ./patch_uci_head.sh /opt/model/p53/p532c-sova/tmp/uci/river/p532sova_2021/OR1_7700_7980.uci OR1_7700_7980.uci
repeat for OXRX
take another tack: all but plankton
or, add adcalc, but that's it since it won't run without adcalc.

rburghol · 2022-06-15T00:03:34Z

Problem was setting FLAGS as default for ALL modules because some modules would fail on blank FLAGS. Need to test default ui[FLAGS] on each module to see where it can help/hinder.

rburghol · 2022-06-15T13:26:33Z

How to debug in python?

print a list (there are no arrays), with basic formatting: print(*a, sep = "\n")
- must use "*" preceding list name, ex:
  - print(*ui['FLAGS'], sep = "\n")

rburghol · 2022-06-15T19:30:24Z

Can run p532 river UCI if we

use code in commit to allow no RQUAL: HARPgroup/HSPsquared@be1c3d8
disable all but HYDR and ADCALC in p532 con file (see below "Code 1")
Regenerate uci
Copy relevant UCI info into runnuing dir: ./patch_uci_head.sh /opt/model/p53/p532c-sova/tmp/uci/river/p532sova_2021/OR1_7700_7980.uci OR1_7700_7980.uci
import uci: hsp2 import_uci OR1_7700_7980.uci OR1_7700_7980.h5
run: hsp2 run OR1_7700_7980.h5
what is the DSN?: h5dump -n OR1_7700_7980.h5 | grep 111
- Choose this: /TIMESERIES/TS111/table
export from cmd: h5dump -d "/RESULTS/RCHRES_R001/HYDR/table" OR1_7700_7980.h5
View in R - see Rscript 1

Rscript 1: Try to access data from h5 file post-run.

R
library("rhdf5")
rchres_data = h5read("OR1_7700_7980.h5", "/RESULTS/RCHRES_R001/HYDR/table")
 quantile(rchres_data$ROVOL)
         0%         25%         50%         75%        100%
  0.1229398   2.8218678   5.1851766   9.1894212 414.5833130

Code 1: Excerpt from config/control/river/p532sova_2021.con

*** active modules for this scenario
MODULES
HYDR
ADCALC
***CONS
***HTRCH
***SEDTRN
***GQUAL
***RQUAL
***OXRX
***NUTRX
***PLANK
***PHCARB
END MODULES

glenncampagna self-assigned this Jun 21, 2022

juliabruneau self-assigned this Jun 22, 2022

rburghol assigned gcambridge, megpritch, nicoledarling, rburghol and jdkleiner Jun 22, 2022

rburghol mentioned this issue Jun 23, 2022

rhdf5 #207

Open

This was referenced Jun 24, 2022

Python hdf5 reader #236

Open

Data Dictionary #237

Open

rburghol mentioned this issue Jun 27, 2022

Data Model for Land Segment, Land-use models #242

Open

rburghol mentioned this issue Jul 6, 2022

Add switch to use hsp2 executable HARPgroup/cbp_wsm#64

Open

33 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HSPF to hsp2 Test Cases #211

HSPF to hsp2 Test Cases #211

rburghol commented Jun 13, 2022 •

edited

Loading

rburghol commented Jun 13, 2022

rburghol commented Jun 14, 2022 •

edited

Loading

rburghol commented Jun 15, 2022

rburghol commented Jun 15, 2022 •

edited

Loading

rburghol commented Jun 15, 2022 •

edited

Loading

HSPF to hsp2 Test Cases #211

HSPF to hsp2 Test Cases #211

Comments

rburghol commented Jun 13, 2022 • edited Loading

Resources/Goals

HSP2 River test case

HSP2 Land Test Case

Miscellaneous

rburghol commented Jun 13, 2022

rburghol commented Jun 14, 2022 • edited Loading

rburghol commented Jun 15, 2022

rburghol commented Jun 15, 2022 • edited Loading

rburghol commented Jun 15, 2022 • edited Loading

rburghol commented Jun 13, 2022 •

edited

Loading

rburghol commented Jun 14, 2022 •

edited

Loading

rburghol commented Jun 15, 2022 •

edited

Loading

rburghol commented Jun 15, 2022 •

edited

Loading