Detailed Work Plan 2022

[rburghol]

Project Timetable And Deliverable Schedule

• 2022 proposal
  DEQ_HARP2022.pdf

HARP Focus Areas

Add links to tasks in place of "*" in cells.

(month)	1	2	3	4	5	6	7	8	9	10	11	12
Program routines for acquiring data from the HD5 format to integrate into existing R summarize functions.	*	*	*	*
migrate southern reversed data into phase 6, and I parameterize files for new sub watersheds from VA Hydro.	*	*	*	*
develop scripts to shuttle data from OM, hsp2 into appropriate WDMs and test.			*	*	*	*
Test and validate baseline hydrology for hsp2		*	*	*	*
Test operational model under development by DEQ modelers		*	*	*	*	*
Implement baseflow separation visualization and analysis routines				*	*	*	*
Testing the DEQ migration of old VWP models							*	*	*	*
Generating FTables for small watersheds								*	*	*	*
evaluate automated testing suite 10 HSP2.			*	*	*	*
Document and share R functions to GITHUB	*	*	*	*	*	*	*	*	*	*	*	*

DEQ Focus Areas Integrated with HARP tasks

• Integration of hsp2 as executable (with wdm_import, and run syntax) for CBP
  • UCI generation tweeks
  • File size optimize h5 (delete after run)
  • Land UCI execution
    • lug
    • p5
    • p6
  • River uci execution
    • lug
    • p5
    • p6
• Integrate HARP stats (Summer & Fall 2022, Winter 2023: Summary of Rscripts and their Arguments #505) - all are HARP unless noted by (DEQ)
  • establish data models (DEQ)
  • Generic Detection of data source Generic script to use in bash to determine if h5 dataset exists in h5 file #349
  • Generic data export fromhdf5 to CSV Generic Script to Export hdf5 data to CSV #538
  • Generic JOIN script to add columns of data from multiple CSVs Script to join a column to an existing csv using sqldf R package #552
  • River stats/run records basics Script to Export river h5s #312
  • Land stats/run record basics Land Model Post-Processing Workflow Summary #342
  • Harvest Water Supply Variables from hydr/wdm inputs, need demand and wd_mgd (demand met) Handle PS and WD (DIVR) in data harvest to _hydr.csv file #541
  • batch re-processing (land Land Use (re)Analysis batch script #539, River )
  • create land segment models (part of code)
  • Water system stats/run record basics
  • Integrate with meta_model Plugin Directory Structure details meta_model#4
    • finalize advanced river stats create summarize_river.R based on old river summary script to take a CSV as argument #556
    • integrate river stats in meta_model "analyze" module (all integration with current OM stats)
    • finalize advanced water system stats
    • integrate water system stats in meta_model "analyze" module (all integration with current OM stats)
    • integrate land segment stats in meta_model "analyze" module (use stats developed by HARP in this project summer 2022)
• specl HSP2 Python Implementation of HSPF SPEC-ACTIONS (withdrawals) HSPsquared#5
  • scope Scoping Code Tasks: HSPsquared#12
  • data passing (specl needs to access everything) Specl and Objects hdf5 data model HSPsquared#20
  • variables (legacy + new)
  • om primitives (see below "Operational model development",
  • om execution: this may be appropriate to do as part of special actions, or it may be a separate model altogether. The crucial part is that since the operational model can affect anything in theory, hooks for operations need to be in every submodule. This makes the argument for integrating them with SPECL since this two has hooks in every Operation.
• Resegmentation Subwatershed Creation in cbp/hspf cbp_wsm#85
  • what is rseg naming convention with nhd+ tribs?
  • what are minimum datasets?
  • Resegmentation scripts:
    • Insure unique ID and no-redundant resegmentation
    • populate routing files
    • Populate Hydro files
    • Populate land use files
    • Store data in VAHydro
  • Generate FTABLEs:
    • Below Bank full
    • Above Bank Full Estimate floodplain based on channel info #562
      • In the process of researching and QA'ing the FTABLEs created with the USGS documented methodology, it was discovered y HARP that the USGS docs did not detail the method used for floodplain parts of the FTABLE. Thus, HARP & Dr. Scott led the effort to explore options to fix the process, and program routines to handle this going forward.
    • Write FTABLEs into CBP Framework FTABLEs for streams #331
    • Assess Error Assess error in floodplain estimation routines as compared to a raster/GIS based method (pick a couple small streams) #546
  • Migrate southern rivers data
    • test expanded meteorology
    • add steps in meta model framework (hsp2_cbp5)
• QA & testing @jdkleiner (see https://github.com/HARPgroup/vahydro/issues/309)
  • Model QA Model Migration QA and validation #736
    • cbp/hsp2 Test base cbp/hsp2 model #737
    • cbp/hsp2+vahydro Test VAHydro hydrography in cbp/hsp2  #738
    • cbp/hsp2 + nhd Test embedded subsheds hydrography in cbp/hsp2 #739
    • cbp/hsp2+vahydro+nhd Hybrid stand-alone and embedded subsheds hydrography  #740
  • Automated Testing Overview of functions:
    • A critical part of developing the new op model code,
    • critical for testing model scenario run parameters when doing permits/planning
    • critical for testing migration
    • What does hsp2 team currently have in place? Plans?
    • crucial intersection with HARP
  • Basic run completion testing (did something just break?)
  • Mathematical/function replicability (more complex)
• Operational model development: Begin coding data model, object classes. HSPsquared#7
  • base object class definitions Begin coding data model, object classes. HSPsquared#7
  • Performance strategy/evaluation for dynamic components Dynamic Object Calculation (form and speed) HSPsquared#35
    • pre-rendered code file
    • Parser and high-speed primitive option Equation Solver: integer keyed tokenized HSPsquared#38
    • object oriented python
    • string index versus integer index Equation evaluation with String Keyed Dict (slow) HSPsquared#42
  • subcomp def (maybe base class should be subcomp and extended class comp?)
    • equations Equation HSPsquared#25
    • lookups/dataMatrix dataMatrix HSPsquared#26
    • Statistics/stacks (may need to decompose for multi-ops like equation or add constant placeholders?)
    • Broadcast channels broadCastObject HSPsquared#36
    • Other (complex objects, should examine building out of components/not migrating since we will be migrating mostly water system elements which are just a basic model container sub classed with no customizations except a "flow_mode" property
  • Final op-code data model
  • Links Model linkages HSPsquared#30
• ETM integration
  • Options/Steps:
    • leverage upstream hspf runoff to drive hsp2 river simulations
    • use wdm import code routines and HARP river exports to populate river WDMs for stream_wdm.exe to run hsp2 seamlessly for hsp2 use Rscript instead of stream_wdm.exe to set upstream reach inflows from .hydr file #481
    • OM integration to wdm OM etm/wdm integration #533
    • Develop full integration of resegmentation in CBP model suite
      • Strategy to handle pump-store tribs
      • Strategy to handle non-pump-store tribs (can use mathematical solution?)
    • (future) redevelop etm code in python: Python ETM is not that needed in the first phase of work, since we need to use hsp2 for the river simulation only with ETM results from fortran version. Fortran ETM can be used for drought models, climate change, and downscaled watersheds with parent unit runoff as well.
    • (future) assemble a "bridge" uci that just combines land use areas and runoff, and routes to the appropriate DSNs in the hd5
    • Custom runoff simulations for downscaled watersheds could need pyETM
    • Speed optimize WDM updating in import_uci mode of hsp2 Optimize/Create Alternative to import_uci to allow no WDM mode, or select list of DSNs HSPsquared#46
• Meta-Model Framework Meta-model Runtime Components meta_model#1
  • Enable p6 hspf and hsp2 in meta_model Examples: Running CBP with hspf and hsp2 meta_model#2
  • Enable p5 hspf and hsp2 in meta_model Examples: Running CBP with hspf and hsp2 meta_model#2
  • SLURM run batch optimization Run with slurm meta_model#3
  • Create script to automated stream order file run lists used by p6 slurm Basin Order with slurm and p6/om cbp_wsm#86
  • test old method against new slurm batch for time (land use is 8x faster on a single model node)
  • plan out the use of shake-tree type methodologies for being parsimonious during re-runs Run with slurm meta_model#3
  • does slurm have a persistent database test slurmdbd meta_model#5
• Documentation
  • evaluate extent to which hsp2 uses doxygen or some other self-doc mechanism
  • Document each script with appropriate doxygen for Rscript and for R functions use doxygen for module addition in https://github.com/HARPgroup/model_meteorology
  • Add doxygen doc support to bash and csh scripts for full tree? This would be ideal!!
  • (HARP) hdf5 documentation and testing
    • generic command line demo h5dump #235
    • hsp2 based demo with R
    • Python: Writing to .h5 file in hsp2 using python #245 & Python hdf5 reader #236

(month)	1	2	3	4	5	6	7	8	9	10	11	12
Program hsp2 operational model.	*	*	*	*
Document and share R functions to GITHUB	*	*	*	*	*	*	*	*	*	*	*	*

[rburghol]

Understanding hdf5

• A portable, file-based, database
• Command line tools
  • h5dump: can list all data sources, extract specific sources.
• R module
  • challenge is that timestamp not filled in: may benefit from filing a bug report
  • is this a problem with timestamp format? The hsp2 timestamp has many many digits