SOVA Phase 6 Runoff Integration

[rburghol]

• pseudo Phase 6, which is linked to Phase 5 land segments
• For model run commands see Running P5.3.2 (Southern Rivers) for new Meteorology #166
• Fix work flow to create new landseg CBP file connectors, last time I tried, the om_element_connection was not reset
• Set up file structure, and link current phase5 met output to CBP convention
  • mkdir /media/model/p532
  • mkdir /media/model/p532/out
  • mkdir /media/model/p532/out/land
  • mkdir /media/model/p532/out/land/p532sova_2021
  • mkdir /media/model/p532/out/land/p532sova_2021/eos
  • mkdir /media/model/p532/out/land/p532sova_2021/landuse
• Create a soft-link aliases for "CFBASE30Y20180615_vadeq" and "CFBASE30Y20180615_vadeq" to point to current model scenario
  • ln -s /media/model/p532/out/land/p532sova_2021 /media/model/p532/out/land/CFBASE30Y20180615_vadeq
  • ln -s /media/model/p532/out/land/p532sova_2021 /media/model/p532/out/land/CFBASE30Y20180615
• Set up land use csv files (for using auto-populate routine for land segs)
  • Go to cbp6 dir cd /opt/model/cbp6/R
  • Edit ./R/Examples/export_all_landuse.R to point to /media/model/p532 and /opt/model/p53/p532c-sova
  • install sqldf (in R cmd prompt)
  • Rscript Examples/export_all_landuse.R
• Set up proptotype land seg with CBPLandDataConnectionFile framework (last time we used this as a batch processor: https://github.com/HARPgroup/om/blob/master/drupal/om/src/om_cbp_lrseg_landuse.php)
  1. Create land data model object (copy from template)
  2. set p5 new paths
  3. set landseg
  4. set landuse path
  5. create om element link to the CBP6 Runoff container on the relevant watershed
  • Ex: om-model-info/4834152 links to om-model-info/6712335/dh_properties
  7. create/populate landuse file
  8. Run model once with small timespan to get a suitable table created in model.scratch
  9. export table def SQL into /opt/model/om/sh/cbp_p5_lseg_runoff_template.sql
  10. pre-populate database
  11. run model
• Create template from above cbp_p5_lseg_runoff_template.sql into model.scratch
  • Ran a 1 month model for Powell river with landseg A51105 in CBP6 folder
  • Dump: pg_dump -s -U postgres -d model_scratch -p 5444 -t cbp_p532_cfbase30y20180615_a51105 > /opt/model/om/sh/cbp_p5_lseg_runoff_template.sql
  • Edit to change name from cbp_p532_cfbase30y20180615_a51105 to cbp_p5_lseg_runoff_template
  • cat /opt/model/om/sh/cbp_p5_lseg_runoff_template.sql | psql model_scratch -p 5444 -U postgres
  • Added the cat line to /opt/model/om/sh/setup.model_scratch.sh
• Create batch script for above work flow: see om_create_landseg.php
  • Find out why when it runs for the first time it creates everything, but skips the step where it pushes data from vahydro
• Modify batch to handle all sova land segments om_cbp_lrseg_landuse.php see https://github.com/HARPgroup/vahydro/issues/15
  • Verify model directory and other hard coded stuff?
  • created om_cbp_lrseg_landuse_file_import.php a stand alone version that just sets the landuse table since the elements are created in another script
• Update all sova (and nova) model components:
  • rename the old sova lrseg models to vahydro-0.9
  • Create query to get all sova land river segs needing models (or needing to be updated)
  • Generate landuse files from p532s_sova (see /opt/model/p6/p6_gb604/out/land/CFBASE30Y20180615/landuse/)
  • use batch to update landuse for each SOVA watershed
  • SOVA Local Runoff Folders to use same Runit data matrix as NOVA
  • use batch to copy landuse_var from template with new settings to insure that it is handled as a string

Batch to update landuse for each SOVA watershed

filename='/var/www/html/files/vahydro/sova_p532_p6_runoff.txt'
version='p532'
scenario='CFBASE30Y20180615_vadeq'
template=6711736

while IFS= read -r line; do
  #echo "Text read from file: $line"
  read river_hydroid lrseg_hydroid pid vahydro_version <<< "$line"
  echo "Line: $river_hydroid $lrseg_hydroid $pid $version"
  lseg_pid=`drush scr modules/om/src/om_create_landseg.php $river_hydroid $lrseg_hydroid $version $scenario`
  drush scr modules/om/src/om_cbp_lrseg_landuse_file_import.php cmd $lseg_pid $version $scenario 
  # this causes an endless loop now...
  drush scr modules/om/src/om_copy_subcomp.php cmd dh_properties $template dh_properties $lseg_pid landuse_var
done < $filename 

SOVA Local Runoff Folders use same Runit data matrix as NOVA

for ro_pid in `cat /var/www/html/files/vahydro/sova_runoff_containers.txt`; do
  drush scr modules/om/src/om_copy_subcomp.php cmd dh_properties 6563947 dh_properties $ro_pid Runit
done


copy (
  select pid
  from dh_properties 
  where propname = '1. Local Runoff Inflows'
  and featureid in (
    select pid from dh_properties 
    where propcode = 'vahydro-1.0'
    and featureid in (
      select hydroid from dh_feature 
      where bundle = 'watershed' and ftype = 'vahydro' 
      and substr(hydrocode,17,2) in ('NR', 'OD', 'OR', 'TU', 'BS', 'MN')
    )
  )
) to '/tmp/sova_runoff_containers.txt' ;

NOVA Land-River runoff elements get updated landuse_var component

• see comment below for tmp view def to create tmp_p6_lrseg_models

copy (
  select * from tmp_p6_lrseg_models 
) to '/tmp/sova_p6_runoff.txt' WITH CSV DELIMITER E'\t';


for ro_pid in `cat /var/www/html/files/vahydro/sova_runoff_containers.txt`; do
  drush scr modules/om/src/om_copy_subcomp.php cmd dh_properties 6711736 dh_properties $ro_pid landuse_var
done

[rburghol]

Testing work flow from: https://github.com/HARPgroup/vahydro/issues/457

Goals:

• set all old landseg models to vahydro-0.9
• create newfile-based models attached to the land segment entity
  • test with vahydro based copy property routines
  • decide on best way forward -- should be use the SAME scenario names, like
  • set landseg, modelpath, and model version
• link land segment entity model to CBP6 folder model on watershed
• Decide on

Run ID	Scenario Name
0	CFBASE30Y20180615
4	CFBASE30Y20180615
5	CBASE1808L55CY55R45P50R45P50Y
6	CBASE1808L55CY55R45P10R45P10Y
7	CBASE1808L55CY55R45P90R45P90Y
8	CFBASE30Y20180615
100	CFBASE30Y20180615

omcbid=343608 # OM elementid for "CBP6 Flows" model object attached to river segment model object 
cbid=4834152 # vahydro pid for "CBP6 Flows" model object attached to river segment model object 
lrseg="A51105_TU3_8880_9230"

drush scr modules/om/src/om_create_pair.php cmd 352129 $omcbid $cbid om_class_cbp_eos_file "$lrseg" prop_feature CBPLandDataConnectionFile 37
drush scr modules/om/src/om_create_pair.php cmd 352129 $omcbid $cbid om_class_cbp_eos_file "$lrseg" prop_feature CBPLandDataConnectionFile 37

drush scr modules/om/src/om.migrate.element.php prop_feature 352127 

[rburghol]

Alternate work flow:

• copy lrseg template in vahydro with om_copy_subcomp.php
  • Ex: drush scr modules/om/src/om_copy_subcomp.php cmd dh_properties 6564010 dh_feature 445692 "File-Based Land Segment Runoff Template" 1
• Synch Land Use
• Push new lrseg to OM as child of "CBP6 Flows" container, copying all props
• Need to know:
  • river segment model om_element_id
  • land segment feature hydroid
  • Get "1. Local Runoff Inflow" -> "CBP6 Flows" element id
• Process
  • Update old p5 runoff propcode to vahydro-0.9 (so we have only 1 vahydro 1.0 element)
  • Check if a p6 element exists already?
  • If not, Clone template to feature
  • Set properties, landseg, riverseg, model_path, etc

[rburghol]

Replace the code below with:

version='p532'
scenario='CFBASE30Y20180615_vadeq'
lseg_pid=`drush scr modules/om/src/om_create_landseg.php 68331 445607 $scenario $version`
drush scr modules/om/src/om_cbp_lrseg_landuse_file_import.php cmd $lseg_pid $version $scenario 

lseg_pid=`drush scr modules/om/src/om_create_landseg.php 68331 445608 $scenario $version`
drush scr modules/om/src/om_cbp_lrseg_landuse_file_import.php cmd $lseg_pid $version $scenario 

lseg_pid=`drush scr modules/om/src/om_create_landseg.php 68331 445670 $scenario $version`
drush scr modules/om/src/om_cbp_lrseg_landuse_file_import.php cmd $lseg_pid $version $scenario 

Test in South Fork Roanoke - only use VAHydro now

1. Get river seg hydroid/model container pid
   • hydroid=68331, pid=4702353, elid=250521
2. Get the ID of the CBP6 container model (in vahydro)
   • note: for this example the pid = 4831858, and elementid = 342846
   • Find local runoff inflow (LRI) attached to model in no. 1
   • Find CBP6 model attached to LRI model
3. Get the land segments that flow in this river segment (all phase5 landsegs on sova will get this)
   • This example: A51063_OR1_8280_8020 (445607), A51121_OR1_8280_8020 (445608), and B51161_OR1_8280_8020 (445670)
   • Can use watershed containment view (see VWP Rmd script)
   • set as var $lseg_hydroid = 445607;
4. Check to see if P6 river segment models exists each land-river segment
   • must be both propcode="vahydro-1.0" and varkey = "om_class_cbp_eos_file"
5. Create land seg model if missing
   • Pid of model parent of template: 6564010
   • Note: om_copy_subcomp.php wants to know parent pid and sub-comp name, so we look to the CBP6 Runoff template, and grab it's subcomp File-Based Land Segment Runoff Template" -- FWIW its pid = 6711736
   • drush scr modules/om/src/om_copy_subcomp.php cmd dh_properties 6564010 dh_feature 445607 "File-Based Land Segment Runoff Template" 1
   • ex pid: 6735093
6. Load landseg feature
   • $lseg_feature = entity_load_single('dh_feature', $lseg_hydroid);
7. Populate attributes:
   • landseg
   • riverseg
   • modelpath: /media/model/p532
8. Run land use populator script https://github.com/HARPgroup/om/blob/master/drupal/om/src/om_cbp_lrseg_landuse.php
9. Create a clone of the OM template object attached to the CBP6 Flows object on the riverseg

• cd /var/www/html/om;
• New LR elementid

13. Push landseg to OM as child of Containing Riverseg "CBP6 Runoff"container

• Load object om_element_connection property
• Set to "One time Push all Properties" to new LR elementid
• save()

[rburghol]

To use dev code on deq2: rm -Rf /var/www/html/d.dh/modules/om; ln -s /opt/model/om/drupal/om /var/www/html/d.dh/modules/om

• Next downstream: OR2_8020_8130 (68105)
• land segments: B51161_OR2_8020_8130 (445586), A51121_OR2_8020_8130 (445651), A51161_OR2_8020_8130 (445654)
• load lsegs:

version='p532'
scenario='CFBASE30Y20180615_vadeq'
lseg_pid=`drush scr modules/om/src/om_create_landseg.php 68105 445586 $scenario $version`
drush scr modules/om/src/om_cbp_lrseg_landuse_file_import.php cmd $lseg_pid $version $scenario 
lseg_pid=`drush scr modules/om/src/om_create_landseg.php 68105 445651 $scenario $version`
drush scr modules/om/src/om_cbp_lrseg_landuse_file_import.php cmd $lseg_pid $version $scenario 
lseg_pid=`drush scr modules/om/src/om_create_landseg.php 68105 445654 $scenario $version`
drush scr modules/om/src/om_cbp_lrseg_landuse_file_import.php cmd $lseg_pid $version $scenario 

[rburghol]

• See model json: drush scr modules/om/src/model_json.php 6735179 1
• force a fast synch with OM via json: drush scr modules/om/src/synch_model.php 6735179

[rburghol]

Debug: land use area is zero. Check:

• landuse matrix (was not properly configured at first, lookup columns not set)
• luyear

[rburghol]

Load them:

#filename='/var/www/html/files/vahydro/sova_p532_p6_runoff-exist.txt'
filename='/var/www/html/files/vahydro/sova_p532_p6_runoff.txt'
version='p532'
scenario='CFBASE30Y20180615_vadeq'
template=6711736

while IFS= read -r line; do
  #echo "Text read from file: $line"
  read river_hydroid lrseg_hydroid pid vahydro_version <<< "$line"
  echo "Line: $river_hydroid $lrseg_hydroid $pid $version"
  lseg_pid=`drush scr modules/om/src/om_create_landseg.php $river_hydroid $lrseg_hydroid $version $scenario`
  drush scr modules/om/src/om_cbp_lrseg_landuse_file_import.php cmd $lseg_pid $version $scenario 
  # this causes an endless loop now...
  drush scr modules/om/src/om_copy_subcomp.php cmd dh_properties $template dh_properties $lseg_pid landuse_var
done < $filename 

The supporting queries:

create temp view tmp_p532_lrseg_models as (
  select a.hydroid as rseg_hydroid, 
    b.hydroid as lseg_hydroid,
    c.pid as model_pid, 
    c.propcode as model_version
  from dh_feature as a 
  left outer join dh_feature as b 
  on (
    substr(b.hydrocode,8,13) = substr(a.hydrocode,17,13)
  )
  left outer join dh_properties as c 
  on (
    c.entity_type = 'dh_feature'
    and c.featureid = b.hydroid
    and c.propcode = 'vahydro-1.0'
  )
  where b.ftype = 'cbp532_lrseg'
  and substr(a.hydrocode,17,2) in ('NR', 'OD', 'OR', 'TU', 'BS', 'MN')
  and b.hydroid is not null
  and a.bundle='watershed'
  and b.bundle='landunit'
  and a.ftype='vahydro'
  and (
   NOT (c.propcode = 'vahydro-0.9')
  OR (c.propcode is null)
  )
  and length(a.hydrocode) = 29
  -- uncomment below to sort by either hydrocode
  -- or in a special order to prioritize a given watershed
  order by a.hydrocode
  -- ORDER BY CASE WHEN substr(a.hydrocode,17,1) = 'O' THEN 1 ELSE 2 END
);

copy (
  select * from tmp_p532_lrseg_models 
) to '/tmp/sova_p532_p6_runoff.txt' WITH CSV DELIMITER E'\t';

-- just ones that have models already
copy (
  select * from tmp_p532_lrseg_models 
  where model_version = 'vahydro-1.0'
) to '/tmp/sova_p532_p6_runoff-exist.txt' WITH CSV DELIMITER E'\t';

P6 runoff

create temp view tmp_p6_lrseg_models as (
  select a.hydroid as rseg_hydroid, 
    b.hydroid as lseg_hydroid,
    c.pid as model_pid, 
    c.propcode as model_version
  from dh_feature as a 
  left outer join dh_feature as b 
  on (
    substr(b.hydrocode,13,13) = substr(a.hydrocode,17,13)
  )
  left outer join dh_properties as c 
  on (
    c.entity_type = 'dh_feature'
    and c.featureid = b.hydroid
    and c.propcode = 'vahydro-1.0'
  )
  where b.ftype = 'cbp6_lrseg'
  and b.hydroid is not null
  and a.bundle='watershed'
  and b.bundle='landunit'
  and a.ftype='vahydro'
  and (
   NOT (c.propcode = 'vahydro-0.9')
  OR (c.propcode is null)
  )
  and length(a.hydrocode) = 29
  order by a.hydrocode
);


copy (
  select * from tmp_p6_lrseg_models 
) to '/tmp/sova_p6_runoff.txt' WITH CSV DELIMITER E'\t';

[rburghol]

• Save the objects to patch those that were snagged by the single-quote in the landuse var table bug
• Use variant SQL view above to order by river segment with O (Roanoke) going first

filename='/var/www/html/files/vahydro/sova_p532_p6_runoff-O-1st.txt'

while IFS= read -r line; do
  #echo "Text read from file: $line"
  read river_hydroid lrseg_hydroid pid vahydro_version <<< "$line"
  echo "Line: $river_hydroid $lrseg_hydroid $pid $version"
  drush scr modules/om/src/om_saveprop.php pid $pid
done < $filename 

[rburghol]

Find runoff container with the erroneous variable landuse_var (they shouldn't have one) and delete that property.

filename="/var/www/html/files/vahydro/sova_runoff_containers_with_badvar.txt"
for i in `cat $filename`; do
  drush scr modules/om/src/om_deleteprop.php pid $i
done

The file:

copy (
  select b.pid
  from dh_properties as a 
  left outer join dh_properties as b
  on (
    b.featureid = a.pid
    and b.entity_type = 'dh_properties'
    and b.propname = 'landuse_var'
  )
  where a.propname = '1. Local Runoff Inflows'
    and b.pid is not null
) to '/tmp/sova_runoff_containers_with_badvar.txt' ;