Data Model for NLDAS2 QA/Process Log Data

[rburghol]

Overview

• spatial
  • import p532 landsegs: see here for both p5 and p6 imports
  • import p6 landsegs
  • download and import NLDAS2 shapefile to vahydro (shapefile in NLDAS_Grid_Reference.zip ), import process in https://github.com/HARPgroup/vahydro/wiki/NLDAS2-Reference
  • See wiki for import details: https://github.com/HARPgroup/vahydro/wiki/Modeling-Geometries
• outline workflow
• define data model
  • grid shape attributes
    • properties (overall status of data startdate, enddate, qa_status)
    • timeseries (annual data summary, qa status of years data)
  • Import script: https://github.com/HARPgroup/dh_weather/blob/main/src/sh/fn.nldas2-import
    • prototype
    • test
  • landseg feature attributes
    • properties
    • timeseries (annual data summary, qa status of years data)
  • landseg model attributes
    • properties
    • timeseries?
• program workflow steps
  • find feature
  • find model
  • create model if missing
  • check data table status
  • use REST to set table status info in vahydro
• test
• Compare against other data sets (noaa gridded) https://github.com/HARPgroup/vahydro/issues/666
• insert into P5 steps, and P6 steps
• Set up models
  • create models for p532 and p6 on landseg dh_features
  • annotate crucial events/props:
    • update all-time met data property "full_timeperiod" start date, end date, modified date (make sure this is updated each time)
      • QA status (date and time)

Data Model

• NLDAS data
  • import grid cells of interest as either raster or shape
  • set data to store overall download QA stuff
  • bundle = weather_sensor
  • ftype = nldas2
  • geometry = multipolygon
• Land segments:
  • land segment has model, 1 model for each version (p6, p532)
  • model met scenarios are stored as properties under "met_scenarios"
  • bundle = landunit
  • ftype = cbp532_landseg, cbp6_landseg
  • geometry = multipolygon
  • Template Model property: om-model-info/6791051

Workflow

Run met data workflow steps all the way, so this gets inserted after pre-populating the database runtime table.

• find feature https://github.com/HARPgroup/om/blob/master/drupal/om/src/dh_search_feature.php
• find model https://github.com/HARPgroup/om/blob/master/drupal/om/src/om_getpid.php
  • create one if missing (copy template) https://github.com/HARPgroup/om/blob/master/drupal/om/src/om_copy_subcomp.php
• use php to analyze met data https://github.com/HARPgroup/om/blob/master/php/src/met_qa.php
  • table_name
  • table_existgs (0 or 1)
    • does it have a table?
  • date_range (startdate, enddate property)
    • what is the date range?
  • any simple calculations to do?
    • TBD (l90_water_balance, l90_wb_year)
  • qa_status = overall, value = 0 or 1
  • return data in json
• Parse analysis json, send to REST
  • R

Cell and WDM QA

Single Cell csv QA

• test a single land segment N51165: ./make_met2date N51165 19840101 20230829 "2021 2022 2023" landseg
• test grid2land: i=N24013; ./grid2land.sh 2022010100 2022123123 /backup/meteorology /backup/meteorology/out/grid_met_csv $i
• take a look inside, search for 2022,12,30 and 2023,8
  • nano /backup/meteorology/out/lseg_csv/1984010101-2023082923/${i}.PRC

Verify data in WDM files

• Can use wdmexp command, whih will generate a summary,
  • met input:
    • Ex: cbp wdmexp met2date met met_N51073.wdm 1000 1984 2023
  • runoff SURO output:
    • Ex: cbp wdmexp vadeq_2021 land soyN51073.wdm 111 1984 2023

Loading /opt/model/p6/vadeq/hspf.config
Outputting to /opt/model/p6/vadeq/input/scenario/climate/met/met2date/
echo met_N51073.wdm,1984,2023,1000 | wdm2text
  program to write hourly ascii output from a wdm
 wdm name, start year, end year, dsn
 hourly average =    3.7912358137427338E-003
 annual average =    33.233973143268805

Check individual grid cells duration in CSV

gc=`./nldas_land_grids $i`
n=0
yr=2021
for i in $gc;do 
  if [ $n -gt 0 ]; then
    lrow=`tail -n 1 out/grid_met_csv/$yr/${i}zPP.txt`
    echo "$i: $lrow"
  fi
  n=$(($n+1))
done
# make the full met2date for this series
a2l_one 1984010101 2023082923 /backup/meteorology/out/grid_met_csv /backup/meteorology/out/lseg_csv N24031