Workflow geo

[rburghol]

• geo - process a given dataset into a time series for a single geographic coverage preferably with a usgs gage to assess with)
  • Exec:
    • Use: run_model raster_met [scenario, note:method is in scenario config] [coverage hydrocode] [work directory] geo [run steps=all]
    • Scenario con needs to set GEO_MET_MODEL and should be either simple_lm or storm_volume
    • Ex: run_model raster_met PRISM usgs_ws_01665500 auto geo
  • Config:
    • geo.config (meta model reformats from [scenario].con): https://github.com/HARPgroup/meta_model/blob/main/models/raster_met/geo/geo.config
      • [scenario].con, ex: PRISM.con: https://github.com/HARPgroup/cbp_wsm/blob/p61/config/control/met/PRISM.con
    • $GEO_MET_MODEL: the selected analysis model is set as a variable in the config file for the model scenario
    • $COVERAGE_PRECIP_FILE: $NLDAS_ROOT/[datasource]/out/[hydrocode]-[datasource]-all.csv
    • $COVERAGE_FLOW_FILE: $MODEL_EXPORT_DIR/usgs/out/[hydrocode]-flow.csv
    • $DAILY_PRECIP_FILE: $NLDAS_ROOT/[datasource]/out/[hydrocode]-[datasource]-weekly.csv
    • $WEEKLY_PRECIP_FILE: $NLDAS_ROOT/[datasource]/out/[hydrocode]-[datasource]-weekly.csv
    • $USGS_GAGE: what gage to use for this coverage? Since this is only run for gage coverages, the gage id is implicit in the hydrocode of the coverage, so, we don't need to pass this in. However, in some future world, we might want this support, and a data model will be required.
    • $RATING_FILE: $MODEL_EXPORT_DIR/met/[datasource]/out/[hydrocode]-[datasource]-[method]-ratings.csv
    • $RATING_TS_FILE: $MODEL_EXPORT_DIR/met/[datasource]/out/[hydrocode]-[datasource]-[method]-rating-ts.csv
  • download Generate the CSV data files to analyze
    • 01_coverage_precip: Makes an area weighted precipitation timeseries for the given coverage and dataset summary from the requested raster dataset
      • Creates $NLDAS_ROOT/[datasource]/out/[hydrocode]-[datasource]-all.csv
      • Only generates data for dates passed by user from config file
      • Ex: /opt/model/meta_model/run_model raster_met PRISM usgs_ws_02031000 auto geo download
      • Generate CSV: calc_raster_ts
        • Use: calc_raster_ts [hydrocode] [varkey] [outfile] [dbhost] [dbname] [remove sql file=y/n]
        • Ex: /opt/model/meta_model/run_model raster_met PRISM usgs_ws_02031000 auto geo download
    • 02_obtain_flow: download a flow dataset for the corresponding USGS gage
      • calls https://github.com/HARPgroup/HARParchive/blob/master/HARP-2024-2025/functions/usgsdata.R
      • Create file CSV: $MODEL_EXPORT_DIR/usgs/out/[hydrocode]-flow.csv
      • Only downloads data for the dates passed by user from config file
      • insure that data is summed to daily timestep
      • download gage info to include drainage area
      • TODO: Add column to flow file containing the drainage area in square miles
    • copy: get copy of data on local storage
  • process
    • Output/Intermediate Files:
      • Weekly precip $NLDAS_ROOT/[datasource]/out/[hydrocode]-[datasource]-weekly.csv
    • 01_precip_cfs - Add columns such to convert precip_in to cfs given the watershed size
      • Operates on file: $NLDAS_ROOT/[datasource]/out/[hydrocode]-[datasource]-all.csv
    • 02_daily_data - create a file with all data summarized at a weekly mean value
      • Takes data from file: $NLDAS_ROOT/[datasource]/out/[hydrocode]-[datasource]-all.csv
      • Creates file: $NLDAS_ROOT/[datasource]/out/[hydrocode]-[datasource]-daily.csv
      • See Workflow step geo -> process -> 02_daily_data: Rscripts to produce daily values for Precipitation and Flow at a given gage #63
    • 03_join_precip_flow - add usgs flow column tp precip file Workflow step geo -> process -> 03_join_precip_flow #69
    • 04_weekly_data - create a file with all data summarized at a weekly mean value
      • Takes data from file step 02: $NLDAS_ROOT/[datasource]/out/[hydrocode]-[datasource]-daily.csv
      • Creates file: $NLDAS_ROOT/[datasource]/out/[hydrocode]-[datasource]-weekly.csv
      • See Workflow step geo -> process -> 04_weekly_data: Create R script to produce weekly mean value file #61
    • 05_xxxx to 99_xxxx: additional processing files needed by individual methods
  • analyze: create outputs to describe precip dataset and its fidelity by coverage
    • Output/Intermediate Files:
      • $MODEL_EXPORT_DIR/met/[datasource]/out/[hydrocode]-[datasource]-[method]-ratings.csv
    • 02_model_XXXX: Accuracy/Preference rating for each time period,
      • Where XXXX is a unique identifier based on the method selected
      • Info to store: location, datasource, time-period (hour or start/end)
      • Create file CSV: $MODEL_EXPORT_DIR/met/[datasource]/out/[hydrocode]-[datasource]-[method].csv
    • 03_met_summary: develop summaries of our coverage meteorology data and add to db via REST REST / RomDataSource, metrics and analysis HARParchive#1354
    • 04_plot_xxxx to 07_xxxx open slots for extra visualizations if needed.
    • 08_create_rating
      • generates a CSV with time periods and ratings, must have columns: start_date, end_date, rating
      • $MODEL_EXPORT_DIR/met/[datasource]/out/[hydrocode]-[datasource]-[method]-ratings.csv
      • rating column has a number between 0.0-1.0 which shows accuracy.