2022/04/12-26 Work Session:

[rburghol]

Overview

Big tasks to complete

• Final Presentation
• QA
  • Get flagging and summary stats script running on terminal
  • Make sure they are functionalized^ and easy to use
  • Implement ability to focus on grids after figuring out which land segments are wrong
  • Look at cells we know are unusual and try to figure out the root of the problem
• Retrieve recent data (@rburghol and @jdkleiner )
• Mashups Mix and match Meteorology to project the upcoming summer #141
  • We need to fudge the dates... (i.e., 2002 becomes 2020...)
  • are get_lseg_data() and get_lseg_csv() duplicative?
• Onboarding
  • Move onboarding material into github folder (especially generic material)
  • Github accounts for new analysts
  • deq2 accounts for new analysts
  • DEQ Vahydro access accounts for new analysts

Collaboration Goals

• Put images and code snippets in front of your collaborators at the conclusion of every work session that you engage in.  Thus, budget your time so that the last 15 minutes can be spent sharing, rather than just getting that last little tweak made. 
  • One way to promote script sharing is to load library functions from a URL, rather than from a local file path. This is appropriate once a given function is fairly stable.
  • Let me start. Here is a code snippet that you can use (you may need to install the data.table library).

# load the lseg met getter from a github repo. 
# Note: I got this link URL by browsing to the script file I wanted and clicking on "raw"
source("https://raw.githubusercontent.com/HARPgroup/HARParchive/master/HARP-2021-2022/get_lseg_data.R")

tmet <- get_lseg_data("N51820")

# analyze it
quantile(tmet$dfPRC$PRC, probs=c(0,0.1,0.25,0.5,0.75,0.8,0.9,0.95,1.0))

# plot it. 
# Note: your plots window needs to be wide or it will give you "Error in plot.new() : figure margins too large"
plot(tmet$dfPRC$PRC)

Script Use Documentation

• Add "Use" and "Ex" code snippets for every step in our work flow here: Updated Scripts to Add Meteorology Data set model_meteorology#29
• What + Why Kate mentioned that the slideshow is currently focused more on "why" than "what" -- next steps are to enhance the "what".  We know that the DEQ folks who will use this know the "why" better than most, and while some of the "why" will help them to think about how to creatively apply the tools that we are developing, they want to see the "what": this slideshow must be about what we have accomplished.  
• Single Met Files I had some recollection that we had generated files with all met data for a land segment in a single file, with one column for each data element. During a brief search, I found that we had a file with summary data for each year, but not the entire time series --
  • I want to make sure that the script that creates this file gets documented in our work flow Updated Scripts to Add Meteorology Data set model_meteorology#29
  • Doing a non-summary version for all hourly data is not something we need to do right now (unless someone sees a real reason for it),
  • The annual summary files could be useful for @snhandelsman work on her class project.
  • These annual summary files will also have utility in the "what" portion of the presentation.
  • Here is an example file: http://deq1.bse.vt.edu:81/met/out/lseg_csv/1984010100-2020123123/N51800SummaryStats.csv
• R Library Needs The Lowe's brought this up, extremely important, and glad you are taking this step. Question: What R libraries are not in place to run the scripts on deq2?  Do you need my/Joey's help getting them installed? Let's create a github issue to track this deployment.

Downloading NLDAS2 Data

• The scripts don't do more than a year at a time (it will crash at the end of the year). Do we need something that will automate this for a series of years? Have we already done something like this? Let's figure these out, since we need it to download the bad segment, and also to do our weekly/monthly downloads during drought analysis.

[jdkleiner]

@rburghol Thanks Rob for organizing this summary issue, this looks great and these are appropriate items to focus on as the analysts wrap up the current phase of the HARP project.

Love the code snippet, I was able to analyze some lseg data in a flash, the more examples like this the better!

[alexwlowe]

Hey team,

Here's a code snipet that I have been using to pull the phase 5 and phase 6 land segment list files:

#example of reading in phase 6 land segment list
p5_landseg_list <- scan(file = "https://raw.githubusercontent.com/HARPgroup/HARParchive/master/GIS_layers/p5_landsegments.txt", what = character())
#example of reading in phase 5 land segment list
p6_landseg_list <- scan(file = "https://raw.githubusercontent.com/HARPgroup/HARParchive/master/GIS_layers/p6_landsegments.txt", what = character())

We can use these lines of code for loops that run through every land segment in a phase, and now we only have to continuously update the files sourced in the command!

[alexwlowe]

Update 4/14/22

I cleaned up the rolling averages plot workflow. I moved the functions to lseg_functions.R and changed it to make it much more user friendly. If anyone wants to look at it and test it out, it's get_rolling_avg_workflow.R

[rburghol]

@alexwlowe this is fantastic.

• I tested it on my laptop and it runs great.
• I made some tweaks to get_rolling_avg_workflow.R to download the full time period, I am making a pull request and will mark you as a reviewer, so then you can decide to merge it if you like the changes. Please review at your earliest convenience, I put some notes in there in terms of some coding choices that are important to leverage our group work.
• As you can see below, not everything is 100% OK with the data :) -- unless of course I did something wrong in my script tweaks. But from my view, some time after 2010 the precip values go way down for the cell x393y97.

All in all another step in the right direction, visualization = QA. I would check many of these cells with this full time period version.

[alexwlowe]

Big tasks to complete (moved to issue)

• Get flagging and summary stats script running on terminal
• Make sure they are functionalized^ and easy to use
• Implement ability to focus on grids after figuring out which land segments are wrong
• Look at cells we know are unusual and try to figure out the root of the problem
• move onboarding material into github folder (especially generic material)