Daily variables for atmospheric demand #338
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 30 commits
December 24, 2022 16:50
- Moved cloud cover, wind speed, and relative humidity to SW_SKY - Added same variables listed above to SW_WEATHER_NOW - Updated `scaleAllWeather()` parameters to include scaling information for cloud cover, wind speed, and relative humidity and added parameter descriptions - Updated calls to `solar_radiation()` and `petfunc()` to take in the "now" value for the required value
- Moved contents of `SW_SKY_init_run()` to `_read_weather_hist()` and deleted `SW_SKY_init_run()` due to no longer doing anything - Removed call to `SW_SKY_init_run()` in `SW_CTL_init_run()` - Updated documentation of `scaleAllWeather()` to mention cloud cover, relative humidity, and wind speed - Added cloud cover, wind speed, and relative humidity to `_clear_hist_weather()` and `SW_WTH_new_day()`
- Due to SW_WEATHER_HIST now containing cloud cover, relative humidity, and wind speed, "TimesTest.InterpolateMonthlyValues" now uses SW_WEATHER_HIST to test `interpolate_monthlyValues()`
- Added call to `Time_new_year()` in `SW_MDL_construct()` to set yearly information for old sky monthly values to use when being interpolated in `_read_weather_hist()` - `SW_WTH_new_day()` now correctly gets daily values from the correct day, the first value in the daily arrays must be skipped
- The previous location of `Time_new_year()` in `SW_MDL_construct()` was before "SW_Model.year" was initialized - The new location is within `_read_weather_hist()` to update day/month values to correspond to the current year being read in for use in the interpolation of monthly input values
- Adjusted `SW_WTH_new_day()` to use current day instead of (day + 1) - Updated `interpolate_monthlyValues()` to adjust for base0 for cloud cover, wind speed, and relative humidity - Since *_daily[] values in `allHist` are defaulted to SW_MISSING, we can test to see if "dailyValues[]" is cloud cover, wind speed, or relative humidity and adjust accordingly
- Tests on lines 91 and 116 in "test_Times.cc" no longer need to test for an unchanged value in the first spot in "cloudcov_daily", and now must be an interpolated value - "doy" can no longer go up to the number of days in a year, but must be one less due to being base0
- Instead of relying on the first element within "dailyValues", `interpolate_monthlyValues()` now takes in an argument specifying if "dailyValues" is to be base1 or base0 - Added new boolean variable "interpAsBase1" inside functions where `interpolate_monthlyValues()` is called - "Times.h" no longer includes "SW_Defines.h" as with this change, it is no longer needed
- `interpolate_monthlyValues()` tests now use "interpAsBase1" - Adjusted tests more to work with base0 (e.g., doy2mday(doy) to doy2mday(doy + 1))
- Added two new sections for flags within "weathsetup.in": - Three flags for the user to specify if they would like to use monthly values instead of daily input - Fourteen flags for the user to specify if they are using daily input for a specific variable - Note: if both a monthly and daily flag are set to 1 for the same variable, the monthly flag will take priority - Monthly flags default to 1 and daily flags aside from minimum/maximum temperature and precipitation, default to 0 - Added "actual vapor pressure" and "shortwave radiation" to scaling factors to be consistent with the rest of the new input variables - Both scaling factors default to 1.0
SW_WEATHER: - Added variables to hold input flags for each possible input variable - Added variables to hold indices where each input variable may reside when reading in weath.YYYY - Added "n_input_forcings" to keep track of how many daily input columns exist within weath.YYYY SW_WEATHER_HIST/SW_WEATHER_NOW: - Added two new variables: shortwave radiation and actual vapor pressure `solar_radiation()`: - Removed "rel_humidity" and "air_temp_mean" as parameters - Added "e_a" as a parameter and removed "e_a" calculation to be moved
- Added three new functions to calculate "actual vapor pressure": `actualVaporPressure1/2/3()` - Functions currently do not return anything useful
- Create a new function skeleton with pseudocode, `checkAllWeather()` - This function will through all weather of all years and days in the simulation and checks if they are reasonable values
- Updated equations in `actualVaporPressure2()` and `actualVaporPressure3()` along with the function documentation to mention them
- Created two arrays to hold input flags and index variables for SW_WEATHER for index calculation - Expanded switch statement to read in the new lines where the flags reside - Expanded `sscanf()` call to read actual vapor pressure and shortwave radiation scaling factors - Added five ternary operators to make sure if both monthly and daily flags are set to 1, the monthly flag takes priority - End of function calculates which index of "weathInput[]" in `_read_weather_hist()` will hold which variables in weath.YYYY
- Expanded `sscanf()` call to take in 15 variables (day of the year, min/max temperature, precipitation, and the new input variables) - Added calculation conditionals for wind speed, relative humidity, actual vapor pressure - Put `interpolate_monthlyValues()` calls in conditionals so they are only called when monthly values are desired to be used
- `checkAllWeather()` now loops through all years and days in the simulation and tests: - relative humidity, cloud cover, actual vapor pressure, temperature max/min, precipitation, shortwave radiation, and wind speed - `checkAllWeather()` also checks if the flags for maximum/minimum temperature and precipitation are set and fail if they are not - Added call to `finalizeAllWeather()` after scaling occurs
- `scaleAllWeather()` now scales actual vapor pressure and shortwave radiation using the new scaling columns in "weathsetup.in" - Updated function header and documentation to include actual vapor pressure and shortwave radiation scaling parameters
- Abandoned idea to set the value of SW_WEATHER flags through an array - Seemingly correct method of doing so would result in "Illegal Instruction: 4" on local machine, almost immediately and permanently eliminating the idea - `SW_WTH_setup()` now individually sets variable indices - Renamed `variableIndices` to `varIndices` to compact the name
- Moved interpolation of monthly values to before `_read_weather_hist()` reads weather values so relative humidity can be available if `actualVaporPressure()` requires it and program is only using monthly values * Cloud cover and wind speed interpolation just follow the movement of relative humidity interpolation - Corrected first and second index for respective max/min temperature data - `actualVaporPressure()` now takes in "yearWeather->r_humidity_daily[doy]" to get the current daily data, interpolated or read-in
- Added shortwave radiation and actual vapor pressure to update the current daily value in `SW_WTH_new_day()` - The setting of flags in `SW_WTH_setup()` now use `itob()` to translate 0/1 to a Bool
- With the new crash conditions in `SW_WTH_new_day()` WaterBalanceTest.WithWeatherGeneratorForSomeMissingValues and WaterBalanceTest.WithWeatherGeneratorOnly would result in crashes - These tests would skip `_read_weather_hist()` resulting in SW_MISSING as daily values, so it has been fixed by setting the daily values to 0 before `SW_CTL_main()` is called
- Replaced the previous setting of "yearWeather->r_humidity_daily" with a new calculation instead of setting it to the respective "weathInput" index/value - Moved previous hurs calculation within actual vapor pressure section to dew point section for further possible calculation of hurs
added 4 commits
January 27, 2023 00:11
- If a value of relative humidity, actual vapor pressure, and/or wind speed is calculated, we now prevent all cases from using SW_MISSING - On the SOILWAT2 side, this is mainly for the future and we cannot hold the chance of using SW_MISSING for any weather variables aside from max/min temperature and precipitation - On rSOILWAT2, this is needed to not calculate any given SW_MISSING values that come with the dataset of "daymet" does not have December 31st on leap years
- On a previous commit, the #include for `SW_Sky.h` was added to `SW_Weather.h` - Since `SW_Weather.h` does not have any dependencies from the included file, the best place for the #include is within `SW_Weather.c` - Weather tests rely on "SW_Sky", so upon the mentioned change, `test_SW_Weather.cc` now includes `SW_Sky.h`
- Previously, the newest weather test subroutines would reset the weather prefix path - This left room for different orders of testing to unexpectedly crash due to existing data - To solve that, we now call `Reset_SOILWAT2_UnitTest()` to reset everything and eliminate any possibility of using data from a previous test
- Weather tests currently test the use of daily values instead of monthly and their calculations - Now, the subroutines for "daymet", "gridmet", and "maca" also explicitly expect daily calculated relative humidity and/or wind speed values instead of monthly - Adjusted comments for expected fail of `_read_weather_hist()` in "ReasonableValuesAndFlags" subroutine to correctly explain what will be on the next line(s)
dschlaep
commented
Jan 27, 2023
There was a problem hiding this comment.
Nice job!
Our checks are currently failing on GHA for windows/cygwin. Maybe some of our new tests still cause side effects?
[----------] 1 test from VegEstabTest
[ RUN ] VegEstabTest.SimulateWithVegEstab
@ generic.c LogErrormake: *** [makefile:344: test_run] Error 255
Error: Process completed with exit code 2.
It would also be great to add a test that checks that generateMissingWeather() does indeed successfully fill in missing values in the new daily variables using the LOCF method.
|
Running our tests in an infinite loop with random order (until one crashes) may point to the same issue that causes our For instance, I get // commit e440352 // Run tests in an infinite loop with random order (until one crashes) // For instance, here a few crashes
or |
added 2 commits
January 28, 2023 15:06
- New test suite "DailyLOCF" - Tests that the function, `generateMissingWeather()`, performs LOCF on new daily weather input variables when they are SW_MISSING and the weather generator method is 1 - Specifically covers the first year cloud cover, actual vapor pressure, and wind speed
- Most weather tests now reset SOILWAT2 before they exit to make sure the next test that's called isn't using the modified globals the current test is using - "VegProd" tests in need, call `finalizeAllWeather()` Reasons for the additions of `finalizeAllWeather()` in VegProd tests: * `SW_WTH_new_day()` has previously gained the ability to test new daily weather input, so if any daily value in those variables is "SW_MISSING" the program crashes * When the monthly flag for humidity is set, we rely on actual vapor pressure being calculated in `finalizeAllWeather()` which is not called, thus actual vapor pressure is not calculated and `SW_WTH_new_day()` crashes
- turn off monthly flags, i.e., don't use mean monthly inputs of cloud cover, wind speed, and humidity (and instead use daily inputs) - reset `allHist` data structure before reading daily values - expect that these datasets provide daily radiation values - expect that daily cloud cover values are missing - fix MACA example: 7th column of input files contain radiation (and not vapor pressure) inputs
- new `desc_rsds` with
* 0: `rsds` represents daily global horizontal irradiation [MJ / m2]
* 1: `rsds` represents flux density [W / m2] for a
(hypothetical) flat horizon averaged over an entire day (24 hour period)
* 2: `rsds` represents flux density [W / m2] for a
(hypothetical) flat horizon averaged over the daylight period of the day
- input file "weathsetup.in" gains input for `"desc_rsds"`
- struct "SW_WEATHER" gains element `desc_rsds`
- `SW_WTH_setup()` now reads updated input file "weathsetup.in" and populates `desc_rsds` of `SW_Weather`
- assign correct value for `SW_Weather.desc_rsds` in tests for DayMet, gridMET, and MACA data sets to complete specifications -- even if `SW_Weather.desc_rsds` is currently unused
… cloud cover - previous behavior * surface irradiation is estimated from extraterrestrial radiation (if observed downward surface shortwave radiation rsds is not available) * cloud effects are estimated as part of atmospheric attenuation using observed cloud cover - additional new behaviors: * cloud effects are ignored if surface irradiation is estimated from extraterrestrial radiation and if observed cloud cover is not available * surface irradiation is derived from observed downward surface shortwave radiation rsds, if available, based on Allen et al. 2006 * cloud cover is estimated from observed radiation and expected cloud-less radiation if observed radiation is available and if observed cloud_cover is not available -> cloud cover is required as argument for `petfunc()` - `solar_radiation()` gained arguments `rsds` (observed radiation) and `desc_rsds` (unit and definition descriptor of `rsds`) - `solar_radiation()` changed input argument `cloud_cover` to input and/or output argument `*cloud_cover` (output generated if cloud cover estimated from observed radiation) - new `actual_horizontal_transmissivityindex()` to calculate observed transmissivity index - `petfunc()` now fails if `cloudcov` is missing - `SW_WTH_new_day()` now requires that at least one cloud cover and radiation inputs is not missing (instead of requiring that cloud cover is never missing) - additional tests in suite "SW2SolarRadiationTest.global" * as previously: test without observed radiation: missing `rsds`; `H_gh` calculated * new: test with previously calculated `H_gh` and missing cloud cover -> returned cloud cover matches what was used to calculate input radation * new: test with observed radiation `rsds` and missing cloud cover - new test "WaterBalanceTest.WithGRIDMET" * uses gridMET example inputs (for two years) * reads daily inputs (and all mean monthly variables, i.e., humidity, cloud cover, wind speed, are turned off) * uses observed radiation to estimate missing cloud cover; estimated cloud cover is then used by `petfunc()`
This was referenced Feb 6, 2023
… into read_more_daily_input
- provided more informative message patterns for new death tests -- utilizing updated `logError()` functionality from commit 8f33a54 - fixed error message typo in `checkAllWeather()`: "minumum" -> "minimum"
…diation Feature observed solar radiation - process daily radiation inputs if available - handle different specifications of radiation inputs - utilize observed daily radiation if available - estimate cloud cover as the factor causing differences between observed and expected daily radiation
|
- moved code chunk that was checking and updating daily (meteorological) input flags from `SW_WTH_setup()` to dedicated new `check_and_update_dailyInputFlags()` -> rSOILWAT2 is utilizing the same code from its own `onSet_SW_WTH_setup()` equivalence to `SW_WTH_setup()
- previous commit d211ccf called `check_and_update_dailyInputFlags()` with incorrect arguments ...
- moved code chunk that was setting column indices for and counting of daily (meteorological) input flags from `SW_WTH_setup()` to dedicated new `set_dailyInputIndices()` -> rSOILWAT2 is utilizing the same code from its own `rSW2_readAllWeatherFromDisk()` equivalence to `SW_WTH_read()
- previously, the call to `readAllWeather()` mixed up monthly use flags for wind speed and humidity -> also fix unit tests for "DailyWeatherInputTest.DailyDayMet" which incorrectly had monthly wind speed use turned off
- previously, unit tests run a simulation only with "standard" inputs and with "gridMET" inputs -> now run tests all supported datasets -> Daymet unit tests currently fail due to radiation values being out of reasonable range
- previously, `solar_radiation()` converted radiation inputs as flux density during daylight with a factor that was too large by the factor pi - previously, the documentation wrote out the correct factor but the implemented value was incorrect - additional and better documented checks to make sure that input radiation and calculated radiation have reasonable values -> simulation run unit tests with Daymet inputs are now passing (see previous commit eaa6484)
- previously, `generateMissingWeather()` with method = 2, i.e., weather generator, was replacing tmax, tmin, and ppt (with generated values) if any of the eight variables (including the five new ones) are missing -> now, `generateMissingWeather()` with method = 2, i.e., weather generator, is replacing tmax, tmin, and ppt (with generated values) if any of tmax, tmin, and ppt are missing (ignoring the ones not considered by the weather generator) - clarify function documentation
- previously, `mvnorm()` was producing at times a NaN value for minimum air temperature - for instance, "rSOILWAT2" was failing on some (but not all computers) with > https://github.com/DrylandEcology/rSOILWAT2/actions/runs/5157599324/jobs/9290219836 > ('test_WeatherGenerator_functionality.R:125:7'): Weather generator any(is_missing_weather(wdf)) is not FALSE -> this was caused by a value pair of `s` and `wTmin_var` of 99.264050000 and 99.264050000 that resulted in both `GT(s, wTmin_var)` and `EQ(wTmin_var, s)` being FALSE and `vc11` was calculated as the square root of a negative number -> the fix implemented with this commit sets `vc11` to 0 if `LE(wTmin_var, s)`
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Handle daily inputs of SOILWAT2 daily driving variables
-> close #341