Change rivers runoff file generation for SalishSeaCast NEMO v202111 #150
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Source was `tools/I_ForcingFiles/Rivers/DailyRiverFlows.py` where Susan developed the module by extracting functions from her development notebook for generating runoff forcing files for SalishSeaCast NEMO v202111 runs. I plan to develop and refactor the module further here on the way to refactoring the `make_runoff_file` worker for production runs of v202111.
Source was `tools/I_ForcingFiles/Rivers/ProductionDailyRiverNCfile.ipynb`: Susan's development notebook for generating runoff forcing files for SalishSeaCast NEMO v202111 runs. I plan to develop and refactor the notebbok further here on the way to refactoring the `make_runoff_file` worker for production runs of v202111.
douglatornell
added
documentation
|
@SusanEAllen, looking at the changes in # Templates for the runoff forcing file names
# **Must be quoted to project {} characters**
file templates:
- b201702: 'R201702DFraCElse_{:y%Ym%md%d}.nc'
+ b202108: 'R202108Dailies_{:y%Ym%md%d}.nc'
prop_dict modules:
- b201702: salishsea_tools.river_201702
+ b202108: salishsea_tools.river_202108Did we have multiple river files (>1 file name template) and propery dict modules at some point? If so, do we need to preserve that possibility, or can we collapse that to file template: 'R202108Dailies_{:y%Ym%md%d}.nc'
prop_dict module: salishsea_tools.river_202108 |
That file described the Fraser River climatology separation from measured flow at Hope. Fraser discharge in v202111 is based fully on observations, no climatology needed now.
|
Let's go with just the dailies. We might need to run some old files while we are still running 201905 but not going forward. |
That file contained the climatology for all water sheds other than the Fraser River. v202111 runoff is based fully on observations, no climatology needed now.
Runoff forcing is connected to model bathymetry, and bathymetry version for 202111 is 212108.
Rivers property dict is connected to model bathymetry, and bathymetry version for 202111 is 212108.
hpfx is supposed to be higher bandwidth, and perhaps more reliable due to fewer users. Added config for a 2nd sr_subscribe instance for HRDPS in hopes of reducing total download time (if bandwidth isn't saturated).
Codecov Report
@@ Coverage Diff @@
## main #150 +/- ##
==========================================
+ Coverage 74.12% 74.71% +0.58%
==========================================
Files 127 129 +2
Lines 16805 17344 +539
Branches 1189 1218 +29
==========================================
+ Hits 12457 12958 +501
- Misses 4298 4333 +35
- Partials 50 53 +3
Flags with carried forward coverage won't be shown. Click here to find out more.
📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
In notebook: * import `make_runoff_files` from `nowcast.daily_river_flows` * change config file path to `../config/nowcast.yaml` * change date to calculate runoff file for to 6-Feb-2023 to avoid Missing Snohomish River obs on 23-Jan-2023 * generalize date handling in calculation of runoff forcing file names for comparison figure In `daily_river_flows` module: * Handle extra fields in some limes of .csv files that trigger `pandas .ParserError` by adding `_parse_long_csv_line()` and `on_bad_lines` args to `pandas.read_csv()` calls * Handle path in grid repo clone for coordinates file in dev envs that don't have access to `/SalishSeaCast/`
Bumps [cryptography](https://github.com/pyca/cryptography) from 39.0.0 to 39.0.1. - [Release notes](https://github.com/pyca/cryptography/releases) - [Changelog](https://github.com/pyca/cryptography/blob/main/CHANGELOG.rst) - [Commits](pyca/cryptography@39.0.0...39.0.1) --- updated-dependencies: - dependency-name: cryptography dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Make it consistent with template in https://ubc-moad-docs.readthedocs.io/en/latest/github_notebooks_readme.html
Bad linebreak characters and missing commas, probably from manual edits.
* remove spaces from file name to that it is properly processed by make_readme.py * correct layout of title and description cell at top of notebook * correct Markdown style re: space after heading markup
We no longer need the possibility of having multiple runoff file name templates for different bathymetries, flatten the config dict.
Also, simplify prop_dict module in config because we no longer need the possibility of having multiple prop_dict modules for different bathymetries. So, flatten the config dict.
Use functools.partial() to reduce repeated code. Add unit tests for .csv reading.
Warning is related to pandas-dev/pandas#49279 though my testing shows that the warning is raised in spite of the lines with extra fields being processed as specified.
Extract repeated code into a function and use inplace=True.
Also add some type hints in docstrings.
Added unit tests and type hints in docstring.
Added unit tests and type hints in docstring.
Renamed function from patch_gaps(). Added unit tests and type hints in docstring.
Unnecessary because invalid flux value is detected by checking numpy.isnan(flux).
Added unit tests and type hints in docstring.
Added unit tests and type hints in docstring.
Added unit tests and type hints in docstring.
Added unit test and type hints in docstring.
GHA pytest-coverage workflow has gird clone at `../../grid/` relative to daily_river_flows.py.
_get_area() uses file from grid repo that is too annoying to make available in Actions workspace.
This was done on 25-Apr-2023 to avoid disruption due to hpfx server maintenance. However, missing file messages and duplicated files issues in HRDPS downloads that started with the change to the continental rotated lat-lon grid in Feb-2023 disappeared with the change to dd.weather.gc.ca. (cherry picked from commit ceddac3)
To avoid disrupting automation for v201905 production.
To avoid disrupting automation for v201905 production.
To avoid disrupting automation for v201905 production.
To avoid disrupting automation for v201905 production.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Use day averaged discharges (lagged 1 day) for all watersheds in addition to the Fraser River watershed.
Add processing to handle missing river discharge observations by persistence or scaling of nearby gauged river.
Prototype code was developed by @SusanEAllen.