Crop HRDPS continental GRIB files to SalishSeaCast sub-domain after they are downloaded #173
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
New default operation processes cropped GRIB files produced by crop_gribs worker. --full-continental-grid option enables processing of HRDPS continental grid files as received from ECCC/MSC datamart/hpfx servers.
From https://github.com/ecmwf/cfgrib/README.rst: By default cfgrib saves the index of the GRIB file to disk appending .idx to the GRIB file name. Index files are an experimental and completely optional feature, feel free to remove them and try again in case of problems. Index files saving can be disabled passing adding indexpath='' to the backend_kwargs keyword argument.
Either poorly/undocumented behaviour of GRIB file format, or a bug in `cfgrib.xarray_to_grib.to_grib()` (which is, admittedly, alpha/experimental) causes the lon/lat arrays in GRIBs cropped using `cfgrib` to a sub-domain that encloses the SalishSeaCast NEMO domain to be incorrect; see https://nbviewer.org/github/SalishSeaCast/analysis-doug/blob/main/notebooks/continental-HRDPS/crop-grib-to-SSC-domain.ipynb.ipynb
Codecov Report
@@ Coverage Diff @@
## main #173 +/- ##
==========================================
+ Coverage 75.80% 76.01% +0.20%
==========================================
Files 127 129 +2
Lines 16916 17195 +279
Branches 1198 1214 +16
==========================================
+ Hits 12824 13070 +246
- Misses 4041 4076 +35
+ Partials 51 49 -2
Flags with carried forward coverage won't be shown. Click here to find out more.
... and 1 file with indirect coverage changes 📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
Use inspect.getfile() instead of relative path to ensure that worker modules are found, regardless of working directory that tests are run in.
It's too noisy about GRIB attributes at WARNING.
|
Deployed branch on |
00Z forecast is normally downloaded on evening of previous day (local time). So, get the forcast date to run crop_gribs for from the GRIBs directory path in the checklist.
time is somehow getting into dataset from cropped GRIB files so using full_grid condition wasn't good enough.
Changing the config["weather"]["download"][resolution]["file template"] key to "ECCC file template" for grib_to_netcdf and crop_gribs didn't get propogated to collect_weather and download_weather.
cfgrib.xarray_to_grib.to_grib() insists on calling ACPC air_temperature. So, we fix it.
grib_to_netcdf, after refactor to use HRDPS continental grid subdomain that is cropped to just what we need to the SalishSeaCast domain, has a much smaller memory demand. So, the dask cluster on salish is no longer required.
|
Automation finally worked without my intervention to launch forecast2 and nowcast runs on 7-Apr. Ready to merge 😁 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
See
https://nbviewer.org/github/SalishSeaCast/analysis-doug/blob/main/notebooks/continental-HRDPS/crop-grib-to-SSC-domain.ipynb.ipynb
for the long, complicated story of how this had to be implemented.