Change crop_gribs to operate on files as they arrive #191
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Use watchdog to detect when grib files are moved into place and crop them immediately instead of waiting until collect_weather finishes. crop_gribs is now launched concurrently with collect_weather.
Replaced the 'on_moved' event with the 'on_closed' event because file system events that occur when HRDPS files are moved from downloads directory to atmospheric forcing tree are create, several modify events, and close. That is presumably because the files are being moved from one files system to another, so the move is actually a copy/delete.
|
Tested successfully on
|
Pass grib_fcst_dir to observer.schedule() as string rather than Path to make type checking happy. Path works fine, but schedule() type hints don't acknowledge that. Improve comments and delete commented out code. Correct super() call in handler on_closed() method override. Add debug log message to show number of grib files left to be processed. This is to try to get insight into the worker finishing its job but not terminating.
Refactored `next_workers` to pass the forecast date as part of the arguments list to the 'crop_gribs' worker. This is necessary because `crop_gribs 06` is normally launched before midnight. So, it needs to be explicitly given the next day's date. Tests were updated to reflect this change.
Significantly updated the 'crop_gribs' file event handling tests. This change introduces two new tests: 'test_crop_expected_file' and 'test_ignore_unexpected_file', ensuring that the worker correctly processes only the expected GRIB files and properly ignores others. The 'test_constructor' was also updated to validate the new config argument.
Changed the type of the config parameter in both the _crop_grib_file and _write_ssc_grib_file functions from dict to a more specific py:class:`nemo_nowcast.Config`. This update provides improved type checking in IDEs.
crop_gribs now operates on files as they are created in the atmospheric forcing GRIB directory tree. So, it must be started before the download_weather worker.
Codecov Report
@@ Coverage Diff @@
## main #191 +/- ##
==========================================
+ Coverage 77.41% 77.58% +0.17%
==========================================
Files 133 133
Lines 18483 18601 +118
Branches 1953 1967 +14
==========================================
+ Hits 14309 14432 +123
+ Misses 4119 4111 -8
- Partials 55 58 +3
Flags with carried forward coverage won't be shown. Click here to find out more.
📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
re: concurrency of crop_gribs and collect_weather workers.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Use watchdog to detect when grib files are moved into place and crop them immediately instead of waiting until collect_weather finishes. crop_gribs is now launched concurrently with collect_weather.