Fix increasing write times with nc outputs #428
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Addresses issue #425 - Previously, when outputting to netCDFs, the bigger the domain, the longer it took to write output after every simulation run - Created a new output within `Input_nc/desc.in` - "deflateLevel" that specifies the level of deflation that will be sent into `nc_def_var_deflate()` * If the value is 0 (default), variables will not have any kind of deflation * Otherwise, the values 1-9 specify the intensity of deflation that will occur - New variable within SW_NETCDF to hold this new input - set to 0 in `SW_NC_init_ptrs()` - Added custom chunking to every non-dimensional variable (e.g., variables that are not time, vert, lat/lon, x/y, etc.) * Previously, the deflation would provide chunk sizes that encompass the entire domain - E.g., chunk size = [10, 10, 366] for a variable with xy = 10 x 10 sites and time = one leap year - The preferred chunk size for the example above should be chunk size = [1, 1, 366] - New function - `writeDummyVal` which is used to write a dummy value to a variable within the given netCDF and detects which type to write * Only use if `deflateLevel` is 0
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## release/devel_v8.1.0 #428 +/- ##
=====================================================
Coverage 73.33% 73.33%
=====================================================
Files 20 20
Lines 6233 6233
=====================================================
Hits 4571 4571
Misses 1662 1662 ☔ View full report in Codecov by Sentry. |
dschlaep
reviewed
Aug 7, 2024
There was a problem hiding this comment.
Very nice! I've tested this for a domain of size 1,000,000 with and without deflation. It is now steadily making progress, excellent!
I made two minor comments about improving documentation and an error message. Please take a look at those and then we can finish this up. Thanks!
dschlaep
approved these changes
Aug 7, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In the latest version of SOILWAT2 (v8.0.0), when the user creates a large domain, there is a noticeable increase in the runtime of individual simulations
SOILWAT2 now makes use of
deflateLevelSee nc-based SOILWAT2 runs slow down with increasing domain size #425 for more information