Avoids memory issues with big files while extracting archives #2192
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
GitHK
added
bug
Codecov Report
@@ Coverage Diff @@
## master #2192 +/- ##
========================================
+ Coverage 73.0% 76.9% +3.8%
========================================
Files 464 249 -215
Lines 17866 10362 -7504
Branches 1759 1026 -733
========================================
- Hits 13045 7969 -5076
+ Misses 4365 2071 -2294
+ Partials 456 322 -134
Flags with carried forward coverage won't be shown. Click here to find out more.
|
odeimaiz
approved these changes
Mar 4, 2021
pcrespov
requested changes
Mar 4, 2021
There was a problem hiding this comment.
I was expecting tests to be in services/web/server/tests/unit/isolated/test_exporter_archiving.py. Can you please provide some? thx
services/web/server/src/simcore_service_webserver/exporter/archiving.py
Outdated
Show resolved
Hide resolved
services/web/server/src/simcore_service_webserver/exporter/archiving.py
Outdated
Show resolved
Hide resolved
pcrespov
reviewed
Mar 4, 2021
services/web/server/src/simcore_service_webserver/exporter/archiving.py
Outdated
Show resolved
Hide resolved
added 6 commits
March 4, 2021 14:09
…re-forked into fix-zip-memory-issues
- fixes an error with with high memory consumption
pcrespov
approved these changes
Mar 4, 2021
|
|
||
| @pytest.mark.parametrize( | ||
| "compress,store_relative_path", | ||
| [[True, True], [True, False], [False, True], [False, False]], |
There was a problem hiding this comment.
FYI: check itertools.product to create cartesian products
|
|
||
| async def unarchive_dir(archive_to_extract: Path, destination_folder: Path) -> None: | ||
| with zipfile.ZipFile(archive_to_extract, mode="r") as zip_file_handler: | ||
| with ProcessPoolExecutor() as pool: |
There was a problem hiding this comment.
aha, now i see where that it gets in a pool, OK
This was referenced Mar 24, 2021
Closed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What do these changes do?
Unzipping was taking way too much RAM, could cause the webserver to crash while importing big projects.
nodeports memory consumption while unarchiving will be lowered with this fix. Nodeports compression has also been removed to enhance performance.
Related issue/s
also related to #2171 regarding the jupyter-labs OOM due to pour nodeports not chunking files while extracting them but doing it all in memory.
How to test
Checklist