Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor SEG-Y export to optimize memory usage. #109

Merged
merged 46 commits into from
Nov 26, 2022

Conversation

tasansal
Copy link
Collaborator

@tasansal tasansal commented Oct 11, 2022

closes #34

ref: Dask Discourse

Proposed graph for 3D below. 2D or 4D is similar, with less or more hierarchy.

There are a few minor changes not related to export.

image

@tasansal tasansal added the enhancement New feature or request label Oct 11, 2022
@tasansal tasansal self-assigned this Oct 11, 2022
@tasansal tasansal force-pushed the feature/refactor_segy_export branch 2 times, most recently from b21e4a7 to 0ced705 Compare October 18, 2022 14:41
@codecov-commenter
Copy link

codecov-commenter commented Oct 25, 2022

Codecov Report

Merging #109 (7397304) into main (6f95fa5) will increase coverage by 0.26%.
The diff coverage is 84.76%.

@@            Coverage Diff             @@
##             main     #109      +/-   ##
==========================================
+ Coverage   81.45%   81.72%   +0.26%     
==========================================
  Files          42       43       +1     
  Lines        1564     1647      +83     
  Branches      171      178       +7     
==========================================
+ Hits         1274     1346      +72     
- Misses        256      264       +8     
- Partials       34       37       +3     
Impacted Files Coverage Δ
src/mdio/segy/_standards_rev0.py 0.00% <0.00%> (ø)
tests/integration/test_segy_import_export.py 100.00% <ø> (ø)
src/mdio/__init__.py 81.81% <50.00%> (-18.19%) ⬇️
src/mdio/converters/segy.py 75.36% <50.00%> (-1.92%) ⬇️
src/mdio/segy/_workers.py 22.95% <66.66%> (-15.88%) ⬇️
src/mdio/segy/byte_utils.py 80.48% <69.23%> (-3.89%) ⬇️
src/mdio/segy/utilities.py 84.84% <78.57%> (-6.07%) ⬇️
src/mdio/converters/mdio.py 71.42% <85.71%> (-8.58%) ⬇️
src/mdio/segy/blocked_io.py 89.13% <87.09%> (-1.04%) ⬇️
src/mdio/segy/creation.py 94.31% <94.64%> (+3.62%) ⬆️
... and 4 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

@tasansal tasansal force-pushed the feature/refactor_segy_export branch 2 times, most recently from d3a2ed5 to 820fe0c Compare October 26, 2022 14:24
@tasansal tasansal force-pushed the feature/refactor_segy_export branch from 576b89d to f10ba40 Compare November 2, 2022 22:24
@tasansal
Copy link
Collaborator Author

tasansal commented Nov 3, 2022

This has significant improvements to memory usage:

dask/distributed#7128

import dask
import distributed

with dask.config.set({"distributed.scheduler.worker-saturation": "1.0"}):
    client = distributed.Client(...)

@tasansal tasansal force-pushed the feature/refactor_segy_export branch from 29580f6 to b87a1ba Compare November 7, 2022 18:58
@tasansal tasansal force-pushed the feature/refactor_segy_export branch from bfd1b1d to 3ed44a1 Compare November 15, 2022 03:58
@tasansal tasansal force-pushed the feature/refactor_segy_export branch from 6d582d3 to c644890 Compare November 18, 2022 22:00
@tasansal tasansal force-pushed the feature/refactor_segy_export branch from 47ea835 to 5f7eb40 Compare November 19, 2022 03:26
@tasansal tasansal marked this pull request as ready for review November 19, 2022 03:28
@tasansal tasansal requested a review from srib November 19, 2022 03:28
@tasansal
Copy link
Collaborator Author

@srib can you take a look when you have a chance, the entire export logic changed here, would be good to have a second set of eyes.

@tasansal tasansal merged commit 48c1817 into main Nov 26, 2022
@tasansal tasansal deleted the feature/refactor_segy_export branch November 26, 2022 13:54
@tasansal tasansal removed the request for review from srib November 28, 2022 16:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Reduce memory consumption during SEG-Y export
2 participants