Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix memory usage for huge speed improvement #17

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

ydkhatri
Copy link

@ydkhatri ydkhatri commented Dec 1, 2023

Multiple copies of the input file data were being created, due to data being copied in a loop. In Python, the slice of a bytes buffer does not create a reference but a copy. Hence this was slow and inefficient.

By adding a memoryview(), this eliminates the repeated copying of data making it much faster.
On my system, a 100MB file, previously took 142 seconds to process, now only takes 21 seconds.
So this is almost 7 times faster now, the actual speed variation though will depend on the file size, larger the file, better the performance (compared to earlier!). This is now better on memory and speed.

@Schramp
Copy link

Schramp commented Dec 8, 2023

Looks fine to me.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants