Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Read files in chunks instead of all at once #13

Open
discentem opened this issue Jun 22, 2022 · 1 comment
Open

Read files in chunks instead of all at once #13

discentem opened this issue Jun 22, 2022 · 1 comment
Labels
AWS S3 FR/Bugs related to AWS S3 store implementation Azure Blob Storage Google Cloud Storage FR/Bugs related to GCS store implementation

Comments

@discentem
Copy link
Owner

Is it "better" if we read in chunks instead of all at once?

Example: https://github.com/discentem/pantri_but_go/blob/main/stores/local/local.go#L91
@natewalck
Copy link
Collaborator

We should absolutely read in chunks for all use cases when possible (downloading, uploading and hashing).

@discentem discentem added the LocalStore FR/Bugs related to local store plugin implementation label Jun 29, 2022
@discentem discentem added the AWS S3 FR/Bugs related to AWS S3 store implementation label Oct 19, 2022
@discentem discentem added Google Cloud Storage FR/Bugs related to GCS store implementation and removed LocalStore FR/Bugs related to local store plugin implementation labels May 11, 2023
@discentem discentem changed the title Consider reading files in chunks instead of all at once Read files in chunks instead of all at once Jun 5, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
AWS S3 FR/Bugs related to AWS S3 store implementation Azure Blob Storage Google Cloud Storage FR/Bugs related to GCS store implementation
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@natewalck @discentem