Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Parallelize file transfer for external blobs #367

Open
shenshan opened this issue May 6, 2021 · 1 comment
Open

Parallelize file transfer for external blobs #367

shenshan opened this issue May 6, 2021 · 1 comment

Comments

@shenshan
Copy link

shenshan commented May 6, 2021

Feature Request

Problem

Fetching from an internal blob field and from external blob fields have big difference in performance.

Same to this issue on the python site: datajoint/datajoint-python#806

Requirements

We should consider parallelizing the file downloads from external storage.

Benchmark

Here is a benchmark plotting showing the performance difference.
This is for princeton database, running on a computer in the lab. The external storage on a shared drive in the institute.

image

@shenshan shenshan added enhancement awaiting-triage Issues awaiting proper priority/milestone assignment. labels May 6, 2021
@guzman-raphael guzman-raphael removed the awaiting-triage Issues awaiting proper priority/milestone assignment. label May 7, 2021
@guzman-raphael guzman-raphael added this to the Release 3.5.0 milestone May 7, 2021
@shenshan
Copy link
Author

shenshan commented Jun 7, 2021

Screen Shot 2021-06-04 at 12 08 06 PM

Screen Shot 2021-06-04 at 11 38 29 AM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants