feat: improved fs uploads performance #2333
Conversation
grumbach
commented
Oct 28, 2024
•
grumbach
commented
- Parallel file uploading with capped concurrency to avoid memory burst
- Parallel chunk uploading with capped concurrency to avoid memory burst
- Local testing reveals this PR improves upload performance of CLI directories upload by x50 FIFTY!!
- Configurable batching for files and chunks through env vars
autonomi/src/client/fs.rs
Outdated
|
|
||
| use super::archive::{Archive, ArchiveAddr}; | ||
| use super::data::{DataAddr, GetError, PutError}; | ||
|
|
||
| pub(crate) const FILE_UPLOAD_BATCH_SIZE: usize = 128; |
There was a problem hiding this comment.
What's the rationale behind 128? Do we need batching at all?
There was a problem hiding this comment.
I tried with unlimited and my memory useage spiked to the moon.
So it seems you were right about batching, and we probably need some. Local testing revealed that 128 was a sweet spot (at least on my machine), not too high for mem, not too slow.
There was a problem hiding this comment.
I do worry a little about the relative performance (e.g. PC vs Raspberry Pi). Can we relate this value to CPU core count?
https://doc.rust-lang.org/std/thread/fn.available_parallelism.html
|
Good one! What about a couple of tests? To check file integrity and probably we can profile in the future. |
grumbach
force-pushed
the
upload_performance
branch
from
75b67c5 to
e8e16b7
Compare
.github/workflows/merge.yml
Outdated
| - name: Run autonomi tests | ||
| timeout-minutes: 25 | ||
| run: cargo test --release --package autonomi --lib --features="full,fs" | ||
|
|
||
| - name: Run autonomi tests |
There was a problem hiding this comment.
i think the same workflow is repeated twice! Run autonomi tests
Maybe some rebase conflicts?
There was a problem hiding this comment.
Good catch! I'm actually sneaking this back in as it was removed as a side effect of:
#2334
grumbach
force-pushed
the
upload_performance
branch
from
e8e16b7 to
ade585f
Compare
grumbach
force-pushed
the
upload_performance
branch
from
12b57ad to
5d4f926
Compare
