process directory entries asynchronously

[whyrusleeping]

This does all the file stats and opens for a directory in async. On my 700MB/s m.2 SSD, its not any faster, but for disks with slower (non-zero) lookup times, i imagine this might be a little faster, especially for lots of smaller files.

@davidar could you test this out and let me know if its any better?

License: MIT
Signed-off-by: Jeromy [email protected]

[rht]

Tested on local SSD.

#prepare the files once for both cases
mkdir -p foo && for i in `seq 999`; do dd if=/dev/urandom of=foo/$i bs=1M count=1; done

time ipfs add -r foo
rm -r ~/.ipfs/blocks && ipfs init -f  # "hard zero-ing"

Current master:

ipfs add -r foo 12.55s user 2.92s system 5% cpu 4:31.69 total
ipfs add -r foo 12.86s user 2.92s system 4% cpu 5:49.27 total
ipfs add -r foo 12.55s user 2.92s system 4% cpu 5:47.31 total

feat/async-files:

ipfs add -r foo 12.39s user 2.87s system 5% cpu 4:32.64 total
ipfs add -r foo 12.59s user 2.94s system 4% cpu 5:54.86 total

[jbenet]

(@rht can use https://github.com/jbenet/go-random-files for this sort of test)

so this means it's not meaningful?

[whyrusleeping]

yeah, he tested on an SSD too though, i'll set up ipfs machine on my raid later and try there to see how it affects rusty metal disks.

[whyrusleeping]

alright, tried it on a 5400rpm disk. didnt get any speedup. closing.

[rht]

For ref, time ipfs get $hash: 4.24s user 1.92s system 13% cpu 46.096 total.
The slow add needs to be fixed.

[davidar]

@whyrusleeping oops, sorry, I meant to test this myself, thanks for doing it for me (getting a dev build onto Pollux is a little bit painful)

The slow add needs to be fixed.

:+100: