Speed up uploading/downloading of file and directories.

[serhiy-storchaka]

Before optimization:
1 minute for neuro storage cp -r neuromation storage:
1 minute for neuro storage cp -r storage:neuromation /tmp

After optimization:
36 seconds for neuro storage cp -r neuromation storage:
34 seconds for neuro storage cp -r storage:neuromation /tmp

For comparison:
From 22 seconds to 1 minute 16 seconds for neuro storage load -r neuromation storage:
18 seconds for neuro storage load -r storage:neuromation /tmp
7 seconds for neuro storage cp neuromation.tar storage:
2 seconds for neuro storage cp storage:neuromation.tar /tmp

[codecov]

Codecov Report

Merging #958 into master will increase coverage by 0.01%.
The diff coverage is 92%.

@@            Coverage Diff             @@
##           master     #958      +/-   ##
==========================================
+ Coverage   92.08%   92.09%   +0.01%     
==========================================
  Files          37       37              
  Lines        4243     4251       +8     
  Branches      643      642       -1     
==========================================
+ Hits         3907     3915       +8     
  Misses        227      227              
  Partials      109      109

Impacted Files	Coverage Δ
neuromation/api/storage.py	98.26% <92%> (+0.04%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update cf1a4ca...00c23a7. Read the comment docs.

[asvetlov]

LGTM.
The PR worth changelog record

[dalazx]

@serhiy-storchaka curious, in case of load, what's the overhead of spawning a job, provided a local docker image is present already?

[asvetlov]

what's the overhead of spawning a job, provided a local docker image is present already?

About 20 secs IIRC

[dalazx]

18 seconds for neuro storage load -r storage:neuromation /tmp
does that include launching a job?

[serhiy-storchaka]

Yes, it is a total time.