Parallelizing download

[agenteo]

Hi, have you already explored parallelizing the files download? What did you learn? Cheers

[AndreyNikiforov]

Hi, have you already explored parallelizing the files download? What did you learn? Cheers

What bottleneck will it fix? What evidence is there to confirm it is indeed a bottleneck?

[agenteo]

Yeah I know what you mean. And that's not the case. My NIC Rx is around 300KB/s. I have a laptop dedicated to this download. And yes my ISP is above that. I bet Apple caps the download.

I had the same slow speed when I was manually downloading from iCloud years ago. One of the reasons I moving my stuff off iCloud.

Going back to my initial question about parallelizing. If someone knows the codebase it could be a fun experiment to test. I am a bit opaque on how this script works, ie. using an Apple API? Scraping iCloud?

[AndreyNikiforov]

Going back to my initial question about parallelizing. If someone knows the codebase it could be a fun experiment to test.

See #191 and search for thread in PRs & issues.

[AndreyNikiforov]

As a suggestion for your testing - run two instances of icloudpd. If combined speed ~doubles, it will support parallelizing downloading.

Also note, that downloading huge collection once is not the primary use case. Yes, everybody starts there, but after initial download it is only incremental updates that matter. If your icloudpd runs as a service and checks for updates every, say, 10min, then increased download speed would not have much impact.

[agenteo]

Going back to my initial question about parallelizing. If someone knows the codebase it could be a fun experiment to test.

See #191 and search for thread in PRs & issues.
Thanks for pointing to this. That's exactly the level of detail I was looking for. I must have missed when I searched the archives :)

And thanks for the tip about starting a new instance.

Also note, that downloading huge collection once is not the primary use case. Yes, everybody starts there, but after initial download it is only incremental updates that matter. If your icloudpd runs as a service and checks for updates every, say, 10min, then increased download speed would not have much impact.

Yes that makes sense. And Apple clearly doesn't make it easy for people to get their data out.

[agenteo]

Just leaving a note for posterity. I tried to run 2 icloudpd on my account but I ended up getting an IO error that mentioned running out of space (I wasn't). It looks like both instances were downloading the same file. It was worth a shot.