Suggestion how to download big amount of photos

[rtrost]

Hi,

I am trying to download my photos from iCloud and so far I haven´t been successful to get them all. I have roughly 80.000 photos in the cloud, but at max 50% the script fails (connection refused by peer), and I wasn´t able to find a reason for it.

So my idea was to just re-run the script as it will check which photos already have been downloaded (which is pretty slow?) and just add the rest - which failed as well with I/O errors. The bug is listed in the issues as well, but I didn´t manage to install the fix, it broke my installation completely.

So given the situation that for some reason the connection breaks after few hours, how would you do it? It would be cool if I could somehow split the downloads, like "download everything from 2021" or so. Any ideas?

[AndreyNikiforov]

@rtrost with icloudpd erroring, do see new files appearing on your local drive? Are you getting them in acceptable rate (e.g. percentage of the size of your collection appearing on your local drive over time) with retrying?

icloudpd will not re-download files that are already on your drive, so running it many times has not harm if you are just downloading without deleting. Unreliable connectivity may be related to your inet provider or Apple throttling (my assumption based on "connection refused by peer" error).

Although we do not have capability to filter by date, I don't see how it can help you. You may see less throttling errors because you'll naturally have pauses between chunks, but overall from start to 80K photos you may be faster with error-then-retry approach.

[rtrost]

Thanks a lot for the fast reply.

do see new files appearing on your local drive?

No, when I get the IO error, there are no files. I don´t think it is a hardware issue, but I will try with a different hdd.

may be related to your inet provider or Apple throttling

I guess thats the case, as I monitored the internet connection and there was no disconnect. If apple is throtteling, it might be cool to have a an option to download a given amount of photos in addition what you already have.

--next INTEGER_RANGE

f.e. --next 5000

Let´s assume I have already downloaded 20K photos, icloudpd would run through those existing and counting from the moment there is a new one downloaded, count to 5000 and then stops. Next time, it would start after 25K and would add next 5000 images.

That would help in case Apple is limiting the API.

(Not sure if my use case is a common one, I do not want to sync, I just want to have a copy of the icloud photos as I am paranoid of losing them ;-))

[AndreyNikiforov]

@rtrost Although we don't know Apple throttling logic, if icloudpd struggles to download even one file , I doubt it is Apple throttle (note that icloudpd downloads one file at a time). If I follow that thinking path, then downloading in chunks would not help.

What cmd are you running? If you can use progress bar, you may be able to get some other insights (e.g. errors after 2G download)... Just thinking loud...

[rtrost]

Ok, I have to admit that the IO error actually IS a hardware issue :-(.

I have 53.000 photos downloaded now, and with re-runnig the command, I always get a few more. Thats fine, it takes time but it works :-).

My command is pretty basic, i just use --directory, --username and --password.

Thanks a million for that great tool!