iCloud Photo Download Stuck Mid-Process

[farman007]

Hi,

I am downloading around 600GB+ of photos from iCloud, but the process gets stuck in the middle without showing any errors. I have approximately 40K files (a mix of videos and photos), and despite having a 100Mbps connection, I've only managed to download around 40GB in an entire day. When the process gets stuck, I have to close and restart the script, which resumes from the last downloaded point.

Below is the log, and you can see it gets stuck on the last file:

2024-07-10 00:13:38 INFO     Downloaded Z:\iCloudPy\2024\03\17\IMG_1329.HEIC
2024-07-10 00:13:38 DEBUG    Downloading Z:\iCloudPy\2024\03\17\IMG_1329_HEVC.MOV...
2024-07-10 00:13:41 INFO     Downloaded Z:\iCloudPy\2024\03\17\IMG_1329_HEVC.MOV
2024-07-10 00:13:41 DEBUG    Downloading Z:\iCloudPy\2024\03\17\IMG_1327.PNG...
2024-07-10 00:13:44 INFO     Downloaded Z:\iCloudPy\2024\03\17\IMG_1327.PNG
2024-07-10 00:13:44 DEBUG    Downloading Z:\iCloudPy\2024\03\17\IMG_1326.PNG...
2024-07-10 00:13:48 INFO     Downloaded Z:\iCloudPy\2024\03\17\IMG_1326.PNG
2024-07-10 00:13:48 DEBUG    Downloading Z:\iCloudPy\2024\03\17\RPReplay_Final1710620101.mp4...
  7%|####9                                                                     | 2703/40299 [18:27<72:40:29,  6.96s/it]

and when I restarted the process, I got below error:

2024-07-10 11:08:56 DEBUG    Z:\iCloudPy\2024\01\24\IMG_9726.PNG already exists
2024-07-10 11:08:56 DEBUG    Z:\iCloudPy\2024\01\24\IMG_9725.MOV already exists
2024-07-10 11:08:56 DEBUG    Z:\iCloudPy\2024\01\24\IMG_9720.HEIC already exists
2024-07-10 11:08:56 DEBUG    Z:\iCloudPy\2024\01\24\IMG_9720_HEVC.MOV already exists
2024-07-10 11:08:56 DEBUG    Z:\iCloudPy\2024\01\24\IMG_9718.HEIC already exists
2024-07-10 11:08:56 DEBUG    Z:\iCloudPy\2024\01\24\IMG_9718_HEVC.MOV already exists
Traceback (most recent call last):
  File "urllib3\response.py", line 444, in _error_catcher
  File "urllib3\response.py", line 831, in read_chunked
  File "urllib3\response.py", line 775, in _handle_chunk
  File "http\client.py", line 642, in _safe_read
http.client.IncompleteRead: IncompleteRead(8440 bytes read, 1800 more expected)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "requests\models.py", line 816, in generate
  File "urllib3\response.py", line 624, in stream
  File "urllib3\response.py", line 816, in read_chunked
  File "contextlib.py", line 158, in __exit__
  File "urllib3\response.py", line 461, in _error_catcher
urllib3.exceptions.ProtocolError: ('Connection broken: IncompleteRead(8440 bytes read, 1800 more expected)', IncompleteRead(8440 bytes read, 1800 more expected))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "starters\icloudpd.py", line 6, in <module>
  File "click\core.py", line 1157, in __call__
  File "click\core.py", line 1078, in main
  File "click\core.py", line 1434, in invoke
  File "click\core.py", line 783, in invoke
  File "icloudpd\base.py", line 645, in main
  File "icloudpd\base.py", line 1234, in core
  File "tqdm\std.py", line 1181, in __iter__
  File "pyicloud_ipd\services\photos.py", line 383, in photos
  File "pyicloud_ipd\services\photos.py", line 363, in photos_request
  File "requests\sessions.py", line 637, in post
  File "pyicloud_ipd\session.py", line 65, in request
  File "requests\sessions.py", line 589, in request
  File "requests\sessions.py", line 747, in send
  File "requests\models.py", line 899, in content
  File "requests\models.py", line 818, in generate
requests.exceptions.ChunkedEncodingError: ('Connection broken: IncompleteRead(8440 bytes read, 1800 more expected)', IncompleteRead(8440 bytes read, 1800 more expected))
[15276] Failed to execute script 'icloudpd' due to unhandled exception!
PS C:\Users\Farman Ullah Khan>

Is there any solution to prevent it from stopping so that it can download all the files without needing my constant monitoring?

Thank you.

[AndreyNikiforov]

There were a number of reports where downloading stuck in the middle (without even an error that you see). IIUC restarting helped to get through the hanging file(s). Are you getting past that large file?

Current hypothesis is that auth session expires in the middle of a long download and that is not handled gracefully.