--only-print-filenames option displays filenames of files that have already been downloaded

[boredazfcuk]

Hi,

I've noticed that if you use the --only-print-filenames option to display the list of files that will be downloaded, it shows files which are already on disk:

/ $ /usr/bin/icloudpd --directory /home/user/iCloud --cookie-directory /config --username [email protected] --folder-structure {:%Y/%m} --only-print-filenames
/home/user/iCloud/2020/09/IMG_1693_HEVC.MOV
/home/user/iCloud/2020/09/IMG_1691_HEVC.MOV
/home/user/iCloud/2020/09/IMG_1685_HEVC.MOV
/home/user/iCloud/2020/06/IMG_1646_HEVC.MOV
/home/user/iCloud/2020/05/IMG_1606_HEVC.MOV
/home/user/iCloud/2020/05/IMG_1605_HEVC.MOV
/home/user/iCloud/2020/05/IMG_1577_HEVC.MOV
/home/user/iCloud/2019/03/IMG_0331_HEVC.MOV
/home/user/iCloud/2019/03/IMG_0245_HEVC.MOV
/home/user/iCloud/2019/03/IMG_0244_HEVC.MOV
/ $ ls /home/user/iCloud/2020/09/IMG_16*
IMG_1685.HEIC      IMG_1687.MOV       IMG_1691.HEIC      IMG_1692.PNG       IMG_1693_HEVC.MOV  IMG_1696.PNG       IMG_1699.PNG
IMG_1685.JPG       IMG_1688.MOV       IMG_1691.JPG       IMG_1693.HEIC      IMG_1694.PNG       IMG_1697.PNG
IMG_1685_HEVC.MOV  IMG_1689.MOV       IMG_1691_HEVC.MOV  IMG_1693.JPG       IMG_1695.PNG       IMG_1698.PNG

This option should return an empty list if all files are already downloaded. However, it does not.

I've noticed that all of the above files are 'Live Photos', if that helps?

Thanks.

[menkej]

Strange... Not all the files, p.e. IMG_1646_HEVC.MOV are in the ls, though... What happens after the next "normal" download run?

[boredazfcuk]

IMG_1646_HEVC.MOV is in the /home/user/iCloud/2020/06 directory. I have listed the /home/user/iCloud/2020/09 directory which only contains the first three files. All files already exist on the drive.

It just never sees the files even through they are already downloaded, just looks like this:

/ # /usr/bin/icloudpd --directory /home/user/iCloud --cookie-directory /config --username [email protected] --folder-structure {:%Y/%m}
2020-11-01 17:19:36 DEBUG    Authenticating...
iCloud Password:
2020-11-01 17:20:05 DEBUG    Looking up all photos and videos from album All Photos...
2020-11-01 17:20:05 INFO     Downloading 79 original photos and videos to /home/user/iCloud ...
/home/user/iCloud/2019/02/IMG_0137.MOV already exists.: 100%|##############################################################################################################################################| 79/79 [00:03<00:00, 20.02it/s]
2020-11-01 17:20:09 INFO     All photos have been downloaded!
/ # /usr/bin/icloudpd --directory /home/user/iCloud --cookie-directory /config --username [email protected] --folder-structure {:%Y/%m} --only-print-filenames
iCloud Password:
/home/user/iCloud/2020/09/IMG_1693_HEVC.MOV
/home/user/iCloud/2020/09/IMG_1691_HEVC.MOV
/home/user/iCloud/2020/09/IMG_1685_HEVC.MOV
/home/user/iCloud/2020/06/IMG_1646_HEVC.MOV
/home/user/iCloud/2020/05/IMG_1606_HEVC.MOV
/home/user/iCloud/2020/05/IMG_1605_HEVC.MOV
/home/user/iCloud/2020/05/IMG_1577_HEVC.MOV
/home/user/iCloud/2019/03/IMG_0331_HEVC.MOV
/home/user/iCloud/2019/03/IMG_0245_HEVC.MOV
/home/user/iCloud/2019/03/IMG_0244_HEVC.MOV
/ # /usr/bin/icloudpd --directory /home/user/iCloud --cookie-directory /config --username [email protected] --folder-structure {:%Y/%m}
2020-11-01 17:20:48 DEBUG    Authenticating...
iCloud Password:
2020-11-01 17:20:55 DEBUG    Looking up all photos and videos from album All Photos...
2020-11-01 17:20:56 INFO     Downloading 79 original photos and videos to /home/user/iCloud ...
/home/user/iCloud/2019/02/IMG_0137.MOV already exists.: 100%|##############################################################################################################################################| 79/79 [00:03<00:00, 20.00it/s]
2020-11-01 17:21:00 INFO     All photos have been downloaded!
/ # /usr/bin/icloudpd --directory /home/user/iCloud --cookie-directory /config --username [email protected] --folder-structure {:%Y/%m} --only-print-filenames
iCloud Password:
/home/user/iCloud/2020/09/IMG_1693_HEVC.MOV
/home/user/iCloud/2020/09/IMG_1691_HEVC.MOV
/home/user/iCloud/2020/09/IMG_1685_HEVC.MOV
/home/user/iCloud/2020/06/IMG_1646_HEVC.MOV
/home/user/iCloud/2020/05/IMG_1606_HEVC.MOV
/home/user/iCloud/2020/05/IMG_1605_HEVC.MOV
/home/user/iCloud/2020/05/IMG_1577_HEVC.MOV
/home/user/iCloud/2019/03/IMG_0331_HEVC.MOV
/home/user/iCloud/2019/03/IMG_0245_HEVC.MOV
/home/user/iCloud/2019/03/IMG_0244_HEVC.MOV
/ #

[menkej]

Ok, I have a fix for that ready, but need to clean up my fork until I can submit a pull request, first. Might take some days, but I think this issue is not so serious. Thanks for pointing it out!

[boredazfcuk]

Awesome. Thanks.

I'm in no rush for you to submit the PR... In my container, I use the output of the --only-print-filenames command to test if the files are actually on disk, if they are, I just ignore them. That way I only trigger a download run if files are missing.

Once your PR is merged, I'll remove the verification check from my script.

[AndreyNikiforov]

I'm in no rush for you to submit the PR... In my container, I use the output of the --only-print-filenames command to test if the files are actually on disk, if they are, I just ignore them. That way I only trigger a download run if files are missing.

@boredazfcuk what is the reason to check separately from downloading (==two phase approach)? If there are no new files, single- and two-phase approaches will result in the same behavior and speed. If there are new files, then two-phase approach will require pulling list of files from icloud and checking disk twice, so it is slower and more work. I am definitely missing something...

[boredazfcuk]

The reason I was checking separately from downloading was due to this issue, which I hit quite a bit: #163

The single-phase approach always detects new files. Attempting to download all 800+ files in my photo stream was quite time consuming. I presumed that this would also increase load on the Apple server due to it checking 800+ files.

In the two-phase approach, retrieving the list of files using --only-print-filenames was comparatively quick and I could use its output to verify if the 'new' files were genuinely new. Ignoring the previously downloaded files meant far fewer 'download' runs.

Once I'd implemented this, it resulted in fewer private db access disabled for this account. Please wait a few minutes then try again. The remote servers might be trying to throttle requests. (ACCESS_DENIED) lockouts.