iOS Shared Libraries

[timrettop]

Is there any suggestions on how to debug not seeing any SharedSync libraries when using
docker exec -it icloudpd-shared /usr/bin/icloudpd --list-libraries
with the icloudpd:testing docker container?

I'm looking to try out backing up a shared library to a separate backup location and am only seeing PrimarySync library.

I use iOS 16 and have shared a new album for testing (and have some that have been around for a while in my icloud library.

[timrettop]

Could it be that the PR here would fix that?
icloud-photos-downloader/icloud_photos_downloader#489

[boredazfcuk]

That’s the patch which I merge to enable shared library downloads. Without it, there is no --list-libraries command line option.

[khargy]

Not clear from the comments, does the latest version of the docker support Shared Libraries?

[boredazfcuk]

Not clear from the comments, does the latest version of the docker support Shared Libraries?

No, the version with the iOS16 shared library patches applied is: boredazfcuk/icloudpd:testing

I ran it for a while on my home server, but it broke after a couple of days. I don't even have shared libraries enabled on my account, so had to roll the code back to the previous version.

[khargy]

Not clear from the comments, does the latest version of the docker support Shared Libraries?

No, the version with the iOS16 shared library patches applied is: boredazfcuk/icloudpd:testing

I ran it for a while on my home server, but it broke after a couple of days. I don't even have shared libraries enabled on my account, so had to roll the code back to the previous version.

Thanks for the reply, I saw your note about the testing repository the other day. So far I haven't had any issues even with downloading from a large shared library.

[epheterson]

Did something change? Looks like this was working but stopped. I'm using icloudpd:latest (not testing, but have never used testing).

2022-12-12 13:16:30 INFO     iCloudPD launch command: /usr/bin/icloudpd --directory /home/(user)/iCloud --cookie-directory /config --folder-structure {:%Y/%m/%d} --username (email) --auto-delete --library SharedSync-(UUID)  2>/tmp/icloudpd/icloudpd_download_error
2022-12-12 13:16:31 ERROR    Failed to download new files
2022-12-12 13:16:31 ERROR     - Can you log into iCloud.com without receiving pop-up notifications?
2022-12-12 13:16:31 ERROR    Error debugging info:
2022-12-12 13:16:31 ERROR    Error: no such option: --library
2022-12-12 13:16:31 ERROR    ***** Please report problems here: https://github.com/boredazfcuk/docker-icloudpd/issues *****

[boredazfcuk]

Did something change? Looks like this was working but stopped. I'm using icloudpd:latest (not testing, but have never used testing).

The iOS shared library patches broke basic functionality for a lot of people (myself included):
#235

So I pulled the patches and added them to the testing container, as it seems that something isn't quite right with them.

[epheterson]

Interesting, I have two other personal library containers and both have been working (save one odd issue). I might move my container to testing, or wait for the code to be fixed and pushed back to latest. Thanks for the update!

[epheterson]

The odd issue I'm seeing is that some files will be downloaded then deleted on the same run, repeatedly. New files do seem to get added, so haven't really seen an issue with it. This never happened before the shared library changes, so might be related. It's still happening on the current latest tag, though, so maybe something else. Let me know if I should file a unique issue.

e.g.

2022-12-12 13:10:27 INFO     Generating list of files in iCloud. This may take a long time if you have a large photo collection. Please be patient. Nothing is being downloaded at this time
/home/username/iCloud/2021/03/19/IMG_8867.jpeg
2022-12-12 13:21:24 INFO     Downloading /home/username/iCloud/2021/03/19/IMG_8867.jpeg
2022-12-12 13:34:11 INFO     Deleting /home/username/iCloud/2021/03/19/IMG_8867.jpeg!

Logs: iCloudPD-logs.txt

[boredazfcuk]

I get this a lot. Have done for a long while. It's usually because I have saved files from an external app, such as Signal or Telegram, deleted them, and then realised I wanted them back, so saved them again (instead of recovering them from 'Recently Deleted'). This means I have two files with the same name. One in Photos and one in recently deleted.

When the script runs, it sees there is a photo in the photo stream named IMG_1234.HEIC (or whatever) and sees it does not exist on the local filesystem, so downloads it. After it finishes downloading, it scans the "Recently Deleted" folder, sees a file in there called IMG_1234.HEIC, sees that it matches a file in the photo stream and so removes it.

I usually just find the matching files in 'Recently Deleted' and remove them permanently.

[epheterson]

Thanks, that explains it. I think in my case this is happening because I used the merge duplicates feature in Photos app.

A simple fix, I think, would be to only delete a file iff it's in recently deleted and the same filename was not detected in the original sync / photo stream as a download/existing photo. This is kind of derailing the point of this thread, I can file a new ticket if we think this is worth doing.

Also seems like this will resolve itself once the items expire from Recently Deleted.

[timrettop]

Seeing you close this reminded me that I could never get the shared libary UUID to show. I just tried again using this command with the :testing branch and get an API error, any thoughts:

/ # icloudpd --list-libraries
iCloud username/email: [email protected]
2023-04-10 12:48:09 DEBUG Authenticating...
iCloud Password:
Traceback (most recent call last):
File "/usr/bin/icloudpd", line 33, in
sys.exit(load_entry_point('icloudpd==1.7.2', 'console_scripts', 'icloudpd')())
File "/usr/lib/python3.10/site-packages/click/core.py", line 722, in call
return self.main(*args, **kwargs)
File "/usr/lib/python3.10/site-packages/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/usr/lib/python3.10/site-packages/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/lib/python3.10/site-packages/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "/usr/lib/python3.10/site-packages/icloudpd-1.7.2-py3.10.egg/icloudpd/base.py", line 293, in main
libraries_dict = icloud.photos.libraries
File "/usr/lib/python3.10/site-packages/pyicloud_ipd/services/photos.py", line 275, in libraries
libraries[zone_name] = PhotoLibrary(
File "/usr/lib/python3.10/site-packages/pyicloud_ipd/services/photos.py", line 154, in init
request = self.service.session.post(
File "/usr/lib/python3.10/site-packages/requests/sessions.py", line 635, in post
return self.request("POST", url, data=data, json=json, **kwargs)
File "/usr/lib/python3.10/site-packages/pyicloud_ipd/base.py", line 100, in request
self._raise_error(code, reason)
File "/usr/lib/python3.10/site-packages/pyicloud_ipd/base.py", line 122, in _raise_error
raise api_error
pyicloud_ipd.exceptions.PyiCloudAPIResponseError: INTERNAL_ERROR (INTERNAL_ERROR)

[boredazfcuk]

The testing container was just something I put together from a PR that was created in the upstream package. It's still not been merged though. Extensive work has been done to my container to fix an issue relating to authentication in China that would prevent me from merging the patches that were in the testing branch, so that's kind of dead now.

If the feature is added upstream, then I'll accommodate it here, but until that's done, there's not much point in keeping this open.

[timrettop]

Ahh, the same error was found and reported in the upstream PR that was never merged.
icloud-photos-downloader/pyicloud#8 (comment)

[boredazfcuk]

The upstream project seemed like it had stalled for a while, but then a flood of updates came in January/February... It's what kind of got me re-interested in the project (as well as increased personal time to work on it).

It's slowed down somewhat over the past month, but hopefully they're still working on it.

[jcbshw]

FYI looks like Shared Library is working now. I never had success with the testing container previously but after seeing recent success in icloud-photos-downloader/pyicloud#8 I gave it another try.

Only issue I encountered is needing to run "chmod 750 /iCloud" to fix permissions but that's unrelated to Shared Library support.

[boredazfcuk]

That is the original PR which I merged into the testing container. It won't work on the latest version of the container as it's based on old code. If the PR gets rebased and merged upstream, I'll update the version bundled in the container and re-enable the functions. Until that happens though, the functionality will just have to stay in the testing container.

I will re-open this as a reminder though.

[tangyue1]

I would love to see the upstream code works with shared library. I noticed that only my personal library is being downlaoded now.

[laurensguijt]

Is there any progress with the shared librarys? I would love to use this for a specific shared library.

[lonevvolf]

Looks like there actually is - check here:
icloud-photos-downloader/icloud_photos_downloader#678

[tangyue1]

Really appreciate the hard work you put in. would like to see this feature added. For once it worked with the testing image but some update broke it.

[epheterson]

Somehow testing image is still working for me 🤷‍♂️ Also appreciate all the hard work and look forward to support in the main image someday!

[jcbshw]

Shared Library support was just merged upstream

icloud-photos-downloader/icloud_photos_downloader#678

[boredazfcuk]

Looks like it's broken things at the minute though: icloud-photos-downloader/icloud_photos_downloader#690

[boredazfcuk]

I've re-enabled the code which should support shared libraries again. I don't actually have any to test with though, so feedback would be appreciated.

[boredazfcuk]

This is what happens for myself:

/ # /opt/icloudpd_latest/bin/icloudpd --username [email protected] --cookie-directory ${config_dir} --domain com --directory /dev/null --list-libraries
2023-10-02 12:22:54 DEBUG    Authenticating...
SharedSync-########-####-####-####-############
PrimarySync

/ # /opt/icloudpd_latest/bin/icloudpd --username [email protected]  --cookie-directory ${config_dir} --domain com --directory /dev/null --list-libraries | sed '1d'
2023-10-02 12:28:47 DEBUG    Authenticating...
PrimarySync

libraries_with_dates is also commented out in sync-icloud.sh preventing folder structure to be followed

I've not implemented this yet. I don't use shared libraries, so wasn't sure if multiple shared libraries was a thing, which is why it was commented out.

Users with shared libraries would just have PrimarySync and SharedSync. It would be nice to have that uncommented so we can have the folder structure followed for the shared library.

Are you running an older version of the container? There was an upstream change which pushed info out to stderr instead of stdout. That could be what's happening... The DEBUG line may be getting pushed to stderr, so displayed in console, but ignored by sed.

I've pushed a new version which enables the configuration items for shared libraries with folder structure. Please test and let me know.

[boredazfcuk]

Tomorrow I will try both libraries, comma seperated in the config. Then the last thing missing would be (maybe as an option) to have all files from both libraries in the same folder structure (folder_structure={:%Y/%m/%d}).

Folder structure wasn't implemented for Shared Libraries. It is in the version I've just pushed now though.

[tangyue1]

I set the photo_library=SharedSync-xxx and folder structure is still not respected in the latest image. Can it NOT add the SharedSync-xxx folder and just create folder structure under the set volume?

[boredazfcuk]

Can it NOT add the SharedSync-xxx folder and just create folder structure under the set volume?

No, people wanting to download from multiple shared libraries would have their libraries merged.

If you want to have libraries with dates, you need to set libraries_with_dates=true, same as how downloading from photo albums is handled.

[tangyue1]

libraries_with_dates=True, photo_library=SharedSync-xxx, Stop condition is either until_found=100 or recent_only=100 The latest image won't download any photo.
2023-10-04 09:22:22 2023-10-04 09:22:22 INFO Downloading library: SharedSync-xxx
2023-10-04 09:23:50 2023-10-04 09:23:50 INFO Synchronisation complete for user

[boredazfcuk]

If you remove the stop conditions, does work?

I'm wondering if the new code to download the photos doesn't consider these settings.

[tangyue1]

If you remove the stop conditions, does work?

I'm wondering if the new code to download the photos doesn't consider these settings.

I removed the stop condition but it still does not download. Here is the debugging, "--library" seems to be invoked twice.

2023-10-04 17:45:11 2023-10-04 17:45:11 INFO Downloading library: SharedSync-xxx
2023-10-04 17:45:11 2023-10-04 17:45:11 DEBUG iCloudPD launch command: /opt/icloudpd_latest/bin/icloudpd --directory /home/user/iCloud --cookie-directory /config --domain com --username xxx --no-progress-bar --library SharedSync-xxx --folder-structure SharedSync-xxx/{:%Y/%Y-%m/%Y-%m-%d} --library SharedSync-xxx 2>/tmp/icloudpd/icloudpd_download_error
2023-10-04 17:45:11 2023-10-04 17:45:11 DEBUG Set owner on iCloud directory, if required
2023-10-04 17:45:31 2023-10-04 17:45:31 DEBUG Set group on iCloud directory, if required
2023-10-04 17:45:50 2023-10-04 17:45:50 DEBUG Set 750 permissions on iCloud directories, if required
2023-10-04 17:46:09 2023-10-04 17:46:09 DEBUG Set 640 permissions on iCloud files, if required
2023-10-04 17:46:28 2023-10-04 17:46:28 INFO Synchronisation complete for user

[boredazfcuk]

Can you update to latest version and test please?

[tangyue1]

It is working now. Listing individual libraries like "PrimarySync,SharedSync-########-####-####-####-############" works. But photo_library="all libraries" does not work.

[lonevvolf]

I am also not seeing the correct folder structure under the SharedSync-### folder in the newest image. It's still flat for me.

[boredazfcuk]

I am also not seeing the correct folder structure under the SharedSync-### folder in the newest image. It's still flat for me.

Did you configure it to download to a dated file structure?

[lonevvolf]

folder_structure={:%Y/%m/%d}

[boredazfcuk]

folder_structure={:%Y/%m/%d}

So no then? folder_structure is ignored by photo/library downloads, in favour of named folders, unless confgured otherwise.

[lonevvolf]

I guess I'm confused - the doc says that setting that value, or leaving it blank will download files into the date structure. How can I configure it to use the date structure then?

EDIT: Never mind - found the parameter you're referring to. It was incorrectly set.

[yieldhog]

Am I having the same issue? I've been using this for about a year or so .. all of my files were downloaded neatly into the default date folders: ie 2023 // 09 // 25

I added the following ENV to my docker container: photo_library: "PrimarySync,SharedSync-ECEXXX"

overnight, it redownloaded all of my 52,000 pictures into a new subfolder "PrimarySync" with no folder structure. What is the best option here? Thanks

[KUMAKAIHA]

Can it NOT add the SharedSync-xxx folder and just create folder structure under the set volume?

No, people wanting to download from multiple shared libraries would have their libraries merged.

If you want to have libraries with dates, you need to set libraries_with_dates=true, same as how downloading from photo albums is handled.

When I use 'libraries_with_dates=true' in Docker settings, whether I use Ture or true, it will still be set to false in the icloudpd.conf file

[KUMAKAIHA]

Can it NOT add the SharedSync-xxx folder and just create folder structure under the set volume?

No, people wanting to download from multiple shared libraries would have their libraries merged.
If you want to have libraries with dates, you need to set , same as how downloading from photo albums is handled.libraries_with_dates=true

When I use 'libraries_with_dates=true' in Docker settings, whether I use Ture or true, it will still be set to false in the icloudpd.conf file

In Docker version 1.0.665. Once photo_library is used, as libraries_with_dates cannot be set to true, photos will not be synchronized according to the file structure of the date. Meanwhile, due to the invalid setting of photo_library="all libraries", only photo_library= "PrimarySync,SharedSync-XXX" is allowed to synchronize personal libraries simultaneously. And due to the use of libraries_with_dates=false, the personal library will not be able to expand according to the date structure.
Currently, once synchronization of shared libraries begins, it will result in personal libraries being unable to be synchronized or being synchronized in the PrimarySync folder without expanding by date.

[boredazfcuk]

Maybe... I think it's not working because of the typo when it generates the conf file, it's something like: libraries_with_datess

I spotted it this morning but not near a computer for a while.

[KUMAKAIHA]

Maybe... I think it's not working because of the typo when it generates the conf file, it's something like: libraries_with_datess

I spotted it this morning but not near a computer for a while.

Like this ?
# if [ "${libraries_with_dates}" ]; then sed -i "s%^librariess_with_dates=.*%librariess_with_dates=${librariess_with_dates}%" "${config_file}"; fi

[boredazfcuk]

Yeah that's it. Turns out I've got the coronavirus, so just trying to sleep through the worst of it. Maybe pick this up in a few days as I don't have my laptop to alter the code.

If any of you guys want to submit a PR with the change I can approve it and kick off the build/deployment from my phone. Otherwise you're just gonna have to wait a few days till I'm good again.

[KUMAKAIHA]

Yeah that's it. Turns out I've got the coronavirus, so just trying to sleep through the worst of it. Maybe pick this up in a few days as I don't have my laptop to alter the code.

If any of you guys want to submit a PR with the change I can approve it and kick off the build/deployment from my phone. Otherwise you're just gonna have to wait a few days till I'm good again.

Wishing you a speedy recovery. I don't know much about GitHub and Docker, but I deleted the extra s in line 108 'librariess_with_dates' and submitted a PR

[tangyue1]

Wish you a speedy recovery! feel better!

[boredazfcuk]

Definitely on the mend, feeling loads better in last few hours. Should be back to normal by tomorrow.

[boredazfcuk]

I've pushed another update today. I found a bug in how albums are processed and made quite a few changes to that.

The Libraries functionality was basically a copy/paste job, so a few of these bugs were in there too. Hopefully it should work a little better now.

[m4rt0s]

Hi Boredazfcuk, first of all thank you for your great work!

My question is about synchronisation when using a shared library, is there a way to make it download both libraries in one folder instead of using separate folders?

Under photos folder I have right now this structure:

|── PrimarySync
│   ├── 2020
│   ├── 2021
│   ├── 2022
│   └── 2023
|── SharedSync-XXXX-XXX-XXXX-XXX-XXXXXX
│   ├── 2020
│   ├── 2021
│   ├── 2022
│   └── 2023

I would like it to look like this:

|── Photos
│   ├── 2020
│   ├── 2021
│   ├── 2022
│   └── 2023

Is there any possibility to merge the two folders into one?

Thanks in advance.

[boredazfcuk]

Hi,

Personally, I don't download from Photo Libraries, so I've no experience of how they behave... but I think you may run into issues attempting to merge the libraries like that.

I'd imagine it's entirely possible that two different photos exist in each library, but use the same name IMG_0000.HEIC, for example.

After the first file is downloaded from the PrimarySync, the second file from SharedSync will be de-duplicated, so named something like IMG_0000-12345.HEIC.

After this, say you decided to delete IMG_0000.HEIC from the SharedSync library, the app finds IMG_0000.HEIC in deleted items. It has no way of knowing which Library that file came from, so it would delete the file on disk named IMG_0000.HEIC, which would be the file from PrimarySync.

[boredazfcuk]

Resolved long ago... just doing some long overdue housekeeping.