Not able to bootstrap the container on macOS Ventura 13.5.2/13.6.0 #391

walnutist · 2023-09-22T04:00:50Z

Description:
I am trying to deploy the docker-icloudpd on a macOS server for downloading multiple users' photos timely.
And the container launch failed as below:

docker run  -it\
        --privileged \
        --network bridge \
        --restart=always \
        --env user=hank \
        --env user_id=501 \
        --env group=staff \
        --env group_id=20 \
        --env force_gid=True \
        --env apple_id="<MY_ID>" \
        --env authentication_type=2FA \
        --env folder_structure={:%Y/%m} \
        --env auto_delete=True \
        --env synchronisation_interval=43200 \
        --env icloud_china=True \
        --env auth_china=True \
        --env TZ="Asia/Shanghai" \
        --volume $(pwd)/config:/config \
        --volume $(pwd)/photos:/home/root/iCloud \
        boredazfcuk/icloudpd

2023-09-22 11:49:45 INFO     ***** boredazfcuk/icloudpd container for icloud_photo_downloader v1.0.628 started *****
2023-09-22 11:49:45 INFO     ***** For support, please go here: https://github.com/boredazfcuk/docker-icloudpd *****
2023-09-22 11:49:45 INFO     Alpine Linux 3.18.2
2023-09-22 11:49:45 INFO     Python version: 3.11.5
2023-09-22 11:49:45 INFO     Loading configuration from: /config/icloudpd.conf
2023-09-22 11:50:13 INFO     Apple ID: <MYID>
2023-09-22 11:50:13 INFO     Authentication Type: MFA
2023-09-22 11:50:13 INFO     Cookie path: /config/<MYID>
2023-09-22 11:50:13 INFO     Cookie expiry notification period: 7
2023-09-22 11:50:13 INFO     Download destination directory: /home/<MYID>/iCloud
2023-09-22 11:50:13 INFO     Download directory does not exist.
2023-09-22 11:50:13 INFO     Creating /home/<MYID>/iCloud and configuring permissions.
2023-09-22 11:50:13 INFO     Folder structure: {:%Y/%m}
2023-09-22 11:50:13 INFO     Synchronisation interval: 43200
2023-09-22 11:50:13 INFO     Synchronisation delay (minutes): 0
2023-09-22 11:50:13 INFO     Set EXIF date/time: false
2023-09-22 11:50:13 INFO     Auto delete: true
2023-09-22 11:50:13 INFO     Delete after download: false
2023-09-22 11:50:13 INFO     Photo size: original
2023-09-22 11:50:13 INFO     Single pass mode: false
2023-09-22 11:50:13 INFO     Skip download check: false
2023-09-22 11:50:13 INFO     Skip live photos: false
2023-09-22 11:50:13 INFO     Number of most recently added photos to download: Download All Photos
2023-09-22 11:50:13 INFO     Downloading photos from album: Download All Photos
2023-09-22 11:50:13 INFO     Stop downloading when prexisiting files count is: Download All Photos
2023-09-22 11:50:13 INFO     Live photo size: original
2023-09-22 11:50:13 INFO     Skip videos: false
2023-09-22 11:50:13 INFO     Convert HEIC to JPEG: false
2023-09-22 11:50:13 INFO     Downloading from: icloud.com.cn
2023-09-22 11:50:13 INFO     Authentication domain: cn
2023-09-22 11:50:13 INFO     Ignore Synology extended attribute directories: Disabled
2023-09-22 11:50:13 ERROR    Group id, 20, already in use by the group: staff - exiting. If you must to add your user to this pre-existing system group, please set the force_gid variable to True

I don't understand why the script says the force_gid is False even it was already explicitly set to True

The text was updated successfully, but these errors were encountered:

walnutist · 2023-09-22T04:09:28Z

If I change the command below, different exception throwed:

docker run -it\
	--privileged \
	--network bridge \
	--restart=always \
	--env apple_id="<MYID>" \
	--env authentication_type=2FA \
	--env folder_structure={:%Y/%m} \
	--env auto_delete=True \
	--env synchronisation_interval=43200 \
	--env icloud_china=True \
	--env auth_china=True \
	--env TZ="Asia/Shanghai" \
	--volume $(pwd)/config:/config \
	--volume $(pwd)/photos:/home/root/iCloud \
	boredazfcuk/icloudpd sync-icloud.sh --init


......
2023-09-22 12:07:15 INFO     Script launch parameters: --init
chown: changing ownership of '/config/.DS_Store': Operation not permitted
chown: changing ownership of '/config/icloudpd.conf': Operation not permitted
chown: changing ownership of '/config/python_keyring': Operation not permitted
chown: changing ownership of '/config': Operation not permitted
2023-09-22 12:07:15 INFO     Directory is writable: /config/python_keyring/
2023-09-22 12:07:15 INFO     Directory is writable: /home/user/.local/share/
Enter iCloud password for <MYID>:
Save password in keyring? [y/N]: y
Traceback (most recent call last):
  File "/opt/icloudpd_v1.7.2_china/bin/icloud", line 33, in <module>
    sys.exit(load_entry_point('pyicloud==1.0.0', 'console_scripts', 'icloud')())
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/icloudpd_v1.7.2_china/lib/python3.11/site-packages/pyicloud/cmdline.py", line 197, in main
    utils.store_password_in_keyring(username, password)
  File "/opt/icloudpd_v1.7.2_china/lib/python3.11/site-packages/pyicloud/utils.py", line 55, in store_password_in_keyring
    return keyring.set_password(
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/icloudpd_v1.7.2_china/lib/python3.11/site-packages/keyring/core.py", line 61, in set_password
    get_keyring().set_password(service_name, username, password)
  File "/opt/icloudpd_v1.7.2_china/lib/python3.11/site-packages/keyrings/alt/file_base.py", line 129, in set_password
    self._write_config_value(service, username, password_base64)
  File "/opt/icloudpd_v1.7.2_china/lib/python3.11/site-packages/keyrings/alt/file_base.py", line 137, in _write_config_value
    self._ensure_file_path()
  File "/opt/icloudpd_v1.7.2_china/lib/python3.11/site-packages/keyrings/alt/file_base.py", line 166, in _ensure_file_path
    with open(self.file_path, 'w'):
         ^^^^^^^^^^^^^^^^^^^^^^^^^
PermissionError: [Errno 13] Permission denied: '/home/user/.local/share/python_keyring/keyring_pass.cfg'
2023-09-22 12:07:28 ERROR    Keyring file does not exist. Please try again.

boredazfcuk · 2023-09-22T06:38:44Z

All the env variables are written to /config/icloudpd.conf and then loaded from there, it's possible the logic is off for force_gid. Try recreating your container without any env variables then editing that file to force_gid=true

walnutist · 2023-09-22T06:52:03Z

All the env variables are written to /config/icloudpd.conf and then loaded from there, it's possible the logic is off for force_gid. Try recreating your container without any env variables then editing that file to force_gid=true

Correct. I also figured out. But even with the force_gid properly configured, the script still fail as my second comment.
So I did a bit investigation myself. looks like the Permission denied error occurs because the line 750 within ConfigurePassword function:

su "${user}" -c 'icloud --username "${0}"' -- "${apple_id}"

Within my container environment, this command eventually got translated into:

su "user" -c 'icloud --username "${0}"' -- "<MYID>"

The thing is, if I log into the container via terminal as inside root, then the icloud --username <MYID> works flawlessly. But if I try the above command which try to execute via user "", it throws the same permission denied error.

Why and how to fix?

Thanks & regards

boredazfcuk · 2023-09-22T07:14:04Z

Great work. I should be able to fix this now you've found the root cause. Hopefully I'll be able to take a look at this later today.

walnutist · 2023-09-22T07:34:55Z

Now I believe the issue is really at line 833. cause the below error occurred right before calling ConfigurePassword:

2023-09-22 07:28:34 DEBUG    Set owner and group on icloudpd temp directory
2023-09-22 07:28:34 DEBUG    Set owner and group on config directory
chown: changing ownership of '/config/.DS_Store': Operation not permitted
chown: changing ownership of '/config/icloudpd.conf': Operation not permitted
chown: changing ownership of '/config/python_keyring': Operation not permitted
chown: changing ownership of '/config': Operation not permitted

Indeed, inside the container, the /config/ folder belongs to root:nobody and is set to 755 (drwxr-xr-x). Also manually try to chown <my_host_user>:<my_host_group> /config/ throws exception "Operation not permitted". This could be the root cause.

both <my_host_user> and <my_host_group> exists inside the container after container creation.

walnutist · 2023-09-22T07:39:53Z

In fact, looks like the chown inside the container does NOT work at all, not even as root, as long as under the /config/ directory.

And here is my docker run command and parameters:

docker run -it\
	--privileged \
	--network bridge \
	--restart=always \
	--volume $(pwd)/config:/config \
	--volume $(pwd)/photos:/home/root/iCloud \
	boredazfcuk/icloudpd sync-icloud.sh --init

icloudpd.conf with privacy info removed:

#force_gid=true
albums_with_dates=false
apple_id=<MYID>
auth_china=true
authentication_type=MFA
auto_delete=true
bark_device_key=
bark_server=
convert_heic_to_jpeg=false
debug_logging=true
delete_accompanying=false
delete_after_download=false
delete_notifications=true
dingtalk_token=
directory_permissions=750
discord_id=
discord_token=
download_notifications=true
download_path=
file_permissions=640
folder_structure={:%Y/%m}
gotify_app_token=
group=<MACOS_HOST_GROUP>
group_id=502
icloud_china=true
iyuu_token=
jpeg_path=
jpeg_quality=90
nextcloud_delete=false
nextcloud_password=
nextcloud_upload=false
nextcloud_url=
nextcloud_username=
notification_days=7
notification_type=
photo_album=
photo_size=original
prowl_api_key=
pushover_sound=
pushover_token=
pushover_user=
recent_only=100
set_exif_datetime=false
single_pass=false
skip_album=
skip_check=false
skip_download=false
skip_live_photos=false
synchronisation_delay=0
synchronisation_interval=43200
synology_ignore_path=false
telegram_chat_id=
telegram_polling=true
telegram_server=
telegram_silent_file_notifications=
telegram_token=
trigger_nextlcoudcli_synchronisation=
until_found=
user=<MACOS_HOST_USER>
user_id=501
webhook_https=false
webhook_id=
webhook_path=/api/webhook/
webhook_port=8123
webhook_server=
wecom_id=
wecom_proxy=
wecom_secret=

walnutist · 2023-09-22T10:42:17Z

I just realized the underlying podman might be the real cirminal here...
See containers/podman#20077 for further information.
In that case, I would then switch to the old plain docker on macOS and give a try. Keep tuned and no need to further invest your time

walnutist · 2023-09-22T12:31:27Z

Switched to orbstack, now the initialization run through flawlessly.
Confirmed this was caused by podman underline filesystem, and seems same issue exists for colima too.
Closing the issue

walnutist closed this as completed Sep 22, 2023

boredazfcuk mentioned this issue Oct 10, 2023

Cannot Initialize icloudpd #400

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Not able to bootstrap the container on macOS Ventura 13.5.2/13.6.0 #391

Not able to bootstrap the container on macOS Ventura 13.5.2/13.6.0 #391

walnutist commented Sep 22, 2023 •

edited

Loading

walnutist commented Sep 22, 2023 •

edited

Loading

boredazfcuk commented Sep 22, 2023

walnutist commented Sep 22, 2023 •

edited

Loading

boredazfcuk commented Sep 22, 2023

walnutist commented Sep 22, 2023

walnutist commented Sep 22, 2023 •

edited

Loading

walnutist commented Sep 22, 2023

walnutist commented Sep 22, 2023

Not able to bootstrap the container on macOS Ventura 13.5.2/13.6.0 #391

Not able to bootstrap the container on macOS Ventura 13.5.2/13.6.0 #391

Comments

walnutist commented Sep 22, 2023 • edited Loading

walnutist commented Sep 22, 2023 • edited Loading

boredazfcuk commented Sep 22, 2023

walnutist commented Sep 22, 2023 • edited Loading

boredazfcuk commented Sep 22, 2023

walnutist commented Sep 22, 2023

walnutist commented Sep 22, 2023 • edited Loading

walnutist commented Sep 22, 2023

walnutist commented Sep 22, 2023

walnutist commented Sep 22, 2023 •

edited

Loading

walnutist commented Sep 22, 2023 •

edited

Loading

walnutist commented Sep 22, 2023 •

edited

Loading

walnutist commented Sep 22, 2023 •

edited

Loading