From e9b85146e10d4a857a0c799e7feb127ad59d3ce9 Mon Sep 17 00:00:00 2001 From: Andrey Nikiforov Date: Sat, 13 Jul 2024 09:39:31 -0700 Subject: [PATCH] doc updates (#912) --- .gitignore | 1 - .vscode/settings.json | 8 +++++ EXPERIMENTAL.md | 43 +--------------------- FAQ.md | 81 ------------------------------------------ README.md | 6 +--- docs/authentication.md | 54 ++++++++++++++-------------- docs/install.md | 28 ++++++++++++++- docs/naming.md | 24 +++++++++++-- docs/nas.md | 10 ++++-- docs/raw.md | 2 +- 10 files changed, 94 insertions(+), 163 deletions(-) create mode 100644 .vscode/settings.json delete mode 100644 FAQ.md diff --git a/.gitignore b/.gitignore index 4d3fe376a..d1c2b9086 100644 --- a/.gitignore +++ b/.gitignore @@ -4,7 +4,6 @@ dist/ wheelhouse/ icloudpd.egg-info tests/fixtures/** -.vscode __pycache__ .pytest_cache .cache diff --git a/.vscode/settings.json b/.vscode/settings.json new file mode 100644 index 000000000..a500989cd --- /dev/null +++ b/.vscode/settings.json @@ -0,0 +1,8 @@ +{ + "cSpell.words": [ + "icloudpd", + "seealso", + "versionadded", + "versionchanged" + ] +} \ No newline at end of file diff --git a/EXPERIMENTAL.md b/EXPERIMENTAL.md index b364caca2..c50fdf62d 100644 --- a/EXPERIMENTAL.md +++ b/EXPERIMENTAL.md @@ -6,45 +6,4 @@ Anything in this section can change without backward compatibility or even compl DANGER ZONE: Code may not work as expected. -## CLI format - -Goal: reduce user confusion and maintenance burden by restructuring CLI interface for better matching use cases and related options - -### New Structure - -Ideas: -- Each use case is matched to command -- Customizations are available as options - -Use Cases: -- Maintain local copy/backup of iCloud: **COPY** command -- Use iCloud as a transfer medium to local storage (and clean iCloud afterwards): **MOVE** command - -Orthogonal Needs: -- Scheduled sync: **WATCH** command -- management of persistent credentials: **AUTH** command -- monitoring/notification/alerting: TBD - - -### How to Use - -Legacy command (compatible with prior versions): - -`docker run -it --rm icloudpd:icloudpd icloudpd --help` - -`docker run -it --rm icloudpd:icloudpd icloud --help` - -`icloudpd-1.22.0-windows-amd64 --help` - -Help: - -`docker run -it --rm icloudpd:icloudpd` - -`icloudpd-ex-1.22.0-windows-amd64 --help` - -Example: - -`docker run -it --rm icloudpd:icloudpd copy my@email.address /path/to/{album}/{date_created:%Y/%Y-%m}` - -`icloudpd-ex-1.22.0-windows-amd64 copy my@email.address /path/to/{album}/{date_created:%Y/%Y-%m}` - +(no experiments yet) \ No newline at end of file diff --git a/FAQ.md b/FAQ.md deleted file mode 100644 index 3fe1f3844..000000000 --- a/FAQ.md +++ /dev/null @@ -1,81 +0,0 @@ -# Nuances and Known Issues - -## iCloud Authentication - -### MFA - -If your Apple account has two-factor authentication (multifactor authentication, MFA) enabled, -you will be prompted for a code when you run the script. Two-factor authentication will expire after an interval set by Apple, -at which point you will have to re-authenticate. This interval is currently two months. Apple requires MFA for all new accounts. - -You can receive an email notification when two-factor authentication expires by passing the -`--smtp-username` and `--smtp-password` options. Emails will be sent to `--smtp-username` by default, -or you can send to a different email address with `--notification-email`. - -If you want to send notification emails using your Gmail account, and you have enabled two-factor authentication, you will need to generate an App Password at - -### FIDO - -Authentication to iCloud with hardware keys (FIDO) is not supported. - -### ADP - -Advanced Data Protection (ADP) for iCloud accounts is not supported because `icloudpd` simulates web access, which is disabled with ADP. - -### Occasional Errors - -Some authentication errors may be resolved by clearing `.pycloud` subfolder in the user's home directory. [Example](https://github.com/icloud-photos-downloader/icloud_photos_downloader/issues/772#issuecomment-1950963522) - -### System Keyring - -You can store your password in the system keyring using the `icloud` command-line tool: - -``` plain -$ icloud --username jappleseed@apple.com -ICloud Password for jappleseed@apple.com: -Save password in keyring? (y/N) -``` - -If you have stored a password in the keyring, you will not be required to provide a password -when running the script. - -If you would like to delete a password stored in your system keyring, -you can clear a stored password using the `--delete-from-keyring` command-line option: - -``` sh -icloud --username jappleseed@apple.com --delete-from-keyring -``` - -## Error on the First Run - -When you run the script for the first time, you might see an error message like this: - -``` plain -Bad Request (400) -``` - -This error often happens because your account hasn't used the iCloud API before, so Apple's servers need to prepare some information about your photos. This process can take around 5-10 minutes, so please wait a few minutes and try again. - -If you are still seeing this message after 30 minutes, then please [open an issue on GitHub](https://github.com/icloud-photos-downloader/icloud_photos_downloader/issues/new) and post the script output. - -## Access from Mainland China - -Access to iCloud.com is blocked from mainland China. `icloudpd` can be used with `--domain cn` parameter to support downloading iCloud photos from mainland China, however, people reported mixed results with that parameter. - -## macOS binary - -`icloudpd` is available as Intel 64bit binary for macOS, but works on ARM macs too (M1, M2). - -Here are the steps to make it working: -- download binary from GitHub [Releases](https://github.com/icloud-photos-downloader/icloud_photos_downloader/releases) into desired local folder -- add executable flag by running `chmod +x icloudpd-1.22.0-macos-amd64` -- start it from the terminal: `icloudpd-1.22.0-macos-amd64` -- Apple will tell you that it cannot check for malicious software and refuse to run the app; click "Ok" -- Open "System Settings"/"Privacy & Security" and find `icloudpd-1.22.0-macos-amd64` as blocked app; Click "Allow" -- Start `icloudpd-1.22.0-macos-amd64` from the terminal again -- Apple will show another warning; click "Open" -- After that you can run `icloudpd-1.22.0-macos-amd64 --help` or any other supported command/option - -## Running on Synology NAS - -The error `Failed to execv() /tmp/staticx-kJmNbp` has a workaround by (from an SSH terminal in my case) running `sudo mount /tmp -o remount,exec`. #788 diff --git a/README.md b/README.md index ffe90b304..5020ac8a0 100644 --- a/README.md +++ b/README.md @@ -5,7 +5,7 @@ - Available as an executable for direct downloading and through package managers/ecosystems ([Docker](https://icloud-photos-downloader.github.io/icloud_photos_downloader/install.html#docker), [PyPI](https://icloud-photos-downloader.github.io/icloud_photos_downloader/install.html#pypi), [AUR](https://icloud-photos-downloader.github.io/icloud_photos_downloader/install.html#aur), [npm](https://icloud-photos-downloader.github.io/icloud_photos_downloader/install.html#npm)) - Developed and maintained by volunteers (we are always looking for [help](CONTRIBUTING.md)). -See [Documentation](https://icloud-photos-downloader.github.io/icloud_photos_downloader/) for more details +See [Documentation](https://icloud-photos-downloader.github.io/icloud_photos_downloader/) for more details. Also, check [Issues](https://github.com/icloud-photos-downloader/icloud_photos_downloader/issues) We aim to release new versions once a week (Friday), if there is something worth delivering. @@ -61,10 +61,6 @@ icloudpd --username my@email.address --password my_password --auth-only > [!TIP] > This feature can also be used to check and verify that the session is still authenticated. -## FAQ - -Nuances of working with the iCloud or a specific operating system are collected in the [FAQ](FAQ.md). Also, check [Issues](https://github.com/icloud-photos-downloader/icloud_photos_downloader/issues). - ## Contributing Want to contribute to iCloud Photos Downloader? Awesome! Check out the [contributing guidelines](CONTRIBUTING.md) to get involved. diff --git a/docs/authentication.md b/docs/authentication.md index 9ada2a099..4d1f2e5a4 100644 --- a/docs/authentication.md +++ b/docs/authentication.md @@ -2,7 +2,7 @@ ## Multi-Factor Authentication -If your Apple account has two-factor authentication (multifactor authentication, MFA) enabled, +If your Apple account has two-factor authentication (multi-factor authentication, MFA) enabled, you will be prompted for a code when you run the script. Two-factor authentication will expire after an interval set by Apple, at which point you will have to re-authenticate. This interval is currently two months. Apple requires MFA for all new accounts. @@ -25,6 +25,9 @@ The choice can be made with `--mfa-provider` parameter. Default: *console* +## Access from Mainland China + +Access to iCloud.com is blocked from mainland China. `icloudpd` can be used with `--domain cn` parameter to support downloading iCloud photos from mainland China, however, people reported mixed results with that parameter. ## FIDO @@ -36,31 +39,7 @@ Advanced Data Protection (ADP) for iCloud accounts is not supported because `icl ## Occasional Errors -Some authentication errors may be resolved by clearing `.pycloud` subfolder in the user's home directory. [Example](https://github.com/icloud-photos-downloader/icloud_photos_downloader/issues/772#issuecomment-1950963522) - -## System Keyring - -You can store your password in the system keyring using the `icloud` command-line tool: - -``` -$ icloud --username jappleseed@apple.com -ICloud Password for jappleseed@apple.com: -Save password in keyring? (y/N) -``` - -If you have stored a password in the keyring, you will not be required to provide a password -when running the script. - -If you would like to delete a password stored in your system keyring, -you can clear a stored password using the `--delete-from-keyring` command-line option: - -``` sh -icloud --username jappleseed@apple.com --delete-from-keyring -``` - -```{note} -Use `icloud`, not `icloudpd` -``` +Some authentication errors may be resolved by clearing `.pyicloud` subfolder in the user's home directory. [Example](https://github.com/icloud-photos-downloader/icloud_photos_downloader/issues/772#issuecomment-1950963522) ## Password Providers @@ -83,4 +62,25 @@ Keyring password provider, if specified, saves valid password back into keyring. Console and Web UI are not compatible with each other. Console or WebUI providers, if specified, must be last in the list of providers because they cannot be skipped. -Default set and order of providers are: *parameter*, *keyring*, *console* \ No newline at end of file +Default set and order of providers are: *parameter*, *keyring*, *console* + +### Managing System Keyring + +You can store your password in the system keyring or delete it from there using the `icloud` command-line tool: + +``` +$ icloud --username jappleseed@apple.com +ICloud Password for jappleseed@apple.com: +Save password in keyring? (y/N) +``` + +If you would like to delete a password stored in your system keyring, +you can clear a stored password using the `--delete-from-keyring` command-line option: + +``` sh +icloud --username jappleseed@apple.com --delete-from-keyring +``` + +```{note} +Use `icloud`, not `icloudpd` +``` diff --git a/docs/install.md b/docs/install.md index 299828cce..4531963ba 100644 --- a/docs/install.md +++ b/docs/install.md @@ -12,7 +12,7 @@ There are three ways to run `icloudpd`: docker run -it --rm --name icloudpd -v $(pwd)/Photos:/data -e TZ=America/Los_Angeles icloudpd/icloudpd:latest icloudpd --directory /data --username my@email.address --watch-with-interval 3600 ``` -Image asset date will be converted to specified TZ and then used for creating folders (see `--folder-stucture` parameter) +Image asset date will be converted to specified TZ and then used for creating folders ([see `--folder-structure` parameter](#folder_structure)) Synchronization logic can be adjusted with command-line parameters. Run the following to get full list: ``` sh @@ -95,3 +95,29 @@ yay -S icloudpd-bin ``` sh npx --yes icloudpd --directory /data --username my@email.address --watch-with-interval 3600 ``` + +## macOS binary + +`icloudpd` is available as Intel 64bit binary for macOS, but works on ARM macs too (M1, M2, M3). + +Here are the steps to make it working: +- download binary from GitHub [Releases](https://github.com/icloud-photos-downloader/icloud_photos_downloader/releases) into desired local folder +- add executable flag by running `chmod +x icloudpd-1.22.0-macos-amd64` +- start it from the terminal: `icloudpd-1.22.0-macos-amd64` +- Apple will tell you that it cannot check for malicious software and refuse to run the app; click "Ok" +- Open "System Settings"/"Privacy & Security" and find `icloudpd-1.22.0-macos-amd64` as blocked app; Click "Allow" +- Start `icloudpd-1.22.0-macos-amd64` from the terminal again +- Apple will show another warning; click "Open" +- After that you can run `icloudpd-1.22.0-macos-amd64 --help` or any other supported command/option + +## Error on the First Run + +When you run the script for the first time, you might see an error message like this: + +``` plain +Bad Request (400) +``` + +This error often happens because your account hasn't used the iCloud API before, so Apple's servers need to prepare some information about your photos. This process can take around 5-10 minutes, so please wait a few minutes and try again. + +If you are still seeing this message after 30 minutes, then please [open an issue on GitHub](https://github.com/icloud-photos-downloader/icloud_photos_downloader/issues/new) and post the script output. diff --git a/docs/naming.md b/docs/naming.md index f1d8715a1..435517a46 100644 --- a/docs/naming.md +++ b/docs/naming.md @@ -2,16 +2,36 @@ Assets on iCloud have names. When downloading assets, `icloudpd` can adjust names. +(folder_structure)= ## Folder Structure ```{versionchanged} 1.7.0 Support for `none` value added ``` +```{versionchanged} 1.22.0 +Support for OS locale added +``` -`icloudpd` uses asset metadata (_created date_) to build folder hierarchy, and it can be adjusted with `--folder-stucture` parameter. +`icloudpd` uses asset metadata (_created date_) to build folder hierarchy, and it can be adjusted with `--folder-structure` parameter. Specifying `--folder-structure none` will put all files into one folder. +### Formatting + +`icloudpd` follows [Python string formatting grammar](https://docs.python.org/3/library/string.html#formatstrings) for `--folder-structure` parameter,e.g. `{:%Y}` means the need to take only 4-digit year out of created date. Full list of format codes is [available](https://docs.python.org/3/library/datetime.html#strftime-and-strptime-format-codes). + +Default format is: `{:%Y/%m/%d}` + +### Language-specific formatting + +Some formatting codes, e.g. `%B` for printing full month, are specific to the language. By default `icloudpd` uses English regardless of the locale of the OS. With `--use-os-locale` the behavior can be changed. + +Example of running `icloudpd` with specific locale under Linux or MacOS: + +```shell +LC_ALL=ru_RU.UTF.8 icloudpd --use-os-locale --version +``` + ## Duplicates ```{versionchanged} 1.20.0 @@ -31,7 +51,7 @@ In large iCloud collections it is possible to have name collisions. To avoid col Live Photo assets have two components: still image and short video. `icloudpd` can download both and allows customizing file name of the video portion with `--live-photo-mov-filename-policy` parameter: - Use video file name the same as still image with `original` policy; use `--file-match-policy name-id7` to avoid clashes of video file with other videos. -- Use suffix from the still image with `suffix` policy: **"IMG_1234_HEVC.MOV"** for **"IMG_1234.HEIC"** still. This is default and works for HIEC still images only +- Use suffix from the still image with `suffix` policy: **"IMG_1234_HEVC.MOV"** for **"IMG_1234.HEIC"** still. This is default and works for HEIC still images only ## Unicode diff --git a/docs/nas.md b/docs/nas.md index edd094a7c..70659a59a 100644 --- a/docs/nas.md +++ b/docs/nas.md @@ -4,7 +4,7 @@ The following are example setups for NAS. ## TrueNAS -Use `Install Custom App` button to set up `icloudpd`: +Use [`Install Custom App` button](https://www.truenas.com/docs/scale/23.10/scaletutorials/apps/usingcustomapp/) to set up `icloudpd`: | Field | Value | Note | |-------|-------|------| Application Name | `icloudpd` | @@ -20,8 +20,12 @@ Portal Configuration/Enable Portal Configuration | checked | Portal Configuration/Portal Name | `icloudpd` | Portal Configuration/Protocol to Portal | `HTTP Protocol` | Portal Configuration/Use Node IP for Portal IP/Domain | checked | -Portal Configuration/Port | `9090` | +Portal Configuration/Port | `9090` | Same as "Port Forwarding/Host Port" above Once the app has started, connect to the [WebUI](webui) to enter password and MFA code one of two ways: - Using browser from your PC to 9090 port of your NAS -- Clicking on `icloudp` button in Detail/Application Info section of TrueNAS portal \ No newline at end of file +- Clicking on `icloudpd` button in Detail/Application Info section of TrueNAS portal + +## Running on Synology NAS + +The error `Failed to execv() /tmp/staticx-kJmNbp` has a workaround by (from an SSH terminal in my case) running `sudo mount /tmp -o remount,exec`. [#788](https://github.com/icloud-photos-downloader/icloud_photos_downloader/issues/788) diff --git a/docs/raw.md b/docs/raw.md index 3830d539b..3b6fcd006 100644 --- a/docs/raw.md +++ b/docs/raw.md @@ -34,7 +34,7 @@ One representation will be `original` [size](size) and another `alternative`. As of June 2024, icloud.com always shows assets with two representations as RAW+JPEG. Photo app on Mac allows choosing which representation to treat as original, but it is not clear what that setting changes. -`icloudpd` disambiguates behavior with `--align-raw` parameter: +`icloudpd` disambiguates the behavior with `--align-raw` parameter: - *original* always treat RAW as original [size](size) - *alternative* always treat RAW as alternative [size](size)