[RFC] Maintaining iCloud Storage Size

[AndreyNikiforov]

[RFC] Maintaining iCloud Storage Size

This is a proposal to control the size of the iCloud storage by automatically deleting old assets already present on the local storage.

1. Context

Many iCloud users store their entire asset collection in the cloud and use the icloudpd tool to maintain a local copy. This approach can become less optimal with larger collections due to:

• Increased cost, especially beyond free 5GB tier
• Limited support for assets originating outside of the Apple ecosystem

It is also possible to use the iCloud as an temporary storage between iDevice and a local storage - icloudpd has --delete-after-download parameter to control deletion of the iCloud assets as soon as they are downloaded to the local storage.

The above are two extremes - "all assets premanently in the cloud" and "no assets permanently in the cloud". There has been a number of requests to support hybrid approach to control the cost of the iCloud service - to keep some assets in the iCloud, while deleteing others. This is the focus of the current proposal.

2. Problem Scope

To control the content of the iCloud storage, two priorities have been identified:

1. Limit the iCloud storage by the total size of all assets, e.g. keep latest 5GB of assets to stay within free tier
2. Limit by the age of the assets, e.g. keep last 30 days of assets to prioritize usefulness

There is a need for exceptions, such as keeping certain assets (e.g., favorites) in the iCloud permanently. These exceptions will contribute to the total size calculation if applicable.

3. Assumptions

Local storage represents the final storage media for assets. Subsequent runs of the icloudpd agains local storage will check for asset presence and only download incremental changes.

Timestamp when the assets was taken (aka "created date") as reported by the iCloud is used for calculating age. For example, asset taken in 2000, but added to the iCloud in 2024, will be 25 years old in 2025. Same timestamp is used for folder structure in the current system.

Assets may be represented in multiple variations, e.g. raw+jpeg, small+medium+original, image+video. Only some variations are counted towards the iCloud total size. Content of smart folders, including Recently Deleted, are not counted towards the total size. This is standard behavior of the iCloud.

4. Decisions

Asset Filters

Asset filters, e.g. --skip-videos, limit assets that are checked by the icloudpd and only these checked assets are subject for deletion as part of storage maintenence. For example, running icloudpd with --skip-videos on a huge iCloud collection of videos will download and delete nothing.

Special whole set filters, --recent and --until-found, will not be supported with the proposed iCloud storage maintenance mode.

Asset Size Filters

Some assets have multiple representations ("sizes" in the icloudpd parameters):

• small, medium, or original size
• raw and jpeg quality
• image and live photo assets types
• original and edited versions

Deletion works at the asset level and all sizes will be removed from the iCloud.

When narrowing the scope of the icloudpd to only certain sizes, the rest of the sizes are not checked. For example, running for --size small will download only small size and then delete whole assets regardless of the presence of other sizes in the local storage.

Downloaded vs Existing Assets

Presence of the asset in the local storage has the same effect as downloading for the purpose of the iCloud storage maintenance. This is different from --delete-after-downloading parameter that deletes only after downloading, but not touching existing files.

Default --file-match-policy of name-size-dedup-with-suffix is not expected to have negative impact on the behavior of the proposed iCloud storage maintenance (beyond known inconsistencies, e.g. in combination with --set-exif-datetime).

Incompatibilities

--auto-delete parameter removes assets in the local storage if they were previously deleted in the iCloud. This functionality is not compatible with the proposed maintenance of the iCloud storage.

Dates and Timezones

Calculation of the age of the assets is expected to take into account timezone offset from the asset itself and timezone of the system running icloudpd.

5. Implementation Choices

Need mechanisms to specify:

• storage maintenance mode - limit to the size, limit to the age, or turn off the maintenance mode
• storage maintenance mode specification - size or age, respectively; applicable only when maintenance mode is not off
• specify exceptions - favorites; applicable only when maintenance mode is not off

(a) One Parameter to Control of What to Keep

New parameter --keep-in-cloud to control mode, spec, and exceptions. Parameter may be specified multiple times to provide non-conflicting values:

• last:G - enable size limiting mode and use "number" as a spec for size in GBs
• last:d - enable age limiting mode and use "number" as a spec for age in days
• all - do not delete anything from the iCloud (default)
• attr:favorite - instruction to keep all favorites assets in the iCloud regardless of age or size; requires enabled limiting mode as well

(b) Multiple Parameters to Control of What to Keep

Multiple parameters to control mode, spec, and exceptions:

• --keep-cloud-total-size - enable size limiting mode and use "number" as a spec for size in GBs
• --keep-cloud-recent-days - enable age limiting mode and use "number" as a spec for age in days
• --always-keep-favorites-in-cloud - instruction to keep all favorites assets in the iCloud regardless of age or size; requires enabled limiting mode as well

If niether --keep-cloud-total-size nor --keep-cloud-recent-days are specified, then storage maintenance mode is turned off and nothing will be deleted from the iCloud.

Recomendation

Use multiple parameters to control storage maintenance mode (option b), as it offers clearer user control.

[jhofker]

Love this. I think this is a great way to meet the needs (demands? 😉 ) of the various requests, issues, and discussions around this.

Presence of the asset in the local storage has the same effect as downloading for the purpose of the iCloud storage maintenance. This is different from --delete-after-downloading parameter that deletes only after downloading, but not touching existing files.

I didn't know this is how it behaved, but it makes sense. I definitely support using the asset's presence in this way.

Out of the options listed, I think B makes the most sense and is the most aligned with the current application. I think it would also allow for shipping parts of it sooner if desired. I've been working on another PR to allow keeping favorites in the cloud, but I was attempting to make it broader and enable users to specify an album to keep in the cloud, even if assets age out. I really only wanted it for favorites, which may behave differently anyway -- assets appear to have an isFavorite flag.

To that end, I'd suggest following your convention with the third parameter:
--keep-cloud-total-size 1G
--keep-cloud-recent-days 30
--keep-cloud-favorites true or --keep-cloud-album Favorites or even --keep-cloud-albums "Favorites, Italy Trip"

[AndreyNikiforov]

To that end, I'd suggest following your convention with the third parameter:
--keep-cloud-total-size 1G
--keep-cloud-recent-days 30
--keep-cloud-favorites true or --keep-cloud-album Favorites or even --keep-cloud-albums "Favorites, Italy Trip"

I completely missed the fact that all favorites are organized into smart album! We can have only --keep-cloud-album param for all exceptions we consider so far. This param should support multiple values to allow arbitrary number of albums, e.g. --keep-cloud-album Favorites --keep-cloud-album "My Other Album". We do not need separate --keep-cloud-favorites in this case.

[jhofker]

This param should support multiple values to allow arbitrary number of albums, e.g. --keep-cloud-album Favorites --keep-cloud-album "My Other Album".

Good call! I think that would work well.

One other thought, do you want the parameters to be explicitly icloud? So instead of --keep-cloud-recent-days it would be --keep-icloud-recent-days, etc?

[AndreyNikiforov]

One other thought, do you want the parameters to be explicitly icloud? So instead of --keep-cloud-recent-days it would be --keep-icloud-recent-days, etc?

I think having a reference to a cloud is important in the param so it is absolutely clear that we are controlling remote, not local presence of the asset. icloud is even more specific that cloud - I like that.

I also looked at --keep-in-cloud-recent-days, but to my eyes extra in does not seem to add value.

If there is another way to name params, let's explore. Confusing behavior (or misunderstanding of parameters) causes a lot of issues down the road.

[jhofker]

I think --keep-icloud-* is clear!

[AndreyNikiforov]

Edge cases:

• --keep-icloud-recent-days 0 should clean the iCloud (after ensuring local copies exist). That behavior is a bit different than --delete-after-downloading, which does not delete from the iCloud files that already exist on local storage.
• --keep-icloud-recent-days should not accept negative values
• --keep-icloud-recent-days should have some reasonable upper bound too
• Similar limits for --keep-icloud-total-size

[AndreyNikiforov]

for total size specifying units in the parameter should help avoid confusion too: --keep-icloud-total-size-gb.

For 5GB free tier, I assume users will have to specify lower value to allow new assets to be added to the iCloud. If we allow only ints for the size value, then for free tier users will have to leave 20% of the storage unused. I am leaning towards allowing floats as GB value to cover free tier use case better.

[jhofker]

I wonder if you might want to allow the unit to be specified instead? I'm mostly think about how the -h flag on the du command shows filesizes, but it could be a useful pattern that allows flexibility here.

--keep-cloud-total-size 10M if you want like...4 pictures
--keep-cloud-total-size 1.5G
--keep-cloud-total-size 1T
--keep-cloud-total-size 150M

Another thought: with --keep-cloud-total-size, it seems self-explanatory that it is specifically talking about the amount of storage used by the icloud photo library, but will the numbers be thrown off if users store other things in there, too?

[AndreyNikiforov]

I wonder if you might want to allow the unit to be specified instead? I'm mostly think about how the -h flag on the du command shows filesizes, but it could be a useful pattern that allows flexibility here.

--keep-cloud-total-size 10M if you want like...4 pictures --keep-cloud-total-size 1.5G --keep-cloud-total-size 1T --keep-cloud-total-size 150M

Another thought: with --keep-cloud-total-size, it seems self-explanatory that it is specifically talking about the amount of storage used by the icloud photo library, but will the numbers be thrown off if users store other things in there, too?

Perfect! To summarize:
a) use parameter name as --keep-cloud-total-size
b) use floats with mandatory unit suffix
c) support case-insensitive suffixes: MiB, GiB, TiB for base 2 and MB, GB, TB for base 10 units

[AndreyNikiforov]

@jhofker are you working or planning to work on other parts of this RFC? If I were to choose, --keep-icloud-album would be higher priority as it seems to match broader use case.

[jhofker]

@jhofker are you working or planning to work on other parts of this RFC? If I were to choose, --keep-icloud-album would be higher priority as it seems to match broader use case.

I have a branch started for keep-icloud-album, but I need to get the recent-days changes merged into it. Ideally I can put up a draft tonight/tomorrow so we can start discussion.

(FWIW, I'm not precious about it, so if you're wanting to tackle it, that's perfectly fine! Since the holiday break is over, my time will be limited to randoms evenings and weekends)

[jhofker]

Draft PR here: #1051

[AndreyNikiforov]

We missed one requirement: if -dry-run is specified, no deletion in iCloud should happen (e.g. for existing files)

[jhofker]

Yes, good point! That's in the code and having it documented here is good.