Workflow to Download and Replace RAWs with HEICs in iCloud

[mmshivesh]

I am attempting to set up @boredazfcuk's (boredazfcuk/docker-icloudpd) iCloud PD on Unraid. My initial strategy for photo management was like so:

1. iCloudPD - downloads all my images including all RAWs, with metadata.
2. Once I confirm this conversion is complete, I replace the RAWs with optimized HEICs on my iPhone (using Shortcuts, etc.)

However, wouldn't that mean that the Docker would then begin to download these converted HEIC files as well, basically ending up duplicating everything on my Server (one copy of the RAW and another copy being the HEIC file)?

I do not wish to do that. My aim is to keep full size RAWs on my server and I can keep edited and converted HEIC files for convenience on my iPhone.

If I'm going about it in a roundabout way and there's a way easier workflow that I'm missing. Wanted to see if anyone had insights before I get ready to script this use case.

[AndreyNikiforov]

Are you planning to perform one time conversion or you are looking in setting up workflow for continuous conversion? Does your camera support saving raw+jpeg/heic and will that work instead of converting on the server?

I do not understand how are you planning to replace raw with hiec in icloud. Did you try that yet? I think when you try to upload heic, iCloud will treat it as a separate assets not linked to raw file.

[mmshivesh]

Are you planning to perform one time conversion or you are looking in setting up workflow for continuous conversion?

This would ideally be an ongoing workflow, where all images are grabbed (including RAWs) but it doesn't grab their converted versions.

Does your camera support saving raw+jpeg/heic and will that work instead of converting on the server?

Most of my photography is through an iPhone so yes, I can leverage this format. But how would this work? My intention is to save on iCloud Storage costs, as I've started brushing up against 200GB, and want to address the root cause of the issue (of large RAWs on iCloud). So having both RAWs and a JPEG attached at the hip as a single photo wouldn't solve the issue of storage right, unless I'm misunderstanding?

I do not understand how are you planning to replace raw with heic in icloud. Did you try that yet?

The conversion with Shortcuts basically involves converting the image while retaining all metadata, and deleting the original. I believe this would break any de-duplication that iCloud PD does (?) as the newly converted photo would be a unique one in its eyes. I can provide a sample image and a converted version with exiftool outputs in case this would be helpful.

I will check to see if such a converted HEIC output has the same name as the original, in case I can de-duplicate them using purely the image file name (and since iCloudPD would already have grabbed the RAW version, it will not download this HEIC version which it will consider a "duplicate").

[AndreyNikiforov]

If your goal is to save on iCloud cost, then how would you prevent converted hiec files from filling it? There were suggestion on the forum to support deletion of the images on iCloud later than X days from today, which will keep iCloud storage small all the time, but that is not implemented (yet).

Re RAW+JPEG: if you are using iPhone to shoot, if may not have an option (==I don't think iPhone support that). icloudpd supports raw+jpeg through sizes. There may be way to build workflows using that.

Re conversion: if hiec version is uploaded and treated by iCloud as a separate asset (and not as raw+jpeg) then they can be manipulated independently (for good or bad). If treated independently, you may be able to download RAWs and delete them from iCloud right away. Then upload converted versions somehow (icloudpd does not have upload).

in summary: the use case of saving iCloud cost is not very well supported out of the box, but you may be able to build something custom through experimentation and trials.

[mmshivesh]

If your goal is to save on iCloud cost, then how would you prevent converted hiec files from filling it? There were suggestion on the forum to support deletion of the images on iCloud later than X days from today, which will keep iCloud storage small all the time, but that is not implemented (yet).

Well, yes obviously, I'm not going to make unreasonable demands and ask for a way to somehow stay at 200GB forever. I am not averse to paying for iCloud if necessary, just want to eke out as much value as I can before I bump my storage up to the next tier. For what it's worth, the savings are substantial:

iPhone 48MP RAW: 77MB
iPhone 48MP HEIC: 6MB (< This is close to a 12x savings)
iPhone 48MP JPEG: 23MB

iPhone 12MP RAW: ~27 MB
iPhone 12MP HEIC: 1.1 MB (< This is close to a 27x savings)
iPhone 12MP JPEG: ~6-7 MB

Re RAW+JPEG: if you are using iPhone to shoot, if may not have an option (==I don't think iPhone support that). icloudpd supports raw+jpeg through sizes. There may be way to build workflows using that.

iPhones can shoot in RAW+HEIC (or RAW+JPEG). The native app doesn't shoot in these modes but this format is well supported by iPhones when using a third party camera app, like Halide, or ProShot:

Re conversion: if hiec version is uploaded and treated by iCloud as a separate asset (and not as raw+jpeg) then they can be manipulated independently (for good or bad). If treated independently, you may be able to download RAWs and delete them from iCloud right away. Then upload converted versions somehow (icloudpd does not have upload).

Yes, the HEIC conversion is a "separate" image which has the same metadata as the original RAW, just very small in size. Additionally, I noticed that the converted image has the same file name as the original one (example images below). Can I trick iCloudPD into thinking that the converted image is a "duplicate" of the RAW using the file name so that HEIC isn't downloaded again? This would basically solve my problems because then the HEIC version will be ignored by iCloudPD

[AndreyNikiforov]

Thanks for the detailed info - I learned a lot.

Can I trick iCloudPD into thinking that the converted image is a "duplicate" of the RAW using the file name so that HEIC isn't downloaded again?

icloudpd reads assets from icloud.com and each asset may have multiple representations, e.g. raw and heic. icloudpd supports such type of assets. treating two separate icloud.com assets as the same on local file system and deterministically choosing one of them is not supported. deleting happens at asset level as well, so it is not possible to delete (through icloudpd) in icloud.com raw out of raw+jpeg.

[mmshivesh]

Thanks for the detailed info - I learned a lot.

No worries, glad to have helped clarify a few use cases!

treating two separate icloud.com assets as the same on local file system and deterministically choosing one of them is not supported.

Yep, that's what I feared. No worries, thank you for helping me out and your patience in understanding my use-case. I might try to test out the actual behavior of icloud PD by trying to add the original image into an album, downloading it, and then replacing it with HEIC and seeing if iCloudPD assumes it's a copy (are they treated as separate images, are they treated as the same one? though I fear given my reading of the code here and Apple assigning unique ID's to photos that they will be considered "unique images" irrespective of what I do).

Or I might have to think about forking and making the duplication algorithm "intentionally worse" to treat them as the same.

Once again, thank you!

[xbd-gh]

You can create a album,after you have new raw photos you can add them to the album.Then use ipd download this album only with --delete-after-download.

[mmshivesh]

@xbd-gh Yes, this would partially work. Where it fails however, is if I want other images from iCloud Photo Library as well (apart from this specific album, for say, saving videos and live photos). In this case, if I convert and keep HEIC versions, this copy is downloaded as well.