zlib.error: Error -3 while decompressing data: invalid stored block lengths

[gf3]

Overview

Downloading photos fails with the following error:

Traceback (most recent call last):
  File "starters/icloudpd.py", line 6, in <module>
  File "click/core.py", line 1161, in __call__
  File "click/core.py", line 1082, in main
  File "click/core.py", line 1443, in invoke
  File "click/core.py", line 788, in invoke
  File "icloudpd/base.py", line 767, in main
  File "icloudpd/base.py", line 1376, in core
  File "icloudpd/base.py", line 989, in download_photo_
  File "icloudpd/xmp_sidecar.py", line 69, in generate_xmp_file
  File "icloudpd/xmp_sidecar.py", line 92, in build_metadata
zlib.error: Error -3 while decompressing data: invalid stored block lengths
[PYI-2473184:ERROR] Failed to execute script 'icloudpd' due to unhandled exception!

Steps to Reproduce

icloudpd --directory ~/Pictures/iCloud --username g*****@r*******.org --set-exif-datetime --auto-delete --xmp-sidecar --use-os-locale --watch-with-interval 3600

Seems to fail around the ~1000th photo for me:

2%|#4                                                                               | 1001/57065 [00:50<1:23:55, 11.13it/s]

Context

icloudpd

version:1.26.0, commit sha:e2bc5f8, commit timestamp:Mon Jan 13 12:57:57 2025 EST

Installed with pip install --user icloudpd

Python 3.13.1

System

NAME="Fedora Linux"
VERSION="41 (Workstation Edition)"
RELEASE_TYPE=stable
ID=fedora
VERSION_ID=41
VERSION_CODENAME=""
PLATFORM_ID="platform:f41"
PRETTY_NAME="Fedora Linux 41 (Workstation Edition)"
ANSI_COLOR="0;38;2;60;110;180"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:41"
DEFAULT_HOSTNAME="fedora"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f41/system-administrators-guide/"
SUPPORT_URL="https://ask.fedoraproject.org/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=41
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=41
SUPPORT_END=2025-12-15
VARIANT="Workstation Edition"
VARIANT_ID=workstation

[gf3]

Note that this fails with pip and when downloading the release directly. But it works when run via docker. It must be something with my system. I wouldn't know where to start debugging but would be happy to provide any further info upon request.

Edit: actually, this is happening when I use the --xmp-sidecar option under all scenarios!

[AndreyNikiforov]

Edit: actually, this is happening when I use the --xmp-sidecar option under all scenarios!

1.26.1 has a number of fixes for xmp edge cases. See if that matches your case.

[gf3]

@AndreyNikiforov unfortunately the error still presents in 1.26.1:

2025-01-29 13:11:38 DEBUG    /home/gianni/Pictures/iCloud/2023/07/30/IMG_5732.HEIC already exists
  2%|#4                                                                                 | 1001/57095 [00:27<18:50, 49.62it/s]Traceback (most recent call last):
  File "starters/icloudpd.py", line 6, in <module>
  File "click/core.py", line 1161, in __call__
  File "click/core.py", line 1082, in main
  File "click/core.py", line 1443, in invoke
  File "click/core.py", line 788, in invoke
  File "icloudpd/base.py", line 767, in main
  File "icloudpd/base.py", line 1376, in core
  File "icloudpd/base.py", line 989, in download_photo_
  File "icloudpd/xmp_sidecar.py", line 69, in generate_xmp_file
  File "icloudpd/xmp_sidecar.py", line 104, in build_metadata
zlib.error: Error -3 while decompressing data: invalid stored block lengths
[PYI-1002976:ERROR] Failed to execute script 'icloudpd' due to unhandled exception!

[ragaskar]

I am also seeing this error with --xmp-sidecar. It appears to be reproducible and occur on the same file every time -- that is, I ran

icloudpd -d . --align-raw original --xmp-sidecar -u <my-icloud-account>

and got

2025-02-09 09:14:40 DEBUG    Downloading 2023/11/14/72167332630__E230717D-614D-4A92-A043-5BE16DC3E813.JPG...
2025-02-09 09:14:40 INFO     Downloaded 2023/11/14/72167332630__E230717D-614D-4A92-A043-5BE16DC3E813.JPG
 18%|##############################3                                                                                                                                         | 3259/18049 [31:59<2:34:11,  1.60it/s]
Traceback (most recent call last):
  File "starters/icloudpd.py", line 6, in <module>
  File "click/core.py", line 1161, in __call__
  File "click/core.py", line 1082, in main
  File "click/core.py", line 1443, in invoke
  File "click/core.py", line 788, in invoke
  File "icloudpd/base.py", line 767, in main
  File "icloudpd/base.py", line 1376, in core
  File "icloudpd/base.py", line 989, in download_photo_
  File "icloudpd/xmp_sidecar.py", line 69, in generate_xmp_file
  File "icloudpd/xmp_sidecar.py", line 104, in build_metadata
zlib.error: Error -3 while decompressing data: invalid stored block lengths
[PYI-2956647:ERROR] Failed to execute script 'icloudpd' due to unhandled exception!

then on rerun of the same command:

2025-02-09 09:38:32 DEBUG    2023/11/14/72167332630__E230717D-614D-4A92-A043-5BE16DC3E813.JPG already exists                                                                                                       18%|#############################7                                                                                                                                          | 3198/18049 [01:20<06:19, 39.13it/s]Traceback (most recent call last):
  File "starters/icloudpd.py", line 6, in <module>
  File "click/core.py", line 1161, in __call__
  File "click/core.py", line 1082, in main
  File "click/core.py", line 1443, in invoke
  File "click/core.py", line 788, in invoke
  File "icloudpd/base.py", line 767, in main
  File "icloudpd/base.py", line 1376, in core
  File "icloudpd/base.py", line 989, in download_photo_
  File "icloudpd/xmp_sidecar.py", line 69, in generate_xmp_file
  File "icloudpd/xmp_sidecar.py", line 104, in build_metadata
zlib.error: Error -3 while decompressing data: invalid stored block lengths

My current assumption is that some icloud metadata for that specific file cannot be handled properly.

When I locate the image in question in iCloud, it is an image I've saved from an iPhone app (it was a photo someone sent me via text, that I then edited in "Snapseed" and saved to my camera roll).

Actually, let me amend that:

There are two images with the exact same name (Using Snapseed, I saved both the original, unmodified image, and then the modified image to my camera roll). Given this, my first instinct would to be suspect some kind of filename collision. If this is correct, then deleting/changing the name of one of the files may allow me to proceed -- let me try that...

ah, no, that seems less likely, because when I look at the dir in question, I see:

/drives/data1/icloud_photo_backup# ls -la 2023/11/14/
total 49524
drwxrwsr-x+  2 root users     4096 Feb  9 09:14 .
drwxrwsr-x+ 16 root users     4096 Feb  9 09:14 ..
-rw-rw-r--   1 root users  2141403 Nov 14  2023 72167332630__E230717D-614D-4A92-A043-5BE16DC3E813.JPG
-rw-rw-r--   1 root users 40208442 Nov 14  2023 IMG_3056.MOV
-rw-rw-r--   1 root users      755 Feb  9 09:38 IMG_3056.MOV.xmp
-rw-rw-r--   1 root users   911464 Nov 14  2023 IMG_3057.HEIC
-rw-rw-r--   1 root users      829 Feb  9 09:38 IMG_3057.HEIC.xmp
-rw-rw-r--   1 root users  3433381 Nov 14  2023 IMG_3057_HEVC.MOV
-rw-rw-r--   1 root users  3986131 Nov 14  2023 IMG_3058.HEIC
-rw-rw-r--   1 root users      830 Feb  9 09:38 IMG_3058.HEIC.xmp

That implies to me that it's barfing on the initial XMP extraction, and not any kind of duplicate filename problem (I would have expected to see one xmp file if there was a dupe file name issue).

Alright, looking at the code in xmp_sidecar.py, I see this:

 if (
        "adjustmentSimpleDataEnc" in asset_record["fields"]
        and not asset_record["fields"]["adjustmentSimpleDataEnc"]["value"].startswith(
            "Y3JkdA"
        )  # "crdt"
        and not asset_record["fields"]["adjustmentSimpleDataEnc"]["value"].startswith("YnBsaXN0MD")
    ):  # "bplist00"
        adjustments = json.loads(
            zlib.decompress(
                base64.b64decode(asset_record["fields"]["adjustmentSimpleDataEnc"]["value"]),
                -zlib.MAX_WBITS,
            )
        )

I'm not very familiar with python, but that looks to me as if the code assumes a compressed format of some sort unless the adjustmentSimpleDataEnc value does NOT start with "Y3JkdA" OR "YnBsaXN0MD".

This seems to be a bad assumption? My current theory is we've found a "fourth case" where that value is NOT compressed. I think even if that 4th case is found and handled, one would probably also want to add some exception handling to any code that assumes the returned value is of a particular format but does not/cannot check to make sure that assumption is correct (ie, if the code just runs whatever it gets through some kind of unzip routine, there should be some error handling for cases where that assumption is bad).

Unfortunately, I'm not sure if there's an easy way for me to see what the response here is without downloading source + adding some debug info ... which ... I dunno if I want to do right now? It'd be nice if this thing rescued errors and then dumped the value it was trying to parse, I think it would make it easier to solve future problems with the code here. I'll see if I get enough time to snag the source + get it running on my machine so I can dump the info it's blowing up on, but if it's not easy I'll probably just manually remove these photos so my download can complete -- since that's the yak I'm actually trying to deal with: running out of space on my iCloud so I have to backup what I've got there and nuke some photos. By the way, this utility is PERFECT for that use-case, so this a great time for me to express my gratitude and say thank you SO MUCH for developing it, I can tell a TON of work went into it and I am super appreciative of all the time everyone has contributed towards it's development. ❤ Hopefully I can help out a little here if I'm able!

(EDIT-ed to add link to the code in question & rephrase a few things to hopefully make them more clear)