zip files generated from Xcode Cloud cannot be streamed during extraction

[sktusing]

Description of the problem

Using generate_appcast with a .app.zip generated from Xcode Cloud results in an error:

sktusing@SKT-MBP Sparkle-2.6.0 % ./bin/generate_appcast /Users/sktusing/Desktop/Vor_Release 
ditto: Vor.app/Contents/Frameworks/DeckLink.framework/DeckLink: No such file or directory
ditto: Couldn't read pkzip signature.
ditto: Couldn't read pkzip signature.
ditto: Couldn't read pkzip signature.
ditto: Couldn't read pkzip signature.
ditto: Couldn't read pkzip signature.

Running generate_appcast a second time results in:

sktusing@SKT-MBP Sparkle-2.6.0 % ./bin/generate_appcast /Users/sktusing/Desktop/Vor_Release 
Could not unarchive /Users/sktusing/Desktop/Vor_Release/Vor.app.zip Error Domain=NSCocoaErrorDomain Code=516 "“Vor.app.zip” couldn’t be copied to “762209f0516ccdac78f286882388fccbe8cfbae5c16c22f1946db52ce85f7926.tmp” because an item with the same name already exists." UserInfo={NSSourceFilePathErrorKey=/Users/sktusing/Desktop/Vor_Release/Vor.app.zip, NSUserStringVariant=(
    Copy
), NSDestinationFilePath=/Users/sktusing/Library/Caches/Sparkle_generate_appcast/762209f0516ccdac78f286882388fccbe8cfbae5c16c22f1946db52ce85f7926.tmp/Vor.app.zip, NSFilePath=/Users/sktusing/Desktop/Vor_Release/Vor.app.zip, NSUnderlyingError=0x6000012e4d20 {Error Domain=NSPOSIXErrorDomain Code=17 "File exists"}}
Error generating appcast from directory /Users/sktusing/Desktop/Vor_Release 
 Error Domain=SUSparkleErrorDomain Code=1001 "No usable archives found in /Users/sktusing/Desktop/Vor_Release" UserInfo={NSLocalizedDescription=No usable archives found in /Users/sktusing/Desktop/Vor_Release}

Running sktusing@SKT-MBP ~ % zipinfo /Users/sktusing/Desktop/Vor_Release/Vor.app.zip shows that Decklink directory exists:

lrwxrwxr-x  2.0 unx       25 bX stor 24-Apr-21 14:21 Vor.app/Contents/Frameworks/DeckLink.framework/DeckLink

Unzipping, then zipping the app causes generate_appcast to function.

This is new with us migrating to Xcode Cloud for notarizing the application.

Do you use Sandboxing in your app?

No

Version of Sparkle.framework in the latest version of your app

2.6.0

Version of Sparkle.framework in the old version of app that your users have (or N/A)

N/A

Sparkle's output from Console.app

default	10:18:31.749037-0400	generate_appcast	Not internal release, disabling SIRL
default	10:18:31.749078-0400	generate_appcast	Enabling System Keychain Always due to platform
default	10:18:31.749119-0400	generate_appcast	[0x600003acc000] activating connection: mach=true listener=false peer=false name=com.apple.securityd.xpc
default	10:18:31.749536-0400	secd	Entitlement com.apple.application-identifier=org.sparkle-project.Sparkle.generate-appcast is ignored because of invalid application signature or incorrect provisioning profile
default	10:18:31.750303-0400	generate_appcast	[0x600003ac0000] activating connection: mach=true listener=false peer=false name=com.apple.cfprefsd.daemon
default	10:18:31.754100-0400	generate_appcast	UNIX error exception: 17
default	10:18:31.759236-0400	generate_appcast	UNIX error exception: 17
default	10:18:31.760170-0400	generate_appcast	UNIX error exception: 17
default	10:18:31.762017-0400	generate_appcast	UNIX error exception: 17
default	10:18:31.763014-0400	generate_appcast	UNIX error exception: 17
default	10:18:31.764159-0400	generate_appcast	UNIX error exception: 17
default	10:18:31.802500-0400	generate_appcast	[0x600003ad8000] activating connection: mach=true listener=false peer=false name=com.apple.cfprefsd.agent
default	10:18:31.813110-0400	generate_appcast	[0x14a205480] activating connection: mach=true listener=false peer=false name=com.apple.lsd.mapdb
default	10:18:31.814565-0400	generate_appcast	[0x600003ac42d0] activating connection: mach=true listener=false peer=false name=com.apple.distributed_notifications@Uv3
default	10:18:32.905052-0400	generate_appcast	Not internal release, disabling SIRL
default	10:18:32.905095-0400	generate_appcast	Enabling System Keychain Always due to platform
default	10:18:32.905138-0400	generate_appcast	[0x600000948000] activating connection: mach=true listener=false peer=false name=com.apple.securityd.xpc
default	10:18:32.905350-0400	secd	Entitlement com.apple.application-identifier=org.sparkle-project.Sparkle.generate-appcast is ignored because of invalid application signature or incorrect provisioning profile
default	10:18:32.906096-0400	generate_appcast	[0x6000009480f0] activating connection: mach=true listener=false peer=false name=com.apple.cfprefsd.daemon
default	10:18:32.909018-0400	generate_appcast	UNIX error exception: 17
default	10:18:32.910882-0400	generate_appcast	UNIX error exception: 17
default	10:18:32.912028-0400	generate_appcast	UNIX error exception: 17
default	10:18:32.913280-0400	generate_appcast	UNIX error exception: 17
default	10:18:32.914386-0400	generate_appcast	UNIX error exception: 17
default	10:18:32.915625-0400	generate_appcast	UNIX error exception: 17
default	10:18:32.952601-0400	generate_appcast	Extracting using '/usr/bin/ditto' '-x' '-k' '-' < '/Users/[user]/Library/Caches/Sparkle_generate_appcast/762209f0516ccdac78f286882388fccbe8cfbae5c16c22f1946db52ce85f7926.tmp/Vor.app.zip' '/Users/[user]/Library/Caches/Sparkle_generate_appcast/762209f0516ccdac78f286882388fccbe8cfbae5c16c22f1946db52ce85f7926.tmp'

Steps to reproduce the behavior

1. Download zipped application from Xcode Cloud
2. Sign .app.zip with generate_appcast
3. generate_appcast errors and fails

[zorgiepoo]

Can you try unzipping the archive with the following command with Terminal:

ditto -x -k Vor.app.zip .

If it fails, theres's a chance the zip file is not that good, and you wouldn't want to distribute a zipped app that will be prone to fail extracting with one of the system unarchivers. Which macOS version are you on, too?

As for the second error, try to remove the offending extracted cache generate_appcast generated, which I think is bad because it failed to originally extract the file properly the first time. As the error you provided indicates, that may be in ~/Library/Caches/Sparkle_generate_appcast/762209f0516ccdac78f286882388fccbe8cfbae5c16c22f1946db52ce85f7926.tmp

[sktusing]

Can you try unzipping the archive with the following command with Terminal:

The command does not fail, and the app is extracted.

Which macOS version are you on, too?

14.4.1 (23E224)

[zorgiepoo]

The last line here shows the exact (barring user name stripped out) command generate appcast uses

default	10:18:32.952601-0400	generate_appcast	Extracting using '/usr/bin/ditto' '-x' '-k' '-' < '/Users/[user]/Library/Caches/Sparkle_generate_appcast/762209f0516ccdac78f286882388fccbe8cfbae5c16c22f1946db52ce85f7926.tmp/Vor.app.zip' '/Users/[user]/Library/Caches/Sparkle_generate_appcast/762209f0516ccdac78f286882388fccbe8cfbae5c16c22f1946db52ce85f7926.tmp'

Does the issue always reproduce with that zip file? I don't think I can do much here without a debuggable repro case (like the zip file. I don't have Xcode cloud set up).

[sktusing]

Does the issue always reproduce with that zip file?

Yep, and others that have been generated since moving to Xcode Cloud for notarization. Happy to provide an un-signed zip file if you'd like.

[zorgiepoo]

Yeah it would be helpful providing a zip file. Feel free to email me (zorgiepoo at gmail dot com) if it's sensitive, otherwise link it here. There might be something wrong in our piping code in our unarchiver.

[zorgiepoo]

Received a couple zip files over email and can easily reproduce.

Unzipping, then zipping the app causes generate_appcast to function.

This is a workaround for now. I cannot reproduce trying several different ways of re-creating the zip file, so oddly it's specific to how Xcode Cloud creates zip files.

This issue seems to also be related to our NSPipe unarchiving code (shared between Sparkle and generate_appcast)

Sparkle/Autoupdate/SUPipedUnarchiver.m

Line 109 in 0cff817

NSPipe *pipe = [NSPipe pipe];

The ditto error is also pretty odd..

If generate_appcast does a signal(SIGPIPE, SIG_IGN) call like our unit tests and Autoupdate do, it won't abruptly halt, but generate_appcast will still emit an error.

Sparkle 1.x can't handle unarchiving these zip files as well, and I believe the code before then hasn't changed much since 2015. I guess I should next rule out if this is specific to generate_appcast or if other components are affected (generate_appcast has an extra symlink step).

[zorgiepoo]

Looking into this I found another unrelated bug I had to fix (#2550), and which I'm still not done completely integrating in older releases.

This issue itself is that these zip files that Xcode Cloud generates cannot be streamed using a pipe during extraction correctly using any of the macOS built in tools (ditto, bsdtar). They will not extract these zip files and fail somewhere. That is not an issue in Sparkle. And this is not a new issue, just that no one has tried passing in these type of files before.

We could work around this by seeing if extraction fails due to SIGPIPE and re-trying without piping the file in to get extraction progress. But I am not sure if I want to support these zip files since we can't get any extraction progress and future support for failing to stream them is unknown (for example bsdtar doesn't even report an error, the files extracted are just incomplete/corrupted). Or we could completely remove in-progress streaming for zip files since it's not done in the most optimal way (via a pipe) but that doesn't sound great. Perhaps I should also more familiarize myself with the ZIP file and the file format to see what steps are needed to create zip files like these.

[zorgiepoo]

This bug is fixed in macOS 15 beta in ditto. However I also have a fix in #2616 for older OS's, which retries extraction without piping when piping fails (this may result in less fine-grained progress reporting, but may not be noticeable if the archive is small).