Notarization for arm64 version of app fails randomly but succeeds on retry

[praneetloke]

I have an Electron app that is built and notarized using electron-builder. Notarization for the arm64 version of the app fails randomly without any additional information. The only log I have is this:

...
electron-notarize:spawn spawning cmd: xcrun args: [
  'notarytool',
  'submit',
  '/var/folders/5q/zcc7tkt11f13mhczs1yfd4qc0000gn/T/electron-notarize-PsiiWM/****.zip',
  '--apple-id',
  '*********',
  '--password',
  '*********',
  '--team-id',
  '*********',
  '--wait',
  '--output-format',
  'json'
] opts: {}
...
...
electron-notarize:spawn cmd xcrun terminated with code: null
electron-notarize:helpers work failed
  ⨯ Failed to notarize via notarytool.  Failed with unexpected result: 

  failedTask=build stackTrace=Error: Failed to notarize via notarytool.  Failed with unexpected result: 
                                                                                 at /Users/runner/work/****/****/node_modules/@electron/notarize/src/notarytool.ts:115:13
    at Generator.next (<anonymous>)
    at fulfilled (/Users/runner/work/****/****/node_modules/@electron/notarize/lib/notarytool.js:28:58)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
From previous event:
    at processImmediate (node:internal/timers:483:21)
From previous event:
    at readDirectoryAndSign (/Users/runner/work/****/****/node_modules/app-builder-lib/src/macPackager.ts:496:29)
    at MacPackager.signApp (/Users/runner/work/****/****/node_modules/app-builder-lib/src/macPackager.ts:506:11)
    at MacPackager.doSignAfterPack (/Users/runner/work/****/****/node_modules/app-builder-lib/src/platformPackager.ts:346:21)
    at MacPackager.doPack (/Users/runner/work/****/****/node_modules/app-builder-lib/src/platformPackager.ts:331:7)
    at MacPackager.pack (/Users/runner/work/****/****/node_modules/app-builder-lib/src/macPackager.ts:215:9)
    at Packager.doBuild (/Users/runner/work/****/****/node_modules/app-builder-lib/src/packager.ts:459:9)
    at executeFinally (/Users/runner/work/****/****/node_modules/builder-util/src/promise.ts:12:14)
    at Packager.build (/Users/runner/work/****/****/node_modules/app-builder-lib/src/packager.ts:393:31)
    at executeFinally (/Users/runner/work/****/****/node_modules/builder-util/src/promise.ts:12:14)

There is no other information about the failure. In a successful execution, the exit code is 0 which is expected. I have set DEBUG=electron-notarize* as per the README in this repo but it doesn't seem to help with emitting any additional information about the failure. Any help/tips on diagnosing this would be appreciated.

[VerteDinde]

Hey @praneetloke, thanks for the report. Does this also repro with a basic repo? Also could you include the version of notarize that you're using?

[fempyrean]

@VerteDinde just to add to this, I'm also having a very similar issue. The error output is identical to the one reported, but in my case, the problematic architecture is x86, arm64 notarizes fine.

The notarization had been working fine for a long time but suddenly it stopped working, but I couldn't figure out exactly when it happened because the last build was from a couple months ago, so no logs for that.

Thought it could be related to GitHub upgrading their macos-latest runner to use macOS 14, but that happened at the start of 2024 and notarization was definitely working after that.

I'm running the latest version of notarize: 2.5.0.

[praneetloke]

Hello @VerteDinde, thank you for your response.

Does this also repro with a basic repo?

Unfortunately, I do not know if that's the case since the failure is sporadic right now. I might have an entire day of zero failures.

Also could you include the version of notarize that you're using?

I use [email protected] which in turn uses @electron/[email protected].

To add additional context, my CI, like @fempyrean, also runs on GH actions. I use macos-latest for the macOS builds which runs macOS 14 arm64 runners. I have a suspicion that macOS is perhaps terminating xcrun and hence the lack of a proper exit code (it's null)? However, I don't know of a way to prove this; it's just a theory. At this point, I am not sure where the issue lies.

These failures started about two months ago with version @electron/[email protected] that we had been running since the beginning of this year. Interestingly, the stack trace was slightly different with that version and also the failure was flip-flopping between x64 and arm64 with the following stack trace:

electron-notarize:spawn spawning cmd: xcrun args: [
  'notarytool',
  'submit',
  '/var/folders/g6/rgtlsw6n123b0gt5483s5_cm0000gn/T/electron-notarize-ajOoEA/***.zip',
  '--apple-id',
  '*********',
  '--password',
  '*********',
  '--team-id',
  '*********',
  '--wait',
  '--output-format',
  'json'
] opts: {}
electron-notarize:spawn cmd xcrun terminated with code: null <----NOTE: null exit code.
electron-notarize:helpers work failed
  ⨯ Unexpected end of JSON input  failedTask=build stackTrace=SyntaxError: Unexpected end of JSON input
SyntaxError: Unexpected end of JSON input
    at JSON.parse (<anonymous>)
    at /Users/runner/work/***/node_modules/@electron/notarize/src/notarytool.ts:79:25
    at Generator.next (<anonymous>)
    at fulfilled (/Users/runner/work/***/node_modules/@electron/notarize/lib/notarytool.js:28:58)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
From previous event:
    at processImmediate (node:internal/timers:478:21)
From previous event:
    at readDirectoryAndSign (/Users/runner/work/***/node_modules/app-builder-lib/src/macPackager.ts:478:29)
    at MacPackager.signApp (/Users/runner/work/***/node_modules/app-builder-lib/src/macPackager.ts:488:11)
    at MacPackager.doSignAfterPack (/Users/runner/work/***/node_modules/app-builder-lib/src/platformPackager.ts:336:21)
    at MacPackager.doPack (/Users/runner/work/***/node_modules/app-builder-lib/src/platformPackager.ts:321:7)
    at MacPackager.pack (/Users/runner/work/***/node_modules/app-builder-lib/src/macPackager.ts:198:9)
    at Packager.doBuild (/Users/runner/work/***/node_modules/app-builder-lib/src/packager.ts:445:9)
    at executeFinally (/Users/runner/work/***/node_modules/builder-util/src/promise.ts:12:14)
    at Packager._build (/Users/runner/work/***/node_modules/app-builder-lib/src/packager.ts:379:31)
    at Packager.build (/Users/runner/work/***/node_modules/app-builder-lib/src/packager.ts:340:12)
    at executeFinally (/Users/runner/work/***/node_modules/builder-util/src/promise.ts:12:14)

About three weeks ago, I updated to @electron/[email protected] and the failure still occurs but only inarm64 now. Although I am not sure if that's just a coincidence.

[bradfair]

I'm having this issue also, and came across this reference: actions/runner-images#9811

I've added this step to the action I'm using, and running it a few times to see if it stabilizes:

      
      - name: xcode-select
        if: matrix.os == 'macos-latest'
        run: sudo xcode-select -s /Library/Developer/CommandLineTools

edit: womp womp.

2024-12-06T13:30:29.517Z electron-notarize:spawn cmd xcrun terminated with code: null
2024-12-06T13:30:29.517Z electron-notarize:helpers work failed
  ⨯ Failed to notarize via notarytool.  Failed with unexpected result: 

final edit: I think @praneetloke's suspicion about the xcrun crash is spot on. i found some other references yesterday that suggested changing back to macos-13 in the action runner, and I did that. The process is stable now, so I'm leaving it at that and moving on.

[praneetloke]

@bradfair

i found some other references yesterday that suggested changing back to macos-13 in the action runner, and I did that. The process is stable now, so I'm leaving it at that and moving on.

That's a great idea!

Also I took a look at GH's hosted runners and their macos-13 runners are Intel-based with 14GB RAM. Their M1 runners are half of that at 7GB. My guess is xcrun is terminated as a result of low memory sometimes. Though, again, I don't have a way to prove this right now.

[mifi]

Same thing just happened here on macos latest:

https://github.com/mifi/lossless-cut/actions/runs/12211950166/job/34069829185
I've had a scheduled job run every day and I believe today is the first time i've seen this error. also macos-latest (needed for apple silicon builds). Maybe we could impleemtn a retry?

[praneetloke]

@mifi it looks like you are using electron-builder, it supports cross-platform compilation for arm64 on Intel-based macOS. However, if this was the first time you've seen a problem, you might want to check if you've got a pending Apple terms agreement waiting to be accepted in your Apple Developer Account. (I got one yesterday that I accepted.) If you've already accepted that, it would be helpful if you could observe your builds for a few days at least to see how often it fails with macos-latest, if at all.

[mifi]

got a pending Apple terms agreement waiting to be accepted in your Apple Developer Account.

That seems like it might have been the problem. I also got pending terms/agreements that day. Now it seems to be succeeding. Will keep a look at the job going forward