Operation not permitted during dmg building on macOS 11 Big Sur #5465

gnattu · 2020-12-10T16:58:19Z

Version: [email protected]

Electron Version:[email protected]
Electron Type (current, beta, nightly):current

Electron updater: [email protected]

Target: dmg for macOS

After setting DEBUG=electron-builder DEBUG_DMG=true, the console logs error during copy:

2020-12-10 11:45:43.585 copy-helper[54295:350710] About to copy "/path/app.app".
2020-12-10 11:45:44.255 copy-helper[54295:350710] copy error (canceling): /Volumes/img/app.app/Contents/Resources/app.asar.unpacked/node_modules/npm/node_modules/libnpmorg/.BC.T_K9LSTf: Operation not permitted
2020-12-10 11:45:44.255 copy-helper[54295:350710] Copy finished with error 1 (Operation not permitted).
................................Error 1 (Operation not permitted).
could not access /Volumes/img/app.app/Contents/Resources/app.asar.unpacked/node_modules/npm/node_modules/libnpmorg/.BC.T_K9LSTf - Operation not permitted

Then building ultimately failed with:

2020-12-10 11:45:45.386 diskimages-helper[54280:350655] *useEffectiveIDs**** euid/egid changed to 501,20 (uid/gid is 501,20)
DIHLDiskImageCreate() returned 1
hdiutil: create: returning 1
hdiutil: create failed - Operation not permitted

...

 ⨯ hdiutil exited with code ERR_ELECTRON_BUILDER_CANNOT_EXECUTE  stackTrace=
                                                                    Error: hdiutil exited with code ERR_ELECTRON_BUILDER_CANNOT_EXECUTE
                                                                        at ChildProcess.<anonymous> (/Users/gnattu/src/poi/node_modules/builder-util/src/util.ts:243:14)
                                                                        at Object.onceWrapper (node:events:483:26)
                                                                        at ChildProcess.emit (node:events:376:20)
                                                                        at maybeClose (node:internal/child_process:1063:16)
                                                                        at Process.ChildProcess._handle.onexit (node:internal/child_process:295:5)

I can create an image with appdmg for the same app.

The text was updated successfully, but these errors were encountered:

gnattu · 2020-12-11T00:22:23Z

By removing "-fs", "HFS+", "-fsargs", "-c c=64,a=16,e=16", dmg creation does work on Big Sur. Looks like HFS+ image creation is broken.

mmaietta · 2020-12-13T18:28:45Z

Copying from the other issue for posterity's sake.

[This] seems to just need a one-liner PR change, but I don't think letting it use the default setting will be backward compatible as APFS is only available High Sierra 10.13+. It would also require verifying that the default value is consistent across all recent OS versions, so an explicit -fs arg might still be necessary.

Also, from a quick search for hdiutil returning error code 1, it might be due to the target file/directory path not existing.
Check out these two related to dmg creation, coincidentally from appdmg.
LinusU/node-appdmg#65
LinusU/node-appdmg#63 (comment)

{ [Error: Error running `hdiutil`! Exit code was 1]
  exitCode: 1,
  stdout: 'Preparing imaging engine…\nReading Protective Master Boot Record (MBR : 0)…\n',
  stderr: '\nhdiutil: convert failed - Invalid argument\n' }

lutzroeder · 2020-12-17T04:58:06Z

@gnattu can you check the folder for any files with extended attributes (using xattr).

gnattu · 2020-12-17T23:54:17Z

@lutzroeder Yes, I can confirm that some files are incorrectly signed:

xattr PULL_REQUEST_TEMPLATE 
com.apple.cs.CodeDirectory
com.apple.cs.CodeEntitlements
com.apple.cs.CodeRequirements
com.apple.cs.CodeRequirements-1
com.apple.cs.CodeSignature

The file is just a text file indicating a PR template, but it is detected as a binary and got signed.

lutzroeder · 2020-12-18T00:40:30Z

@gnattu are you using [email protected] or [email protected] and macOS BigSur? What is the name and extension of the file that got misclassified? Trying to understand if this is the same issue as #5487.

gnattu · 2020-12-18T02:27:53Z

It's [email protected]. The file has no extension at all, like PULL_REQUEST_TEMPLATE.

gnattu · 2020-12-19T02:28:18Z

For my case we need more than a revert as upstream detects files with an emoji as binary: gjtorikian/isBinaryFile#22

And npm's PULL_REQUEST_TEMPLATE does contain emoji, so PULL_REQUEST_TEMPLATE will still get signed after the revert.

Maybe add a case to ignore PULL_REQUEST_TEMPLATE:

if (path.basename(filePath)[0] !== '.' && path.basename(filePath) != 'PULL_REQUEST_TEMPLATE') {

As a workaround?

mmaietta · 2020-12-19T15:57:24Z

As opposed to using a specific name-based check, can we use another comparable package that is designed for binary detection? Such as: https://www.npmjs.com/package/istextorbinary

@gnattu Is there a minimum/sample repo where I can easily repro/test this on?

gnattu · 2020-12-19T18:15:38Z

@mmaietta You can use the electron-quick-start.

Then add some dependencies inlcuding emoji text files to package.json. For example, npm.

Then run electron-builder with code sign enabled after npm i, you will get the error.

You can then check with xattr for files in app.asar.unpacked, and you will notice text files with emoji get signed.

mmaietta · 2020-12-20T06:18:13Z

@gnattu I've tried as you described and, unfortunately, I can't repro locally on both Big Sur or Catalina. No app.asar.unpacked is created. Running with DEBUG=electron-osx-sign npx electron-builder --dir doesn't show the file(s) being signed either.
What's your EB build config?
If you can create a sample repo or patch file(s) for me to apply the changes locally, it'd help a bunch.

Since I cant test atm, my suggestion for the interim is to flip the binary validation to no longer just check extensions and try istextorbinary:

import { isText, isBinary, getEncoding } from 'istextorbinary'

isBinary(aFilename, aBuffer) // returns true if a binary file otherwise false, checks filename then buffer

The quickest swap would probably just be to use isBinary(filePath, null) for the current extension-check functionality. Best usage of the package would be to include the Buffer for it to check 24 bytes from start/middle/end.
Can you try that out?

gnattu · 2020-12-20T06:23:27Z

I'm not using build config, which will load the default one I guess.

The package.json looks like below:

{
  "name": "electron-quick-start",
  "version": "1.0.0",
  "description": "A minimal Electron application",
  "main": "main.js",
  "scripts": {
    "start": "electron .",
    "build": "electron-builder -p never"
  },
  "repository": "https://github.com/electron/electron-quick-start",
  "keywords": [
    "Electron",
    "quick",
    "start",
    "tutorial",
    "demo"
  ],
  "author": "GitHub",
  "license": "CC0-1.0",
  "dependencies": {
    "npm": "^6.14.8"
  },
  "devDependencies": {
    "electron": "^11.1.0",
    "electron-builder": "^22.10.3"
  }
}

gnattu · 2020-12-20T06:28:14Z

By the way, I do have code signing using DEBUG=electron-osx-sign npx electron-builder --dir as well:

  electron-osx-sign Signing... /Users/gnattu/src/electron-quick-start/dist/mac-arm64/electron-quick-start.app/Contents/Resources/app.asar.unpacked/node_modules/npm/node_modules/libnpmaccess/PULL_REQUEST_TEMPLATE +143ms

gnattu · 2020-12-20T07:36:53Z

@mmaietta I tried istextorbinary and it does not work the same way as isBinaryFile. It will only perform extension check for filenames, and we have to provide buffer for content encoding checking ourselves.

istextorbinary's isBinary does return false for files contain emoji.

…. Exposing `getFilePathIfBinarySync` to app-builder-lib to consolidate binary detection logic

mmaietta · 2020-12-21T00:58:15Z

I was able to get it working in my limited local testing. Would you mind giving this branch a try?
https://github.com/mmaietta/electron-builder/tree/fix/BinaryDetectionSigning

…. Exposing `getFilePathIfBinarySync` to app-builder-lib to consolidate binary detection logic

gnattu · 2020-12-21T19:22:34Z

@mmaietta the branch is not runnable for my project(the actual one):

getFilePathIfBinaryAsync is not defined  stackTrace

https://github.com/mmaietta/electron-builder/blob/f8bc3ec31c905ec019f6f0c2b9b14d4a7d7c5237/packages/app-builder-lib/electron-osx-sign/util.js#L204 should be changed as well to getFilePathIfBinarySync.

But after manually editing, the build still fails as the icns icon files get signed incorrectly, we should not drop the extension check as the extension check provided in istextorbinary is not good enough.

After adding back if (path.extname(filePath).indexOf(' ') >= 0) { to https://github.com/mmaietta/electron-builder/blob/f8bc3ec31c905ec019f6f0c2b9b14d4a7d7c5237/packages/app-builder-lib/electron-osx-sign/util.js#L215, the project complied successfully.

mmaietta · 2020-12-21T22:28:54Z

Sorry bout that. I've pushed an update to the branch, consolidated more of the logic within the binary check function. This also propagates the improvements to unpackDetector

* Check if each binary path exists, otherwise resolve all binaries from the artifact's app Contents path * #5465: Switching from `isBinary` to `istextorbinary`. Exposing `getFilePathIfBinarySync` to app-builder-lib to consolidate binary detection logic * Consolidating binary path detection with extension overrides. * Update lockfile Co-authored-by: Mike Maietta <[email protected]>

gnattu · 2020-12-31T06:04:07Z

Close as #5493 merged

tmacwill · 2021-02-10T06:20:23Z

I'm not sure the above PR fixes this issue in all cases. I'm using macOS Big Sur 11.1 on an Apple M1 chip in a Rosetta 2 terminal. I've create a minimal repro case with electron-builder 22.10.4 at this repository, using the macOS JDK as a test: https://github.com/tmacwill/electron-builder-dmg-repro

When I run yarn dist, I get the following error:

  • signing         file=dist/mac/electron-webpack-quick-start.app identityName=Developer ID Application: xxx identityHash=xxx provisioningProfile=none
  • building        target=macOS zip arch=x64 file=dist/electron-webpack-quick-start-0.0.0-mac.zip
  • building        target=DMG arch=x64 file=dist/electron-webpack-quick-start-0.0.0.dmg
  • building embedded block map  file=dist/electron-webpack-quick-start-0.0.0-mac.zip
  ⨯ hdiutil exited with code ERR_ELECTRON_BUILDER_CANNOT_EXECUTE  stackTrace=
                                                                    Error: hdiutil exited with code ERR_ELECTRON_BUILDER_CANNOT_EXECUTE
                                                                        at ChildProcess.<anonymous> (/Users/tmac/tmp/electron-webpack-quick-start/node_modules/builder-util/src/util.ts:243:14)
                                                                        at Object.onceWrapper (events.js:421:26)
                                                                        at ChildProcess.emit (events.js:314:20)
                                                                        at maybeClose (internal/child_process.js:1021:16)
                                                                        at Process.ChildProcess._handle.onexit (internal/child_process.js:286:5)

Using xattr in the dist directory shows that several files have extended attributes:

$ xattr dist/mac/electron-webpack-quick-start.app/Contents/Resources/static/jdk/lib/modules
com.apple.cs.CodeDirectory
com.apple.cs.CodeEntitlements
com.apple.cs.CodeRequirements
com.apple.cs.CodeRequirements-1
com.apple.cs.CodeSignature

$ xattr dist/mac/electron-webpack-quick-start.app/Contents/Resources/static/jdk/lib/security/cacerts
com.apple.cs.CodeDirectory
com.apple.cs.CodeEntitlements
com.apple.cs.CodeRequirements
com.apple.cs.CodeRequirements-1
com.apple.cs.CodeSignature

However, they don't have extended attributes before the build:

$ xattr static/jdk/lib/modules
$ xattr static/jdk/lib/security/cacerts
$

If you remove the files that are getting extended attributes added and then run yarn dist, the build succeeds:

$ rm -rf static/jdk/lib/modules static/jdk/lib/security/cacerts
$ yarn dist

The static/jdk directory also contains a number of binary files that are not given extended attributes after building. For instance, static/jdk/bin/java is a binary file, but it's not given extended attributes in dist:

$ xattr dist/mac/electron-webpack-quick-start.app/Contents/Resources/static/jdk/bin/java
$

Thanks!

EDIT: It looks like that PR might have been actually merged after the 22.10.4 release was published, but I created a small test file that just runs getFilePathIfBinarySync from master on the above files that incorrectly get extended attributes, and the behavior is the same. Is there ever a case where a non-executable binary file that doesn't pass the existing checks (e.g., ends in dylib) should be code signed? If not, then getFilePathIfBinarySync could also check for fs.accessSync(filePath, fs.constants.X_OK).

mmaietta · 2021-02-14T05:07:53Z

Is there ever a case where a non-executable binary file that doesn't pass the existing checks (e.g., ends in dylib) should be code signed? If not, then getFilePathIfBinarySync could also check for fs.accessSync(filePath, fs.constants.X_OK).

Honestly, I'm not sure. It seems electron-builder and electron-osx-sign both have similar odd issues in discerning binaries and the core issue is not particularly fixable with ease.

Happy to run any tests/suggestions for you though. (Also happy to help you get a local dev/test env setup with electron-builder using yalc
I tried your fs.access and receive this error in my testing:

//  fs.accessSync(filePath, fs.constants.X_OK)


⨯ EACCES: permission denied, access '/electron-builder-dmg-repro/dist/mac/electron-webpack-quick-start.app/Contents/Info.plist' 
stackTrace=
Error: EACCES: permission denied, access '/electron-builder-dmg-repro/dist/mac/electron-webpack-quick-start.app/Contents/Info.plist'
    at Object.accessSync (fs.js:203:3)
    at module.exports.getFilePathIfBinarySync (/electron-builder-dmg-repro/.yalc/app-builder-lib/electron-osx-sign/util.js:162:10)

I found another npm package, https://github.com/mscdex/mmmagic, that could potentially help here, but it's a native module. Not sure if that's too heavy of a package to introduce into this tool, but right now I can't even get it to rebuild with the right node version

tmacwill · 2021-02-14T05:23:46Z

Thanks for responding! I think that's actually the expected behavior of fs.accessSync--it throws an exception if the given permissions don't exist on the file (as opposed to returning true or false): https://nodejs.org/api/fs.html#fs_fs_accesssync_path_mode.

If we're not sure about which files should be signed vs. left unsigned, though, another potential option is to allow projects to specify an exclude list in the configuration or have a boolean option like onlySignExecutables.

I'm also happy to help test + submit a PR, but indeed wasn't sure how to get a dev environment set up :)

mmaietta · 2021-02-14T17:40:01Z

Okiedokie, two items to do!

Optional signing

For specifically adding signing or ignoring files you can leverage either/both of these two EB features (ignore and binaries). This should be able to resolve the problem you're immediately facing right now.

supplemental binaries: https://github.com/electron-userland/electron-builder/blob/master/packages/app-builder-lib/electron-osx-sign/sign.js#L135-L141
ignore function or regex: https://github.com/electron-userland/electron-builder/blob/master/packages/app-builder-lib/electron-osx-sign/sign.js#L124-L131

To setup a local dev environment

Original dev env setup can be found here: #5426 (comment)
Updated instructions are below:

For local development, I used yalc so I could test changes with the electron-quick-start app
Env setup from scratch (line by line for an easy copy-paste)
yarn global add yalc

git clone https://github.com/electron-builder/electron-builder.git
git clone https://github.com/electron/electron-quick-start.git

cd electron-quick-start
npm install && npm install -D electron-builder

# Setup link for yalc's local "repo"
yalc link app-builder-lib builder-util builder-util-runtime dmg-builder electron-builder electron-publish electron-builder-squirrel-windows electron-forge-maker-appimage electron-forge-maker-nsis electron-forge-maker-nsis-web electron-forge-maker-snap electron-updater
The magical script for whenever you make changes to electron-builder! Rebuilds electron-builder, patches the npm module in electron-quick-start, and begins packaging. Ready for copy-paste into terminal
pushd ../electron-builder
yarn compile
find packages/ -type d -maxdepth 1 -print0 | xargs -0 -L1 sh -c 'cd "$0" && yalc push'
popd

tmacwill · 2021-02-14T19:40:46Z

Got it, thanks! Is there a way to specify ignore in package.json? I see binaries listed on https://www.electron.build/configuration/mac.html, but not ignore. I tried adding ignore and got this error:

 - configuration.mac has an unknown property 'ignore'. These properties are valid:
   object { appId?, artifactName?, asar?, asarUnpack?, binaries?, bundleShortVersion?, bundleVersion?, category?, compression?, cscInstallerKeyPassword?, cscInstallerLink?, cscKeyPassword?, cscLink?, darkModeSupport?, detectUpdateChannel?, electronLanguages?, electronUpdaterCompatibility?, entitlements?, entitlementsInherit?, entitlementsLoginHelper?, extendInfo?, extraDistFiles?, extraFiles?, extraResources?, fileAssociations?, files?, forceCodeSigning?, gatekeeperAssess?, generateUpdatesFilesForAllChannels?, hardenedRuntime?, helperBundleId?, helperEHBundleId?, helperGPUBundleId?, helperNPBundleId?, helperPluginBundleId?, helperRendererBundleId?, icon?, identity?, minimumSystemVersion?, protocols?, provisioningProfile?, publish?, releaseInfo?, requirements?, signIgnore?, strictVerify?, target?, type? }

Is there a different field I should be adding ignore to?

mmaietta · 2021-02-15T01:54:54Z

Haha, I just discovered that it's piped from elsewhere
https://github.com/electron-userland/electron-builder/blob/master/packages/app-builder-lib/src/macPackager.ts#L212-L243

So I think you'll need to use signIgnore and it's a string/regex array: opts.signIgnore?.map(it => new RegExp(it))

tmacwill · 2021-02-15T09:11:18Z

Ah got it, that worked for me! I added the two paths that were being incorrectly signed to signIgnore, and my DMG builds successfully now.

I think it'd be helpful to add documentation for signIgnore to https://www.electron.build/configuration/mac.html, since the combination of signIgnore and binaries can indeed fix any issues with electron-builder detecting / not detecting files to sign. (It might also be good not to sign non-executable binary files by default, though I'm not 100% sure what's best.)

Thanks for all of your help, I really appreciate it!

ym13349140 · 2022-02-09T11:59:27Z

Ah got it, that worked for me! I added the two paths that were being incorrectly signed to signIgnore, and my DMG builds successfully now.

I think it'd be helpful to add documentation for signIgnore to https://www.electron.build/configuration/mac.html, since the combination of signIgnore and binaries can indeed fix any issues with electron-builder detecting / not detecting files to sign. (It might also be good not to sign non-executable binary files by default, though I'm not 100% sure what's best.)

Thanks for all of your help, I really appreciate it!

@tmacwill Hello, I recently use signIgnore to ignore some .dll files in my projects, these dll files are in ./dist_electron/mac/application/Contents directory, I tried many times of different value , including ./dist_electron/mac/application/Contents/xxx.dll , .dll$ and array value '[.dll$]', but it never make sense, can you show me how you set this param value? I'd be really appreciate if you can reply me.

gnattu mentioned this issue Dec 10, 2020

feat: add arm64 build support poooi/poi#2344

Closed

gnattu mentioned this issue Dec 11, 2020

feat:(mac): Apple Silicon builds (arm64) #5426

Merged

This was referenced Dec 17, 2020

hdiutil exited with code ERR_ELECTRON_BUILDER_CANNOT_EXECUTE #5431

Closed

Codesign all binary-like files #5322

Merged

lutzroeder mentioned this issue Dec 17, 2020

electron-builder 20.10.3 fails with Operation not permitted #5487

Closed

mmaietta pushed a commit to mmaietta/electron-builder that referenced this issue Dec 20, 2020

electron-userland#5465: Switching from isBinary to istextorbinary…

adfe4c9

…. Exposing `getFilePathIfBinarySync` to app-builder-lib to consolidate binary detection logic

mmaietta pushed a commit to mmaietta/electron-builder that referenced this issue Dec 21, 2020

electron-userland#5465: Switching from isBinary to istextorbinary…

a50e745

…. Exposing `getFilePathIfBinarySync` to app-builder-lib to consolidate binary detection logic

lutzroeder mentioned this issue Dec 21, 2020

Revert "Codesign all binary-like files" #5488

Merged

mmaietta pushed a commit to mmaietta/electron-builder that referenced this issue Dec 21, 2020

electron-userland#5465: Switching from isBinary to istextorbinary…

f8bc3ec

…. Exposing `getFilePathIfBinarySync` to app-builder-lib to consolidate binary detection logic

mmaietta mentioned this issue Dec 21, 2020

fix: binary detection signing #5493

Merged

gnattu closed this as completed Dec 31, 2020

This was referenced Mar 7, 2021

Bump electron-builder from 22.9.1 to 22.10.5 in /electron wireapp/wire-webapp#10666

Merged

Bump electron-builder from 22.9.1 to 22.10.5 Threor/soundboard-frontend#23

Merged

apjanke mentioned this issue Feb 1, 2024

hdiutil create fails with permission denied on file in DMG target, only on some machines octave-app/octave-app#252

Open

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Operation not permitted during dmg building on macOS 11 Big Sur #5465

Operation not permitted during dmg building on macOS 11 Big Sur #5465

gnattu commented Dec 10, 2020

gnattu commented Dec 11, 2020

mmaietta commented Dec 13, 2020

lutzroeder commented Dec 17, 2020

gnattu commented Dec 17, 2020

lutzroeder commented Dec 18, 2020 •

edited

Loading

gnattu commented Dec 18, 2020

gnattu commented Dec 19, 2020

mmaietta commented Dec 19, 2020 •

edited

Loading

gnattu commented Dec 19, 2020 •

edited

Loading

mmaietta commented Dec 20, 2020

gnattu commented Dec 20, 2020

gnattu commented Dec 20, 2020 •

edited

Loading

gnattu commented Dec 20, 2020

mmaietta commented Dec 21, 2020

gnattu commented Dec 21, 2020 •

edited

Loading

mmaietta commented Dec 21, 2020

gnattu commented Dec 31, 2020

tmacwill commented Feb 10, 2021 •

edited

Loading

mmaietta commented Feb 14, 2021 •

edited

Loading

tmacwill commented Feb 14, 2021

mmaietta commented Feb 14, 2021 •

edited

Loading

tmacwill commented Feb 14, 2021 •

edited

Loading

mmaietta commented Feb 15, 2021

tmacwill commented Feb 15, 2021

ym13349140 commented Feb 9, 2022 •

edited

Loading

Operation not permitted during dmg building on macOS 11 Big Sur #5465

Operation not permitted during dmg building on macOS 11 Big Sur #5465

Comments

gnattu commented Dec 10, 2020

gnattu commented Dec 11, 2020

mmaietta commented Dec 13, 2020

lutzroeder commented Dec 17, 2020

gnattu commented Dec 17, 2020

lutzroeder commented Dec 18, 2020 • edited Loading

gnattu commented Dec 18, 2020

gnattu commented Dec 19, 2020

mmaietta commented Dec 19, 2020 • edited Loading

gnattu commented Dec 19, 2020 • edited Loading

mmaietta commented Dec 20, 2020

gnattu commented Dec 20, 2020

gnattu commented Dec 20, 2020 • edited Loading

gnattu commented Dec 20, 2020

mmaietta commented Dec 21, 2020

gnattu commented Dec 21, 2020 • edited Loading

mmaietta commented Dec 21, 2020

gnattu commented Dec 31, 2020

tmacwill commented Feb 10, 2021 • edited Loading

mmaietta commented Feb 14, 2021 • edited Loading

tmacwill commented Feb 14, 2021

mmaietta commented Feb 14, 2021 • edited Loading

Optional signing

To setup a local dev environment

tmacwill commented Feb 14, 2021 • edited Loading

mmaietta commented Feb 15, 2021

tmacwill commented Feb 15, 2021

ym13349140 commented Feb 9, 2022 • edited Loading

lutzroeder commented Dec 18, 2020 •

edited

Loading

mmaietta commented Dec 19, 2020 •

edited

Loading

gnattu commented Dec 19, 2020 •

edited

Loading

gnattu commented Dec 20, 2020 •

edited

Loading

gnattu commented Dec 21, 2020 •

edited

Loading

tmacwill commented Feb 10, 2021 •

edited

Loading

mmaietta commented Feb 14, 2021 •

edited

Loading

mmaietta commented Feb 14, 2021 •

edited

Loading

tmacwill commented Feb 14, 2021 •

edited

Loading

ym13349140 commented Feb 9, 2022 •

edited

Loading