How does asaar work in Electron for Sharp?

[damms005]

Question about an existing feature

What are you trying to achieve?

I want to understand why the installation docs still have:

Ensure sharp is unpacked from the ASAR archive file using the asarUnpack

option.

{
  "build": {
    "asar": true,
    "asarUnpack": [
      "**/node_modules/sharp/**/*",
      "**/node_modules/@img/**/*"
    ]
  }
}

Meanwhile, the linked Electron doc has:

asar = true [AsarOptions](https://www.electron.build/configuration/configuration.html#AsarOptions) | Boolean | “undefined” - Whether to package the application’s source code into an archive, using [Electron’s archive format](http://electron.atom.io/docs/tutorial/application-packaging/).

Node modules, that must be unpacked, will be detected automatically, you don’t need to explicitly set [asarUnpack](https://www.electron.build/configuration/configuration.html#configuration-asarUnpack) - please file an issue if this doesn’t work.

asarUnpack Array<String> | String | “undefined” - A [glob patterns](https://www.electron.build/file-patterns) relative to the [app directory](https://www.electron.build/configuration/configuration.html#MetadataDirectories-app), which specifies which files to unpack when creating the [asar](http://electron.atom.io/docs/tutorial/application-packaging/) archive.

Context:
I have an Electron app that I want to implement Sharp for, but whenever I package it for production and I use Sharp, the API exposed via contextBridge is not available in the renderer process.

export function bootMainWindow() {
  contextBridge.exposeInMainWorld('schoolserver', getSchoolServerBridge()) // schoolserver is undefined when I use Sharp in the code
}

When you searched for similar issues, what did you find that might be related?

I think that section of the Sharp docs need to be removed or updated to prevent confusion for rookies like me

Please provide a minimal, standalone code sample, without other dependencies, that demonstrates this question

N/A

Please provide sample image(s) that help explain this question

N/A

PS: @lovell can you help or provide paid consultation to teach me how cross-platform deployments work in node? I think I am having an issue related to this in my OSS project and it is saddening that I do not have any clue on how to approach solving it 😞

[lovell]

I want to understand why the installation docs still have...

The default settings in Electron builder results in .node files (shared libraries) being dlopened from temporary, non-deterministic locations. By explicitly stating that all sharp-related files should be unpacked, the relative locations of these become deterministic.

// schoolserver is undefined when I use Sharp in the code

There's not enough information here to help, sorry, and getSchoolServerBridge isn't part of any sharp logic. The issue you've linked to looks like it relates to sqlite rather than sharp.

My current advice to anyone working on multi or cross platform Node.js apps is to use either yarn 3+ or pnpm to manage dependencies via the supportedArchitectures feature.

[damms005]

Yeah @lovell you're spot on that getSchoolServerBridge isn't part of any Sharp logic. I will try again with asarUnpack config. Thank you.

Regarding using pnpm/yarn 3+ for multi/cross platform Node apps with the supportedArchitectures feature, thanks a lot for the advice despite unrelated to Sharp. So, just to be clear, do you mean that I can start tackling the issue I referenced in my repo by switching to pnpm/yarn 3+ and ensuring that all architectures I want to support is listed in supportedArchitectures?

[lovell]

@damms005 Were you able to make any progress with this?

[damms005]

Still working on it. Will definitely update this issue ASAP

[damms005]

Now I think I made some progress, @lovell

After some days in the rabbit hole, I was able to piece together something that may help improve the cross-platform support of Sharp, and hopefully fix the compilation issue I am having.

First, this is my test env:

❯ node --version
v20.12.0

❯ npm --version 
10.5.0

Like I mentioned above, the issue I have is with an Electron app. This Electron app is built with Quasar.

When I build the project, I noticed this portion of the output:

 App •  WAIT  • Bundling app with electron-builder...
  • electron-builder  version=24.13.3 os=5.15.0-101-generic
  • writing effective config  file=dist/electron/Packaged/builder-effective-config.yaml
  • rebuilding native dependencies  [email protected] platform=linux arch=x64
  • packaging       platform=linux arch=x64 electron=29.1.6 appOutDir=dist/electron/Packaged/linux-unpacked
  • building        target=AppImage arch=x64 file=dist/electron/Packaged/SchoolServer-14.5.61.AppImage
  • application Linux category is set to default "Utility"  reason=linux.category is not set and cannot map from macOS docs=https://www.electron.build/configuration/linux

 App •  DONE  • electron-builder built the app • 15537ms


 Build succeeded

From this output, I was led to believe that electron-builder was only able to "see" sqlite3 as a native dependency package. I have sharp also installed in the same project. So I find it weird that something about sharp makes electron-builder not "see" it as a native dependency and hence did not build it.

To be sure this is not some randomly occurring issue, I then created a new project, looked for some other random native package, included sharp and sqlite3, so that I have 3 native packages in the project.

Interestingly, the 2 other dependencies were built, but sharp was still not built, as shown in the build output, again:

 App •  WAIT  • Bundling app with electron-builder...
  • electron-builder  version=24.13.3 os=5.15.0-101-generic
  • writing effective config  file=dist/electron/Packaged/builder-effective-config.yaml
  • rebuilding native dependencies  [email protected], [email protected] platform=linux arch=x64
  • install prebuilt binary  name=sqlite3 version=5.1.7 platform=linux arch=x64 napi=
  • install prebuilt binary  name=drivelist version=11.2.2 platform=linux arch=x64 napi=
  • build native dependency from sources  name=drivelist
                                          version=11.2.2
                                          platform=linux
                                          arch=x64
                                          napi=
reason=prebuild-install failed with error (run with env DEBUG=electron-builder to get more information)
                                          error=prebuild-install info begin Prebuild-install version 7.1.2
    prebuild-install warn install prebuilt binaries enforced with --force!
    prebuild-install warn install prebuilt binaries may be out of date!
    prebuild-install info looking for local prebuild @ prebuilds/drivelist-v11.2.2-napi-v8-linux-x64.tar.gz
    prebuild-install info looking for cached prebuild @ /home/damms005/.npm/_prebuilds/cad401-drivelist-v11.2.2-napi-v8-linux-x64.tar.gz
    prebuild-install http request GET https://github.com/balena-io-modules/drivelist/releases/download/v11.2.2/drivelist-v11.2.2-napi-v8-linux-x64.tar.gz
    prebuild-install http 404 https://github.com/balena-io-modules/drivelist/releases/download/v11.2.2/drivelist-v11.2.2-napi-v8-linux-x64.tar.gz
    prebuild-install warn install No prebuilt binaries found (target=8 runtime=napi arch=x64 libc= platform=linux)
    
  • build native dependency from sources  name=sqlite3
                                          version=5.1.7
                                          platform=linux
                                          arch=x64
                                          napi=
reason=prebuild-install failed with error (run with env DEBUG=electron-builder to get more information)
                                          error=prebuild-install info begin Prebuild-install version 7.1.2
    prebuild-install warn This package does not support N-API version 36
    prebuild-install warn install prebuilt binaries enforced with --force!
    prebuild-install warn install prebuilt binaries may be out of date!
    prebuild-install info looking for local prebuild @ prebuilds/sqlite3-v5.1.7-napi-v36-linux-x64.tar.gz
    prebuild-install info looking for cached prebuild @ /home/damms005/.npm/_prebuilds/c8064a-sqlite3-v5.1.7-napi-v36-linux-x64.tar.gz
    prebuild-install http request GET https://github.com/TryGhost/node-sqlite3/releases/download/v5.1.7/sqlite3-v5.1.7-napi-v36-linux-x64.tar.gz
    prebuild-install http 404 https://github.com/TryGhost/node-sqlite3/releases/download/v5.1.7/sqlite3-v5.1.7-napi-v36-linux-x64.tar.gz
    prebuild-install warn install No prebuilt binaries found (target=36 runtime=napi arch=x64 libc= platform=linux)
    
  • packaging       platform=linux arch=x64 electron=29.1.6 appOutDir=dist/electron/Packaged/linux-unpacked
  • building        target=AppImage arch=x64 file=dist/electron/Packaged/Sharpie-0.0.1.AppImage
  • application Linux category is set to default "Utility"  reason=linux.category is not set and cannot map from macOS docs=https://www.electron.build/configuration/linux

 App •  DONE  • electron-builder built the app • 68289ms


 Build succeeded

Now, this is where I need your help. I do not have the expertise to debug why sharp is the only native package left out. This is why I have made the reproduction repo for this issue, perhaps if you pull it down and look at it, and compare your approach of this native thing with how those other two (driverslist and sqlite3) do their thing and are able to be successfully picked up by the builder, then perhaps it may help in fixing this issue for a larger audience.

I ensured that the README is descriptive enough to allow you reproduce it locally. Furthermore, I configured GitHub Actions and ensured an action reproduces the issue, to ensure it is not some local issue with my setup.

I hope this provides a little bit more of info to help figure this out.

Please let me know if you have any questions or need any more details.

Thank you.

[damms005]

And yes, I don't have a lot of options with yarn neither. That's why I've got to go through all the troubles of the efforts I made with my previous comment with the repro repo.

[lovell]

Thanks for the updates, I can't find where you've added the asarUnpack configuration to https://github.com/damms005/sharpie - am I missing something?

https://github.com/search?q=repo%3Adamms005%2Fsharpie%20asarUnpack&type=code

It might be worth asking the quasar maintainers again about yarn support, as the last update was 2 years ago, and in that time yarn v1 has bit-rotted by another 2 years. (An open source library that continues to dogmatically recommend yarn v1 over all other package managers has, rather sadly, become a red flag for me.)

[damms005]

Yeah, you're correct about the red flag. It is now much so in retrospect, but not much so few years ago when I opted to use quasar for the project. Stuck in this hole, sadly.

Also, this should have been a different issue at this stage, because the progress of this thread has steered off the original asarUnpack issue.

Unfortunately, the quasar team don't have builder improvements on their roadmap at the moment. I mean it is even Yarn v4+ already and they are still insisting on v1.

Perhaps I should look for help with the maintainers of the builder package itself at electron-builder?

However, if you can find some time later to look into this, I can create a new issue/discussion titled something like "Appeal for improved cross-platform support" or similar, and move this there.

But I appreciate the time you have given to this thread already and if you are not willing to look into why/how those two other packages are able to work nicely with electron-builder while sharp cannot, I understand and respect that, too.

[damms005]

It just dawned on me that I tried every single other fix going down that rabbit hole except the very instruction about asarUnpack

It now works. Thank you.

However, please how does this asarUnpack option work, compared to the rebuilding approach?

I now understand, buy this rabbit hole, that rebuilding essentially builds the package from source based on the current OS and arch. This ensures that the bundles Electron app's node_modules folder contains the necessary binaries for the native package. Can you spare some sweat and explain how this asarUnpack option works to complement this understanding of mine, please?

PS: This sounds noob and stupid, but I don't know if it has any bearing on getting sharp to build natively with electron-builder: the other packages have their binding.gyp file in the root of the project: (ref 1 and ref 2), while sharp's is nested

[damms005]

Just thought to add that scanning seems to be done one-level deep? as mentioned in this electron-builder comment. The point I made above with nesting binding.gyp?

[lovell]

As of v0.33.0, sharp no longer includes a binding.gyp in the package root, instead nesting it as you've seen. This is to actively prevent packaging tooling from attempting to invoke node-gyp on it, which more often than not fails with unhelpful error messages.

Given almost all platforms are fully-supported with prebuilt binaries, sharp declares a series of optionalDependencies that are arch/platform/libc specific.

sharp/package.json

Lines 143 to 163 in 02fd565

	"optionalDependencies": {
	"@img/sharp-darwin-arm64": "0.33.3",
	"@img/sharp-darwin-x64": "0.33.3",
	"@img/sharp-libvips-darwin-arm64": "1.0.2",
	"@img/sharp-libvips-darwin-x64": "1.0.2",
	"@img/sharp-libvips-linux-arm": "1.0.2",
	"@img/sharp-libvips-linux-arm64": "1.0.2",
	"@img/sharp-libvips-linux-s390x": "1.0.2",
	"@img/sharp-libvips-linux-x64": "1.0.2",
	"@img/sharp-libvips-linuxmusl-arm64": "1.0.2",
	"@img/sharp-libvips-linuxmusl-x64": "1.0.2",
	"@img/sharp-linux-arm": "0.33.3",
	"@img/sharp-linux-arm64": "0.33.3",
	"@img/sharp-linux-s390x": "0.33.3",
	"@img/sharp-linux-x64": "0.33.3",
	"@img/sharp-linuxmusl-arm64": "0.33.3",
	"@img/sharp-linuxmusl-x64": "0.33.3",
	"@img/sharp-wasm32": "0.33.3",
	"@img/sharp-win32-ia32": "0.33.3",
	"@img/sharp-win32-x64": "0.33.3"
	},

The npm registry contains this data, which is used by package managers such as npm, yarn and pnpm to install the relevant package(s).

$ npm view --json @img/sharp-linux-x64 cpu os libc
{
  "cpu": "x64",
  "os": "linux",
  "libc": "glibc"
}

This approach is becoming more prevalent (see also parcel, esbuild, swc etc.) and I suspect electron-builder would benefit from using the same approach as other package managers by inspecting npm registry data to determine which packages to include.

[damms005]

That's a 💯

Thanks a lot, @lovell