macOS - Electron 3 and sharp 0.21.1 issue

[dziudek]

Hi,

During developing the newest version of Publii I have discovered a very strange issue with using latest version of the sharp module.

On development version all is working fine, but when I create production build and start thumbnail regeneration, the regeneration process crashes with the following errors:

(sharp:8950): GLib-GObject-WARNING **: 19:43:41.423: cannot register existing type 'VipsObject'

(sharp:8950): GLib-CRITICAL **: 19:43:41.423: g_once_init_leave: assertion 'result != 0' failed

(sharp:8950): GLib-GObject-WARNING **: 19:43:41.423: invalid class cast from 'VipsForeignLoadCsv' to '<invalid>' 

I have discovered that using sharp 0.20.8 solves this strange issue.

I am using Electron v.3.0.11, node v.10.14.2 of course I am also using electron-rebuild (v.1.8.2) to compile the native packages. The problem exists only on macOS (newest mojave release), on Windows 10 I did not noticed this issue.

[dziudek]

By the way - why the newest release weights over 2 times more than the old one?

(left - sharp 0.20.8, right - sharp 0.21.1).

[lovell]

Hello, my best guess would be that v8.6.x of libvips is installed globally on the OS X machine. If so, sharp v0.20.x will use it, but sharp v0.21.x will download a more recent libvips v8.7.0 and use that (hence the increased disk usage).

Electron and shared libraries can be a bit of a minefield so I'd recommend either upgrading or removing the globally-installed libvips.

[dziudek]

Hi @lovell ,

Thank you for your answer.

Hello, my best guess would be that v8.6.x of libvips is installed globally on the OS X machine. If so, sharp v0.20.x will use it, but sharp v0.21.x will download a more recent libvips v8.7.0 and use that (hence the increased disk usage).

What about machines without installed libvips? We provide our app to many users and it always works. Regarding your answer - it should mean, that our application cannot generate thumbnails on machines without globally installed libvips.

And the most important thing - do you have any idea regarding the main problem of this issue with the newest sharp? For me it is not a problem to use older version, but I afraid that on the newer electron versions (electron 4 is in beta) I will have to upgrade sharp and then the mentioned problem will occur.

If I can do anything what can help you to investigate this issue - just let me know :)

[lovell]

My reply was specific to the OS X scenario you described rather than general to everyone.

sharp will look for and use a globally-installed libvips (of a minimum version, see the changelog) before downloading its own local copy.

Is there a globally-installed version of libvips on the OS X machine? It is the latest version? If not, does upgrading or removing it help?

[lovell]

@dziudek Were you able to make any progress with this?

[dziudek]

@lovell - sorry for the late reply, in fact due the holidays I totally forgot about this issue - sorry :)

Regarding the issue - I have checked if sharp or libvips is installed globally on my mac:

npm list -g sharp
/usr/local/lib
└── (empty)

ld -llibvips
ld: library not found for -llibvips

So for me it seems that libvips is not installed globally.

[lovell]

Thanks, what does pkg-config --libs vips-cpp return, if anything?

[dziudek]

pkg-config --libs vips-cpp
Package vips-cpp was not found in the pkg-config search path.
Perhaps you should add the directory containing `vips-cpp.pc'
to the PKG_CONFIG_PATH environment variable
No package 'vips-cpp' found

[lovell]

Thanks, what does sharp.versions return for both versions on this machine? (This exposes which dependencies sharp found at runtime.)

[dziudek]

All tests has been made under Electron 3.0.11

• Sharp 0.20.8 - dev version works
• Sharp 0.20.8 - production version works
• Sharp 0.21.3 - dev version works
• Sharp 0.21.3 - production version not work

For sharp 0.20.8 during running dev version and after creating a production package it returns:

{ cairo: '1.14.12',
  croco: '0.6.12',
  exif: '0.6.21',
  fontconfig: '2.12.6',
  freetype: '22.0.16',
  gdkpixbuf: '2.36.11',
  gif: '5.1.4',
  glib: '2.54.3',
  gsf: '1.14.42',
  harfbuzz: '1.7.4',
  jpeg: '',
  lcms: '2.9',
  orc: '0.4.28',
  pango: '1.40.14',
  pixman: '0.34.0',
  png: '1.6.34',
  svg: '2.42.0',
  tiff: '4.0.9',
  vips: '8.6.1',
  webp: '0.6.1',
  xml: '2.9.4' }

For sharp 0.21.3 during running dev version it returns:

{ cairo: '1.16.0',
  croco: '0.6.12',
  exif: '0.6.21',
  fontconfig: '2.13.1',
  freetype: '22.1.16',
  fribidi: '1.0.5',
  gdkpixbuf: '2.38.0',
  gif: '5.1.4',
  glib: '2.58.1',
  gsf: '1.14.44',
  harfbuzz: '2.0.2',
  jpeg: '9.3',
  lcms: '2.9',
  orc: '0.4.28',
  pango: '1.42.4',
  pixman: '0.34.0',
  png: '1.6.35',
  svg: '2.44.8',
  tiff: '4.0.9',
  vips: '8.7.0',
  webp: '1.0.0',
  xml: '2.9.4' }

For sharp 0.21.3 production version (which is not working) it returns (for me it is the same as for dev version but please check by yourself):

{ cairo: '1.16.0',
  croco: '0.6.12',
  exif: '0.6.21',
  fontconfig: '2.13.1',
  freetype: '22.1.16',
  fribidi: '1.0.5',
  gdkpixbuf: '2.38.0',
  gif: '5.1.4',
  glib: '2.58.1',
  gsf: '1.14.44',
  harfbuzz: '2.0.2',
  jpeg: '9.3',
  lcms: '2.9',
  orc: '0.4.28',
  pango: '1.42.4',
  pixman: '0.34.0',
  png: '1.6.35',
  svg: '2.44.8',
  tiff: '4.0.9',
  vips: '8.7.0',
  webp: '1.0.0',
  xml: '2.9.4' }

[lovell]

Thanks for confirming.

What makes the "production" version different? Could something be altering the order of require statements?

Is https://github.com/GetPublii/Publii/blob/master/app/package.json the relevant list of dependencies? I notice keytar is included, which is a native module that I think might also have a dependency on glib/gobject. sharp provides its own glib/gobject but keytar might rely on a system-wide version.

[dziudek]

I have checked production build without keytar, also I have checked production build without generating ASAR package. In both cases the issue still exists.

[dziudek]

Also upgrade to Electron 4.0.2 does not help.

[lovell]

Thanks, the next step is for you to isolate the difference in the "production" environment vs the working "dev" environment. (If you're using Electron 4 then please see #1522.)

[lovell]

Closing due to inactivity but please feel free to re-open with details of the difference between the "production" environment and the working "dev" environment.

[JakeRadMSFT]

@lovell can we re-open this? What information can I help you get?

If I copy sharp from my node_modules into the app it works. Here is a Dif between what works (sharp) and what doesn't (sharp_backup).

$ diff -rq sharp sharp_backup/
Only in sharp/build/Release/obj.target/sharp/src: common.o
Only in sharp/build/Release/obj.target/sharp/src: metadata.o
Only in sharp/build/Release/obj.target/sharp/src: operations.o
Only in sharp/build/Release/obj.target/sharp/src: pipeline.o
Only in sharp/build/Release/obj.target/sharp/src: sharp.o
Only in sharp/build/Release/obj.target/sharp/src: stats.o
Only in sharp/build/Release/obj.target/sharp/src: utilities.o
Files sharp/build/Release/sharp.node and sharp_backup/build/Release/sharp.node differ
Only in sharp/node_modules: .bin

[JakeRadMSFT]

Okay ... I think I figured out the problem ... We're suppose to ship symlinks/aliases.

Electron-packager does a deref and is actually creating copies for each symlink during electron-packager. Since they are copies instead of symlinks ... it is trying to load it multiple times instead of once?

Does that seem plausible?

Worked around it with an afterCopy hook:
(buildPath, electronVersion, platform, arch, callback) => {
let sourcePath = path.resolve(__dirname, "../node_modules/sharp/vendor/lib");
let destPath = path.resolve(buildPath, "./node_modules/sharp/vendor/lib");
fs.removeSync(destPath);
fs.copy(sourcePath,destPath, {
dereference: false,
overwrite: true,
recursive: true,
});
console.log(buildPath);
callback();
},

[lovell]

@JakeRadMSFT electron-packager provides a --no-deref-symlinks flag - see https://github.com/electron-userland/electron-packager/blob/master/usage.txt

[KillerCodeMonkey]

I switched to travis ci for macosx builds (with electron-builder) for my electron (4.2.0) app.

The build runs like charme and no global vips are installed.
brew info vips, says that :D

After i download the artifact and open it on my local macosx (without global vips) i get this error:

(sharp:6164): GLib-GObject-WARNING **: 09:07:12.504: cannot register existing type 'VipsObject'

(sharp:6164): GLib-CRITICAL **: 09:07:12.504: g_once_init_leave: assertion 'result != 0' failed

(sharp:6164): GLib-GObject-WARNING **: 09:07:12.504: invalid class cast from 'VipsForeignLoadCsv' to '<invalid>'

Strange thing: local build has around 212mb, built with the ci: 500mb

node: 10.15.3
yarn: 1.16.0
electron: 4.2.0
electron-builder: 20.39.0
sharp: 0.22.1

[lovell]

https://github.com/electron-userland/electron-packager/blob/master/docs/api.md#derefsymlinks

Please see also #1725, a similar example of webpack mistreating symlinks as real files.