Help with a strange native package — @mapbox/mapbox-gl-native-5.0.2

[michalrus]

Hello!

Could you give some pointers on how to nix @mapbox/mapbox-gl-native-5.0.2? https://www.npmjs.com/package/@mapbox/mapbox-gl-native/v/5.0.2

It seems to be using node-pre-gyp to download pre-built binaries from AWS.

On the other hand, when I try to build it from source ( https://github.com/mapbox/mapbox-gl-native/tree/node-v5.0.2 ), it still complains about lack of internet access…

Perhaps this one needs something in between?

Thanks a lot!

[svanderburg]

I guess you want to include mapbox-gl-native as a dependency of a development project? I suspect that you are running into trouble because one of the dependencies of the project (or a transitive dependency thereof) contains a script directive that most likely attempts to download a binary package from an external URL.

Sadly, this what I have seen for several packages that require native dependencies -- since NPM does not support them out of the box, people invent their own ways and delivery mechanisms to provide them.

As you may probably already know, in pure build environments in Nix (on Linux) network access is restricted, except for so-called fixed output derivations in which output hashes must be known in advance (which is used for functions, such as fetchurl and fetchgit).

To fix this problem, there are two things you can do. For both options, you must first create an override expression to adapt the build of your project (in the README.md of node2nix you can find various examples).

Then you can either "cheat", by disabling sandboxing, by providing the following argument as an override to the package function:

__noChroot = true;

the alternative is patching the build, e.g. by executing sed instructions to change the problematic packages, so that it no longer downloads the binary. The basic idea is that you download and provide this binary yourself by using the Nix expression language and use patch instructions to inject it into the dependency tree yourself.

The builder environment should also support the preInstall hook -- you can use this to carry out your modifications before the builder runs npm install.

I haven't investigated your package specifically, but I have also noticed that some of these packages (that try to download binaries) might have a fall back to compiling the binaries from sources. Maybe that's also possible with one of your dependencies.

I hope these pointers help. Patching the modules before the installation is a bit tricky, but that's works the best to give an optimal deployment experience with Nix.

[michalrus]

I guess you want to include mapbox-gl-native as a dependency of a development project?

That will be most likely needed, but that alone doesn’t work.

I suspect that you are running into trouble because one of the dependencies of the project (or a transitive dependency thereof) contains a script directive that most likely attempts to download a binary package from an external URL.

Yes, that’s the case. :/

Then you can either "cheat", by disabling sandboxing, by providing the following argument as an override to the package function:

__noChroot = true;

Ohhh, and then the builder will have network access? :o Whoa, I didn’t know that, eerie.

the alternative is patching the build, e.g. by executing sed instructions to change the problematic packages, so that it no longer downloads the binary. The basic idea is that you download and provide this binary yourself by using the Nix expression language and use patch instructions to inject it into the dependency tree yourself.

I would most like to do this, but I don’t know enough about Node packaging.

I hope these pointers help.

Yes, thanks a lot! What I did temporarily was extraction of tileserver-gl from their official OCI/Docker image, which can be downloaded using pkgs.dockerTools.pullImage. I’ll try to do it the way you described. Thanks!

[svanderburg]

@michalrus ah ok, at least you know where to get the binary. Then the only challenge is to make sure that it's in the right place, and (maybe) force the script to not download it.

To debug these kinds of scenarios strace is my friend.

[CMCDragonkai]

I guess you want to include mapbox-gl-native as a dependency of a development project? I suspect that you are running into trouble because one of the dependencies of the project (or a transitive dependency thereof) contains a script directive that most likely attempts to download a binary package from an external URL.

But node2nix does seem to support prebuildify based native packages. For example I don't have any problem using node2nix with fd-lock, leveldown, or utp-native. They end up look like this in the nix/store:

                  │   │   ├── prebuilds
                  │   │   │   ├── android-arm
*                 │   │   │   │   └── node.napi.armv7.node
                  │   │   │   ├── android-arm64
*                 │   │   │   │   └── node.napi.armv8.node
                  │   │   │   ├── darwin-x64+arm64
*                 │   │   │   │   └── node.napi.node
                  │   │   │   ├── linux-arm
*                 │   │   │   │   ├── node.napi.armv6.node
*                 │   │   │   │   └── node.napi.armv7.node
                  │   │   │   ├── linux-arm64
*                 │   │   │   │   └── node.napi.armv8.node
                  │   │   │   ├── linux-x64
*                 │   │   │   │   ├── node.napi.glibc.node
*                 │   │   │   │   └── node.napi.musl.node
                  │   │   │   ├── win32-ia32
*                 │   │   │   │   └── node.napi.node
                  │   │   │   └── win32-x64
*                 │   │   │       └── node.napi.node

And because these are under the prebuilds directory, that has to mean that the generated derivation ends up fetching these while running npm install right? These shared objects are hosted on GitHub release pages. And I don't see any compilation messages, if they were compiling from source.

[CMCDragonkai]

I just realised that those files are there because they were distributed as part of the npm package repository. They were using prebuildify, and thus it expected that prebuilds was already installed by npm. This means the OP's problem is because the package does a separate download step done after the package is installed with npm install, and that's where the problem lies.