Environment variable for building npm package (rather than just downloading the binary)

[scd31]

Is your feature request related to a problem? Please describe.
I'm trying to package tileserver-gl for Nix, which requires the build process to be sandboxed. Since this npm package tries to download from GH, that doesn't work.

Describe the solution you'd like
I'd like an env var that, when set, causes the npm package to be built from source, rather than downloaded.

Thanks!

[louwers]

There is an old PR from @acalcutt that intergrates cmake-js you might find interesting. #535

[acalcutt]

The main issue with that is the source code required to build the binaries is not included in the npm package, especially after we moved package.json to platform/node

In #535 it was before that move, so it was easier to include the source files. though that made the package very big and the build from source was a bit slow.

[scd31]

Maybe it would make sense to have 2 packages if we can't put the source into the existing one? And then upstream we could decide which one to use via e.g. an environment variable or whatever

[acalcutt]

I've been thinking a bit how I would do this now and started working on https://github.com/acalcutt/maplibre-native-with-source/ . I made this initial test version that worked on linux here ( https://www.npmjs.com/package/@acalcutt/maplibre-native-with-source/v/6.2.1-pre.0 )

As a new npm package I though we could include the 'maplibre-native' repository as a submodules and set up npm to publish only the files required for the build to work, like I did with https://github.com/maplibre/maplibre-native/pull/535/files#diff-34958b02363eec9a904716f0b4a8782b021c6b33b2e8599019e36c88e045240b . I decided to use a custom script combined with the new node presets I am trying to add in #3760 instead of cmake-js, because i remember I was running into option issues the last time I was using cmake-js.

I created a node-ci on the repo which is a modified version of ours here that uses the the build script. as far as I can tell right now it is should work on linux and macos, but I am having trouble getting it to build in windows there. For example, this ci run https://github.com/acalcutt/maplibre-native-with-source/actions/runs/17422970207/job/49464736179

I set the package up so the 'install' script will still try to use node-pre-gyp to try and download binaries, but if that fails it will run the 'build' script. the build script is basically javascript that will run cmake with the presets set. This makes it so you could upload binaries and make the install quicker, but it can fallback and build (or you can now use the --build-from-source flag to force it to build).

I also added a prepack and postpack script, because I found the .npmignore file in the maplibre-native repo basically caused all the files to get excluded. so the prepack renames this file before publish and the postpack renames it back after the publish.

[scd31]

Thanks! Sorry for disappearing there.

What I've done for now is package this into a Nix build script: https://gitlab.scd31.com/stephen/tileserver-gl-nix/-/blob/master/packages/maplibre-gl-native.nix?ref_type=heads
It's a little hacky because we pre-download everything before build time (it's a Nix requirement, to keep things pure)

And then when I need to use it, I symlink it into the node_modules directory so that it doesn't try to download: https://gitlab.scd31.com/stephen/tileserver-gl-nix/-/blob/master/packages/tileserver-gl.nix?ref_type=heads#L19

[acalcutt]

I think you could also use node-pre-gyp's mirror options, though I am not 100% how to use it
https://github.com/mapbox/node-pre-gyp?tab=readme-ov-file#download-binary-files-from-a-mirror

Last time I tried to use it I think I used this syntax with maplibre, but I am not sure this is still valid
npm i --mbgl_binary_host_mirror=https://wifidb.net/demo/maplbre-native/mirror/ubuntu_22.04/

[scd31]

If there was a way to use a file rather than a mirror that would be perfect!

[acalcutt]

But yes, I have suggested something like what you are doing in that past. if you put the lib files in the proper location, like
node_modules\@maplibre\maplibre-gl-native\lib\node-v127\mbgl.node replacing node-v127 with your node ABI version, node-pre-gyp will not try to download the binaries because it sees they already exist (unless you specify flags to redownload)

I use this sometimes locally when i build my own binaries on systems not quite compatible with the binaries we make.

[acalcutt]

A local mirror might be a nice feature to get added into node-pre-gyp though.

[etnav]

But yes, I have suggested something like what you are doing in that past. if you put the lib files in the proper location, like node_modules\@maplibre\maplibre-gl-native\lib\node-v127\mbgl.node replacing node-v127 with your node ABI version, node-pre-gyp will not try to download the binaries because it sees they already exist (unless you specify flags to redownload)

I use this sometimes locally when i build my own binaries on systems not quite compatible with the binaries we make.

@acalcutt Have a similar issue, behind a firewall and cannot use node-pre-gyp to download the binaries, so I tried this in a (GitLab) CI pipeline and had problems. You reckon this also works when using npm i / ci whilst having maplibre-gl-native as a dependency or only when calling node-pre-gyp manually as you mentioned?

[acalcutt]

yes, it should work. if you pre-populate the mbgl.node file at the path I mentioned, and do 'npm i' (or npm ci), node-pre-gyp will see the binary exists and skip downloading it.

You can get the official mbgl.node binaries from the github node releases, like for example ( https://github.com/maplibre/maplibre-native/releases/tag/node-v6.2.0 ) and downloading the binary file that matches your OS/CPU/NodeABI , something like node-v127-linux-x64-Release.tar.gz. Inside that file is the node-v127/mbgl.node that needs to go into node_modules\@maplibre\maplibre-gl-native\lib\. node-pre-gyp typically downloads and extracts that for you, but if you do it yourself it should see the file exists and won't try to download it unless you specify extra flags to rebuilt or redownload.

when you build the binaries yourself, the mbgl.node files end up in platform\node\lib\[node abi folder]\mbgl.node , so you can just copy those [node abi folder] folders to your node modules folder node_modules\@maplibre\maplibre-gl-native\lib\. there are multiple abi versions generated at once that you can either copy just the abi folder you need or all of them (it doesn't really matter it will only use the one that matches your node abi version)