Configure support for depending on pure ESM packages #3841
Labels
Comments
stefreak
added a commit
that referenced
this issue
Mar 6, 2023
We do not use dependabot with npm yet because our build process is currently incompatible with "pure ESM build" dependencies. See also #3841
stefreak
added a commit
that referenced
this issue
Mar 6, 2023
This means more dependabot PRs, and hopefully less manual work keeping Dockerfiles up to date. In some cases careful testing will be required before merging Docker PRs but at least we have a to-do list in the form of dependabot PRs. We do not use dependabot with npm yet because our build process is currently incompatible with "pure ESM build" dependencies. See also #3841
vvagaytsev
pushed a commit
that referenced
this issue
Mar 13, 2023
This means more dependabot PRs, and hopefully less manual work keeping Dockerfiles up to date. In some cases careful testing will be required before merging Docker PRs but at least we have a to-do list in the form of dependabot PRs. We do not use dependabot with npm yet because our build process is currently incompatible with "pure ESM build" dependencies. See also #3841
vvagaytsev
pushed a commit
that referenced
this issue
Mar 13, 2023
This means more dependabot PRs, and hopefully less manual work keeping Dockerfiles up to date. In some cases careful testing will be required before merging Docker PRs but at least we have a to-do list in the form of dependabot PRs. We do not use dependabot with npm yet because our build process is currently incompatible with "pure ESM build" dependencies. See also #3841
|
@Walther Do you have an example of a package that is broken once updated? Just to have a starting point for working on this. |
|
Still causing issues for us, see also #5182 |
stefreak
added a commit
that referenced
this issue
Nov 6, 2023
This had many different effects that derived from it, making this PR so massive. **Modules**: The first stage was to rewrite all the imports to use .js endings everywhere. Then we needed to replace and bump a bunch of dependencies to their ESM version. **Bundling**: Following that we tried to use rollup to bundle everything back into CommonJS so that we can use it in pkg since pkg does not support ESM. This was unsuccessful, and after trying several different single application solutions, we decided we will have to write our own. **Single Executable Binary**: We now use a custom Rust binary which bundles a zipped version of NodeJS, the native extensions and the bundled code. We bundle the code for tree-shaking and size reduction, but also because otherwise we had issues with how we ensure that native modules are loaded correctly and with resolving the imports correctly cross-platform. Then on first run, the binary extracts those files to the filesystem and runs node from there. There are checksum files that we bundle together with the archives and that we store on disk to see if we need to re-extract or not. This also gives us a huge performance boost in some cases - especially when lots of filesystem reads are required - since pkg was patching and proxying those methods and making things slow. **Kubernetes client**: We also had to update the Kubernetes library to the 1.0.0-rc3, with a patch from a fork that contains the changes for kubernetes-client/javascript#1341 That is because the library was using request-promise behind the scenes, which started showing up with unhandled rejection warnings and errors about req not being defined in some cases. That library update required us to change almost every k8s call since the interface now uses objects instead of positional arguments, which makes things much nicer to read and use. We also had to introduce a workaround for using custom certificates together with proxies, something that was previously globally monkey-patched by global-agent. **request-promise**: We no longer depend on request-promise and fully removed the dependency. Fixes #3841 Fixes #4898
github-merge-queue bot
pushed a commit
that referenced
this issue
Nov 6, 2023
This had many different effects that derived from it, making this PR so massive. **Modules**: The first stage was to rewrite all the imports to use .js endings everywhere. Then we needed to replace and bump a bunch of dependencies to their ESM version. **Bundling**: Following that we tried to use rollup to bundle everything back into CommonJS so that we can use it in pkg since pkg does not support ESM. This was unsuccessful, and after trying several different single application solutions, we decided we will have to write our own. **Single Executable Binary**: We now use a custom Rust binary which bundles a zipped version of NodeJS, the native extensions and the bundled code. We bundle the code for tree-shaking and size reduction, but also because otherwise we had issues with how we ensure that native modules are loaded correctly and with resolving the imports correctly cross-platform. Then on first run, the binary extracts those files to the filesystem and runs node from there. There are checksum files that we bundle together with the archives and that we store on disk to see if we need to re-extract or not. This also gives us a huge performance boost in some cases - especially when lots of filesystem reads are required - since pkg was patching and proxying those methods and making things slow. **Kubernetes client**: We also had to update the Kubernetes library to the 1.0.0-rc3, with a patch from a fork that contains the changes for kubernetes-client/javascript#1341 That is because the library was using request-promise behind the scenes, which started showing up with unhandled rejection warnings and errors about req not being defined in some cases. That library update required us to change almost every k8s call since the interface now uses objects instead of positional arguments, which makes things much nicer to read and use. We also had to introduce a workaround for using custom certificates together with proxies, something that was previously globally monkey-patched by global-agent. **request-promise**: We no longer depend on request-promise and fully removed the dependency. Fixes #3841 Fixes #4898 Co-authored-by: Steffen Neubauer <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
A lot of packages in the Node ecosystem have moved to being pure ECMAScript modules (ESM)
Our current build configurations and scripts do not support depending on ESM.
This is blocking us from updating a lot of our dependencies.
Blocked by
pkgvercel/pkg#1291The text was updated successfully, but these errors were encountered: