build: generate multiple formats (cjs, esm) #8736

manju-reddys · 2021-12-02T17:40:20Z

PR Checklist

Please check if your PR fulfills the following requirements:

The commit message follows our guidelines: https://github.com/nestjs/nest/blob/master/CONTRIBUTING.md
Tests for the changes have been added (for bug fixes / features)
Docs have been added / updated (for bug fixes / features)

PR Type

What kind of change does this PR introduce?

What is the current behavior?

The current grunt build process can only generate cjs module format but the lots of other projects moved on to use the mjs (aks ESM) format.

Issue Number: N/A

What is the new behavior?

This PR adds new build process (prod) to generate multiple formats using tusp.

npm run build:multi will compile and generate cjs and mjs format and also *.d.ts

Does this PR introduce a breaking change?

Yes
No

Other information

jmcdo29

Also, what was the reason for using tsup over the existing gulp process we have?

jmcdo29 · 2021-12-02T17:43:26Z

tsconfig.json

@@ -13,6 +13,7 @@
    "allowJs": true,
    "outDir": "dist",
    "lib": ["es7"],
+    "esModuleInterop": true,


What's the reason for setting this to true? Is it necessary?

It pretty well explained https://stackoverflow.com/a/56348146/3130446

Okay, so because this will allow us to start letting Nest be used via ESM we need to make the rest of the CJS imports ESM compatible, correct?

Correct, it just adds the compatibility

jmcdo29 · 2021-12-02T17:44:01Z

packages/platform-fastify/adapters/fastify-adapter.ts

@@ -99,6 +100,7 @@ export class FastifyAdapter<
    TRawResponse
  > = FastifyInstance<TServer, TRawRequest, TRawResponse>,
 > extends AbstractHttpAdapter<TServer, TRequest, TReply> {
+  // @ts-ignore


Why all the @ts-ignores? Is there a typing that isn't right?

TS was complaining the typing isn't right!

Well, we should probably fix it rather than ignore it, right?

Correct, but I just didn't want to break anything.

Correct, but I just didn't want to break anything.

ts-ignore = I know it may break things but I do accept it.

manju-reddys · 2021-12-02T18:28:32Z

Also, what was the reason for using tsup over the existing gulp process we have?

TSC can't generate the .mjs and the gulp compiles the TS using gulp-typescript which internally runs TSC.

manju-reddys · 2021-12-03T20:04:46Z

@jmcdo29 how do I build all the module's under the @nestjs scope?

manju-reddys · 2021-12-08T17:44:38Z

This PR is all good, I tested against my codebase (12+ apps) and everything seems to be working without any issue and not even typing issues I found.

jmcdo29 · 2021-12-08T18:35:32Z

That's great. I'm still waiting to find out what problem ESM solves with regards to Node and Typescript. Seems it's brought about way more headaches than solutions

JumpLink · 2021-12-09T00:02:45Z

@jmcdo29 more and more node.js packages are switching to ESM. It can be very problematic to use this packages in a CJS module / app. E.g. I use node-fetch in my nest.js project. Since V3 I can't it use anymore so I need to stay on v2.x (async import has not worked for me). Which is a shame because I like to keep my project up to date.

And there are also other packages I can't upgrade for this reason.

See this gist, many packages wich are ESM only now are linking to that.

See also this blog post: https://blog.sindresorhus.com/get-ready-for-esm-aa53530b3f77

jmcdo29 · 2021-12-09T00:04:45Z

@jmcdo29 more and more node.js packages are switching to ESM. It can be very problematic to use this packages in a CJS module. E.g. I use node-fetch in my nest.js project. Since V3 I can't it use anymore so I need to stay on v2.x. Which is a shame because I like to keep my project up to date.

And there are also other packages I can't upgrade for this reason.

See this gist, many packages wich are ESM only now are linking to that.

See also this blog post: blog.sindresorhus.com/get-ready-for-esm-aa53530b3f77

@JumpLink I've seen that more packages are using ESM, but my question still remains why? Is it just because it's now the accepted JS standard? I'm all for change and getting the ball rolling, but I want to know why everyone is switching to ESM, if there's something I'm missing other than "it's the new standard".

JumpLink · 2021-12-09T00:25:22Z

@jmcdo29 One reason is that you can also use such packages in the browser and I think it is also easier to use such packages in Deno. ESM is a cross-platform standard. CommonJS only for node.

There are also other JavaScript runtimes that cannot use CommonJS, for example GJS under GNOME.

The JavaScript world would be much easier and simpler if all would use the same module system.

Here are more reasons: https://twitter.com/sindresorhus/status/1349312503835054080?s=20

jmcdo29 · 2021-12-09T00:29:34Z

@JumpLink so when it comes down to it, it's really because "this is the new standard" more than anything else, yeah?

The JavaScript world would be much easier and simpler if all would use the same module system.

This I agree with. I just don't know why ESM was necessary. It seemed that CJS was doing just fine as it was. Like I said before, I'm all for change, just want to know the why

JumpLink · 2021-12-09T00:41:25Z

@jmcdo29 You could also use browser modules in node easier. And other runtimes can use this modules, too. Node.js is the only one with another standard. So there are many runtimes wich are using the same module system and can be compatible with the others, node could also be compatible and all sides would benefit from that.

ESM also has fewer disadvantages. It is problematic to use ESM modules in CJS, but if your application uses ESM you can still integrate CJS modules in node without problems, the other way round not so well.

I want to use ESM modules in my Nest.js application but currently I can't.

By the way: I heard tree shaking should also be more manageable in ESM.

Summary:

Top level await support
Unified syntax
Top-level await
Import both default export and named exports in the same statement
Re-export syntax
Potentially faster import step
Official loader hooks (still draft)
Cross Runtime compatible
Same syntax in TypeScript
Potentially better tree shaking
Less complexity, the tools no longer have to support multiple module systems
No more converting / polifills between the module systems necessary (webpack, Typescript, Babbel, browserify, ...)

jmcdo29 · 2021-12-09T00:48:36Z

You could also use browser modules in node easier.

Generally true until you start having the DOM types show up and Typescript starts throwing errors left and right because of environment differences.

ESM also has fewer disadvantages.

Not sure what disadvantages are being considered here other than use of ESM inside CJS or CJS inside ESM. To my knowledge mocking and many test frameworks still have some troubles with ESM. Not a huge deal in Nest projects due to being able to provide our own mocks and the dependency injection, but just something to call out.

By the way: I heard tree shaking should also be more manageable in ESM.

I'd be curious as to why this is. It's not like ESM uses a new AST, whatever is doable, in terms of tree shaking, in ESM I think should be doable in CJS as well.

I'll keep looking around and finding out more of why this change came in the first place and what kind of impact it will have overall.

manju-reddys · 2021-12-09T05:26:49Z

@JumpLink summarizes the benefits and one addition to it is the security which is the main concern in the crypto world as many node_modules are easy to introduce threats into CJS example.

const fs = require('fs'); 
// any method of the fs can be changed here after

in ESM

import {readFile} from 'fs';
// Not possible

jmcdo29 · 2021-12-09T05:37:34Z

@JumpLink summarizes the benefits and one addition to it is the security which is the main concern in the crypto world as many node_modules are easy to introduce threats into CJS example.
const fs = require('fs'); 
// any method of the fs can be changed here after
in ESM
import {readFile} from 'fs';
// Not possible

Yep, this is why testing frameworks have a problem with figuring out how to do mocks for ESM, because you can't just change the original implementation like you can with CJS

rat-matheson · 2021-12-18T16:11:47Z

Angular 13 removed CommonJS outputs so libraries created with it only use ES modules. I have some custom angular libraries that I share with the backend and frontend and due to upgrading to angular 13, I can no longer build my nest backend as I get Error [ERR_REQUIRE_ESM]: require() of ES Module.

Will this commit correct that issue?

wodka · 2022-02-16T09:30:56Z

@jmcdo29 is there any way to help move this pr forward? There are more and more packages switching to ESM only and it would be great to be able to upgrade them...

like https://github.com/sindresorhus/serialize-error

as for why? https://gist.github.com/sindresorhus/a39789f98801d908bbc7ff3ecc99d99c

kamilmysliwiec · 2022-02-16T09:44:18Z

I personally share the concerns of everyone who collaborated (on looking for a viable solution and seamless path forward) in this issue fastify/fastify#2847

rat-matheson · 2022-04-04T16:25:59Z

Any idea why this PR is blocked? Is there anything that can help move it along?

victorm-sp · 2022-05-04T00:01:51Z

Is possible to check this PR @jmcdo29 ? I'm not able to update my version of Angular to 13 and Angular 14 is almost here D:

jmcdo29 · 2022-05-04T01:28:13Z

As Kamil has already mentioned, there's concerns around how ESM integrate with Node packages, as you can see in the linked Fastify discussion. I understand that Angular is using ESM, and that browser side JS ESM is the standard, and now Node can run in an ESM mode that supports ESM modules.

If you need to use an ESM module inside a Nest application, right now you can await import('esm-module') and make use of it that way (wrap it in a top level function if you need to). I'm still yet to see real reasons to change over to ESM other than it's what the browser uses and it's "the future" of JavaScript packages. Until I know why we'd want to change over to it, even though I see that there is a request to do so, I can't really feel comfortable with this.

manju-reddys · 2022-05-04T03:29:15Z

@jmcdo29 Why ESM is highlighted in the comments and i'm not understand what else is that you what to know.

@kamilmysliwiec @jmcdo29 The request is not change everything to ESM but publish both ESM + CJS which is very much possible with modern tooling ( as I already put together in this PR).

kbzowski · 2022-09-04T11:57:25Z

Lazy solution is just wrap each ESM lib into service which use https://github.com/cspotcode/tsimportlib or different solution mentioned in this thread.
In case of https://github.com/sindresorhus/execa v6 that service can look like:

import {Injectable} from "@nestjs/common";
import {dynamicImport} from "tsimportlib";

export type ExecaModule = typeof import('execa')
export type Execa = typeof import('execa').execa;
export type ExecaCommand = typeof import('execa').execaCommand

@Injectable()
export class ExecaService {
    private execaModule : ExecaModule;

    public execa: Execa;
    public execaCommand: ExecaCommand;
    
    constructor() {
        this.loadExeca().then(() => {}).catch(err => {
            // Suitable error handling
        });
    }

    async loadExeca() {
        this.execaModule = (await dynamicImport(
            'execa',
            module,
        )) as ExecaModule;
        this.execa = this.execaModule.execa;
        this.execaCommand = this.execaModule.execaCommand;
    }
}

It is known that such a solution does not solve all use cases, but for me it works without a problem. Of course, it would be good to abandon them at some point and have support for ESM in NestJS.

muratbeser · 2022-09-12T06:12:40Z

Just read this comment this is great news to hear.
#10267 (comment)

ephys · 2023-01-11T07:50:52Z

I recommend against transpiling to both ESM and CJS. For a project like nest, that will eventually lead to issues described in Node's documentation about dual ESM/CJS packages.

I've implemented the ES module wrapper approach in a few libraries and it has worked without issues so far.

It's a very simple approach but requires some maintenance, here's an example I set up in a different library:

You need an mjs wrapper that defines the named ESM exports
An exports config in package.json to make ESM load the wrapper (⚠️ Adding an exports config is typically a breaking change)
And a test that ensures the mjs wrapper stays up-to-date

4-1-8 · 2023-01-11T07:52:13Z

We are also in an issue on our project with the lack of support for ESM. Wouldn't it be possible to directly ask the Nest community somehow?

nitedani · 2023-02-02T06:58:14Z

If you really want/need to, you can already use Nest with ESM and Vite, today. https://github.com/cyco130/vavite/tree/main/examples/nestjs

ephys · 2023-02-02T07:44:45Z

Well yes, just like we can compile our codebase to CJS, but the goal is to use native ESM (good workaround nonetheless)

To be honest we're already using nest with native ESM today. Node somehow manages to generates the named imports correctly (I have no idea how, nest's exports are not statically analyzable) but it breaks if you use --disallow-code-generation-from-strings

Alivers · 2023-02-10T07:18:17Z

Any updates on this?

ohmree · 2023-02-15T15:48:07Z

That's great. I'm still waiting to find out what problem ESM solves with regards to Node and Typescript. Seems it's brought about way more headaches than solutions

Well for one it would simplify vite-based workflows like the one shown in this example.

Currently we have resorted to using hacks like vite-plugin-commonjs (forgot why but I think it solved something) and telling vite to generate cjs using a deprecated option (for compatibility with the @nestjs/swagger typescript transformer that currently only generates cjs), and I just ran into another issue when trying to shoehorn a repl.ts to this setup using vite-node instead of the nest cli where it was trying to load dependencies as esm but everything else in our setup was assuming cjs, had to set ssr.noExternal = true but only when running the repl script.

Lonolf · 2023-02-18T17:54:11Z

@ohmree I have the same problem: a monorepo with some nestjs backends, a vitejs frontend and a library (esnext) of common things (types, constants, validators, ecc).
Vitejs can access the library without any problem, and refresh when updating it, but if I try to change it to commonjs it breaks a lot of things.

I resolved at the moment by having two libraries, the second (constantsjs) is simply compilation of the first one (constants) with commonjs, using a tsconfig like this:

{
  "compilerOptions": {
     ...
    "module": "commonjs",
    "outDir": "lib",
    "rootDir": "../constants/src",
  },
  "include": ["../constants/src"]
 }

Obliviously using directly the first library for everything would be better.

Shinigami92 · 2023-02-20T08:27:07Z

Hi @jmcdo29, I would like to explain a bit better CJS vs ESM

CJS is a convention brought into the ecosystem by NodeJS
ESM however is a specification that is introduces by ECMAScript and will be adopted by the whole ecosystem

So the difference is that everyone can theoretically write a function called require and you could accidentally import that and use it
import ... from ... is safe to be used because it uses statical keywords that cant be overridden

2-3 years ago packages slowly started to switch to ESM
in this point of time it starts to get more faster and also starts to get more radically
I would say NestJS needs to support ESM this year, or projects will start to migrate away from NestJS

I cannot upgrade to chalk v5 and graphql-upload v16 because of that and it is frustrating that a Framework is holding me back

kamilmysliwiec · 2023-02-20T08:47:57Z

in this point of time it starts to get more faster and also starts to get more radically

More radically? So which popular packages, except those 2 you mentioned, have already migrated to ESM?

Nest can't migrate to ESM before the entire ecosystem of tools it uses & integrates with is ready for this change:

Fastify fastify/fastify#2847 (comment)

Jest framework that Nest template projects use still doesn't have stable support for ESM https://jestjs.io/docs/ecmascript-modules

And a few others

Shinigami92 · 2023-02-20T09:35:12Z

in this point of time it starts to get more faster and also starts to get more radically

More radically? So which popular packages, except those 2 you mentioned, have already migrated to ESM?

Nest can't migrate to ESM before the entire ecosystem of tools it uses & integrates with is ready for this change:

The year is just in its second month, so I just wanted to say NestJS should not "reject because of to much maintenance burden" but start to try to actively support it

Fastify fastify/fastify#2847 (comment)

So looks like Fastify already started to support it? Great!

Jest framework that Nest template projects use still doesn't have stable support for ESM jestjs.io/docs/ecmascript-modules

Why is NestJS even bound to Jest? Why cant we use another more modern provider like vitest?

And a few others

You mentioned two example, I mentioned two examples 😉
Jokes away... At FakerJS we think e.g. about to drop CJS support because this would cut the package size in half! From ~10mb to ~5mb (because of locale data)

Apollo v4 (for GraphQL) looks like they are also starting to jump to ESM
In the GraphQL ecosystem like graphql-upload https://github.com/jaydenseric/graphql-upload/issues?q=is%3Aissue+cjs+is%3Aclosed and https://github.com/graphql/graphql-js/releases v17 they are starting to partially only support ESM

It might be that I'm very dependent on the GraphQL ecosystem, so that might influence my point of view to ESM support

But I also see other packages around me to also start to move to ESM or even drop CJS support

kamilmysliwiec · 2023-02-20T09:39:55Z

Apollo v4 (for GraphQL) looks like they are also starting to jump to ESM

and https://github.com/graphql/graphql-js/releases v17 they are starting to partially only support ESM

Which is great. So if they migrate, we'll migrate too. That's exactly what I meant

ephys · 2023-02-20T09:44:30Z

While I think it's true that ESM is gaining momentum (lots of apps are being migrated to ESM because many libraries have become ESM-only and can only be imported using ESM), I don't think it would be wise for nest to migrate to ESM yet.

But the good news is you don't have to migrate to ESM. You can support both ESM & CJS somewhat easily as I indicated above in #8736 (comment)

That being said if graphql-js is going to become ESM-only anyway, then it's not impossible that nest is going to have to become ESM-only too. That would have a massive impact on the adoption of ESM in this community

If that happens I think we will eventually not need to support CJS in Sequelize either 😄

Nosfistis · 2023-03-16T18:57:54Z

Can we have a working guide then on how to import ESM dependencies?

I am trying to import https://github.com/keycloak/keycloak/tree/main/js/libs/keycloak-admin-client which has switched to ESM over a year ago, and makes me work with the latest CJS version.

hkjeffchan · 2023-03-17T08:31:02Z

Can we have a working guide then on how to import ESM dependencies?

I am trying to import https://github.com/keycloak/keycloak/tree/main/js/libs/keycloak-admin-client which has switched to ESM over a year ago, and makes me work with the latest CJS version.

You can use any ESM package with "const xxx = await import" syntax.

Ghirigoro · 2023-03-17T12:14:13Z

You can use any ESM package with "const xxx = await import" syntax.

That works easily if whatever is consuming that import is already async but if it's not it means making any function that touches that import async (and whatever touches THAT function...).

In my experience dynamic imports are usually a pain and if possible using something like dynamic imports might be easier: https://docs.nestjs.com/fundamentals/dynamic-modules

Nosfistis · 2023-03-27T09:16:48Z

It also does not seem to work with e2e tests, as the import promise never resolves or rejects.

h16rkim · 2023-04-17T07:01:34Z

We need same feature. Any updates ..?

kamilmysliwiec · 2024-03-21T07:46:44Z

Node.js has an experimental support for require(esm) now https://joyeecheung.github.io/blog/2024/03/18/require-esm-in-node-js/

This means everyone should be able to use esm libraries in their Nest projects.

manju-reddys changed the title ~~build: generate multiple formats (cjs, esm)~~ WIP: build: generate multiple formats (cjs, esm) Dec 2, 2021

jmcdo29 reviewed Dec 2, 2021

View reviewed changes

manju-reddys requested a review from jmcdo29 December 3, 2021 20:07

manju-reddys changed the title ~~WIP: build: generate multiple formats (cjs, esm)~~ build: generate multiple formats (cjs, esm) Dec 8, 2021

jmcdo29 mentioned this pull request Dec 11, 2021

ES Module specs preventing models from bootstrapping #8775

Closed

15 tasks

rat-matheson mentioned this pull request Dec 15, 2021

ES Module broken for back end nrwl/nx nestJS app for mono-repo angular/angular#44433

Closed

davidcalhoun mentioned this pull request Dec 27, 2021

Nest.js: Unable to import davidcalhoun/tle.js#27

Closed

manju-reddys added 2 commits February 3, 2022 17:10

build: generate multiple formats (cjs, esm)

a67c212

build: added build task

311b722

manju-reddys force-pushed the multi_format branch from 6e37f8c to 311b722 Compare February 4, 2022 01:10

LeParadoxHD mentioned this pull request Feb 14, 2022

@nestjs/ng-universal not working with Angular 13 - Error [ERR_REQUIRE_ESM]: require() of ES Module nestjs/ng-universal#833

Open

jmcdo29 mentioned this pull request Sep 5, 2022

Can't use d3 lib with NestJS #10239

Closed

15 tasks

ganigeorgiev mentioned this pull request Oct 26, 2022

This does not work with nest js pocketbase/js-sdk#52

Closed

Tony133 mentioned this pull request Dec 14, 2022

Vite as builder for NestJS #10721

Closed

1 task

ganigeorgiev mentioned this pull request Dec 22, 2022

Error when using with NestJS pocketbase/js-sdk#113

Closed

ephys mentioned this pull request Jan 10, 2023

--disallow-code-generation-from-strings breaks nest's named ESM exports #10846

Closed

15 tasks

micalevisk mentioned this pull request Feb 2, 2023

nest cannot handle src/main.mts #11021

Closed

15 tasks

valerii15298 mentioned this pull request Mar 29, 2023

Error: Schema must contain uniquely named types but contains multiple types named "JSON" MichalLytek/typegraphql-prisma#371

Closed

manuel-rw mentioned this pull request Apr 11, 2023

Error [ERR_REQUIRE_ESM]: require() of ES Module - Jellyfin SDK 0.8.1 ESM manuel-rw/jellyfin-discord-music-bot#167

Open

nestjs locked and limited conversation to collaborators Apr 17, 2023

kamilmysliwiec closed this Nov 6, 2024

build: generate multiple formats (cjs, esm) #8736

build: generate multiple formats (cjs, esm) #8736

Conversation

manju-reddys commented Dec 2, 2021 • edited Loading

PR Checklist

PR Type

What is the current behavior?

What is the new behavior?

Does this PR introduce a breaking change?

Other information

jmcdo29 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

manju-reddys Dec 2, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

manju-reddys commented Dec 2, 2021

manju-reddys commented Dec 3, 2021

manju-reddys commented Dec 8, 2021

jmcdo29 commented Dec 8, 2021

JumpLink commented Dec 9, 2021 • edited Loading

jmcdo29 commented Dec 9, 2021

JumpLink commented Dec 9, 2021 • edited Loading

jmcdo29 commented Dec 9, 2021

JumpLink commented Dec 9, 2021 • edited Loading

jmcdo29 commented Dec 9, 2021

manju-reddys commented Dec 9, 2021

jmcdo29 commented Dec 9, 2021

rat-matheson commented Dec 18, 2021

wodka commented Feb 16, 2022 • edited Loading

kamilmysliwiec commented Feb 16, 2022 • edited Loading

rat-matheson commented Apr 4, 2022

victorm-sp commented May 4, 2022

jmcdo29 commented May 4, 2022

manju-reddys commented May 4, 2022

kbzowski commented Sep 4, 2022

muratbeser commented Sep 12, 2022

ephys commented Jan 11, 2023 • edited Loading

4-1-8 commented Jan 11, 2023

nitedani commented Feb 2, 2023

ephys commented Feb 2, 2023 • edited Loading

Alivers commented Feb 10, 2023

ohmree commented Feb 15, 2023

Lonolf commented Feb 18, 2023

Shinigami92 commented Feb 20, 2023

kamilmysliwiec commented Feb 20, 2023

Shinigami92 commented Feb 20, 2023

kamilmysliwiec commented Feb 20, 2023

ephys commented Feb 20, 2023 • edited Loading

Nosfistis commented Mar 16, 2023

hkjeffchan commented Mar 17, 2023

Ghirigoro commented Mar 17, 2023

Nosfistis commented Mar 27, 2023

h16rkim commented Apr 17, 2023

kamilmysliwiec commented Mar 21, 2024

manju-reddys commented Dec 2, 2021 •

edited

Loading

manju-reddys Dec 2, 2021 •

edited

Loading

JumpLink commented Dec 9, 2021 •

edited

Loading

JumpLink commented Dec 9, 2021 •

edited

Loading

JumpLink commented Dec 9, 2021 •

edited

Loading

wodka commented Feb 16, 2022 •

edited

Loading

kamilmysliwiec commented Feb 16, 2022 •

edited

Loading

ephys commented Jan 11, 2023 •

edited

Loading

ephys commented Feb 2, 2023 •

edited

Loading

ephys commented Feb 20, 2023 •

edited

Loading