Memory usage grows exponentially of the esbuild process

[thomaschaaf]

When using serverless-offline and making changes to files the memory usage of the esbuild process seems to grow exponentially.

I start out with about 1 GB of ram usage and after 10 changes the esbuild process is using 8 GB of ram.

[olup]

I can confrim this. I am running into FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory continuously - and my colleagues on the project too.
It is probably linked to the alchemy between chokidar and esbuild.

[olup]

From what I can see this does the leak:

serverless-esbuild/src/index.ts

Line 160 in ad3e93e

this.bundle(true)

true option here is using incremental bundling. Maybe a lead.

[olup]

incremental build does keep things in memory to speed up subsequent bundling: https://esbuild.github.io/api/#incremental. Could it be a leak in esbuild itself? Should we call dispose to free up resources from time to time ?

Or we could leave the watching to esbuild itself : https://esbuild.github.io/api/#watch, and see if memory is better handled but I guess this was sub-optimal @floydspace ?

[thomaschaaf]

@olup I think it's just used incorrectly. We need to call rebuild() instead of initiating a new esbuild instance with build(). That way rebuilding would be even faster aswell.

[olup]

Yes true, otherwise the cache is a bit lost anyway. I'll flesh a pr tomorrow. Beyond this one subject, serverless offline, in watch mode, is terribly leaking so the rest of the process will end up eating the whole memory anyway. But we can improve what depends on us.

[thomaschaaf]

I rewrote it to use rebuild. Sadly this does not resolve the memory issue. #101 so I think it might actually be an esbuild issue.

But I am unable to verify it with:

async function example() {
  let result = await require('esbuild').build({
    entryPoints: ['src/index.ts'],
    bundle: true,
    outfile: 'out.js',
    incremental: true,
  });

  for (let i = 0; i < 150; i++) {
    console.log('a' + i);
    await result.rebuild();
  }

  console.log('done');
}

example();

[mattisdada]

I am reporting I've had similar issues, each build just linearly increases the amount of memory esbuild is using (about 2.5GB per file change, it pretty quickly gets up to 20GB), if I was to hazard a guess, I'm thinking it has something to do with node_modules and this quote from the esbuild docs:

Files are stored in memory and are not re-read from the file system if the file metadata hasn't changed since the last build. This optimization only applies to file system paths. It does not apply to virtual modules created by plugins.

I have a feeling something is causing all the metadata change from ESBuilds perspective and it might not garbage collect old metadata (I imagine it doesn't watch for that). But I haven't looked at any of the code for serverless-esbuild so I might be completely wrong...

incremental build does keep things in memory to speed up subsequent bundling: https://esbuild.github.io/api/#incremental. Could it be a leak in esbuild itself? Should we call dispose to free up resources from time to time ?

Or we could leave the watching to esbuild itself : https://esbuild.github.io/api/#watch, and see if memory is better handled but I guess this was sub-optimal @floydspace ?

I'd be curious what this responds back with when logging is set to max

I might experiment a little tonight with this

[anthony-zhou]

I've been experiencing the same problem of exponential growth in memory usage.

As far as I can tell, rebuild seems to have better memory usage results than calling build every time.

With the changes in my PR, memory usage seems to cap out around 1 GB (for my current codebase), whereas it used to increase exponentially until crashing.

Let me know if the PR #123 solves your problem @olup or @thomaschaaf.

[mattisdada]

That PR partially fixed the issue for me (it still uses a lot of memory, but not as much), but serverless-offline still consumes a lot of memory unless allowCache is enabled and eventually exhausts itself (and dies) - this may be an issue with our particular code base though. Not sure if others are experiencing this particular pain point.

The problem of enabling allowCache is that when ESBuild recompiles, the running code does not change to the newly compiled files, so to see changes it requires resetting the server or having a large memory leak.

I put in a very blunt workaround in the compiled serverless-esbuild files after the build that does seem to work wonderfully which is just:

Object.keys(require.cache).forEach(function(key) { delete require.cache[key] })

This patched version + using serverless-offline in allowCache mode, has allowed us to have best of both worlds. An "official" solution should only clear the cache of the compiled files, not everything like this...

Not sure if serverless-offline has an "official" way of bundlers to manage cache...

[olup]

Indeed, I put the link somewhere, but the memory leak in serverless offline is well known and affecting everyone, and the relation with the caching option is discussed around, but the accepted answer seems to be "that's how it is". So no official way to fix this.

A workaround in our codebase would be a very good idea, as the memory leak is a real pain, thanks a lot for sharing this ! Only we should probably put it behind some options and disabled by default, and document it.

[mattisdada]

Before a more official solution is in place, this is what we ended up using:

# serverless.yml
custom:
  serverless-offline:
    allowCache: true # Important: This will prevent serverless-offline from eating all of your memory
    # ...
  esbuild:
    plugins : ./esbuild/plugins.js
    # ....

// esbuild/plugins.js
let refreshServer = {
  name: "refresh-server",
  setup(build) {
    build.onEnd((result) => {
      if (build.initialOptions.incremental) {
        console.log(`refresh-server: Clearing Cache for ${build.initialOptions.entryPoints.join(", ")}...`);
        // Remove all items from the cache (this will force node to reload all of the built artifacts)
        Object.keys(require.cache).forEach(function (key) {
          const resolvedPath = require.resolve(key);
          if (resolvedPath.includes(build.initialOptions.outdir)) {
            delete require.cache[key];
          }
        });
      }
    });
  },
};
module.exports = [refreshServer];

By utilising the esbuild plugin system we just did the clearing there instead of in the serverless-esbuild side. We now have a watch that can run for a long time, that gets updated whenever we change code and our memory usage remains relatively static. Would strongly recommend for anyone running into any of these memory issues or having to restart serverless whenever code changes (because of attempts to resolve the memory issues).

Hope that helps someone else...

[olup]

Brilliant ! And it's much better than having inside the lib, indeed.
But it is such a central pb with serverless offline we should definitely link this in the doc!

[mattisdada]

Only small problem is the esbuild plugins don't get any serverless context, I'll continue playing around with this and see if I find any issues regarding that.

EDIT: Already ran into an issue with this if it's running during pack it'll cause some issue with the archiver lib (should hopefully be resolved by checking if incremental is enabled)

I made a few changes to the plugin to be much more selective instead of clearing out everything, it should only clear out entries in the esbuild output directory.

[shadrech]

@mattisdada your solution seems to work for use with require, is there a equivalent solution for when using ES Modules?

[Joshuabaker2]

In addition to ameliorating memory eating-issues, I was also having issues where hot reloading wasn't working for me, and the solution from @mattisdada also fixed that. Thank you!

[mrjavi91]

@mattisdada I'm trying to implement your solution on a TypeScript project with around 8 functions, but I'm getting a weird issue, after running serverless offline, make one code change and save, the 'refreshServer' plugin runs in an infinite loop incrementing the ram usage until exhaustion.

// serverless.ts
custom: {
  'serverless-offline': {
    apiKey: API_KEY,
    allowCache: true,
  },
  esbuild: {
    bundle: true,
    minify: false,
    sourcemap: true,
    exclude: ['aws-sdk', 'pg-native'],
    target: 'node14',
    define: { 'require.resolve': undefined },
    platform: 'node',
    concurrency: 10,
    plugins: './esbuild/plugins.js',
  },
},

Any idea why this could be happening?

[matthew-gladman-oua]

@mattisdada I'm trying to implement your solution on a TypeScript project with around 8 functions, but I'm getting a weird issue, after running serverless offline, make one code change and save, the 'refreshServer' plugin runs in an infinite loop incrementing the ram usage until exhaustion.

// serverless.ts
custom: {
  'serverless-offline': {
    apiKey: API_KEY,
    allowCache: true,
  },
  esbuild: {
    bundle: true,
    minify: false,
    sourcemap: true,
    exclude: ['aws-sdk', 'pg-native'],
    target: 'node14',
    define: { 'require.resolve': undefined },
    platform: 'node',
    concurrency: 10,
    plugins: './esbuild/plugins.js',
  },
},

Any idea why this could be happening?

Hmm, it's a pretty simple and dumb plugin, so I have to guess the events is being mangled somehow, none of the actions it's doing should be triggering a new esbuild build. Out of curiosity when it is removed (but nothing else has been changed) does the infinite loop still occur? I have to guess it's a file system related issue (esbuild is generating something that is then being picked up by esbuild, and repeats; or something similiar)

I'm also not sure why this was added, I can only expect it to break things 🤔 :

define: { 'require.resolve': undefined },, it will def break that plugin but it might break everything. I'd actually try removing that first unless you have a particular reason for needing it

[mrjavi91]

@mattisdada I'm trying to implement your solution on a TypeScript project with around 8 functions, but I'm getting a weird issue, after running serverless offline, make one code change and save, the 'refreshServer' plugin runs in an infinite loop incrementing the ram usage until exhaustion.

// serverless.ts
custom: {
  'serverless-offline': {
    apiKey: API_KEY,
    allowCache: true,
  },
  esbuild: {
    bundle: true,
    minify: false,
    sourcemap: true,
    exclude: ['aws-sdk', 'pg-native'],
    target: 'node14',
    define: { 'require.resolve': undefined },
    platform: 'node',
    concurrency: 10,
    plugins: './esbuild/plugins.js',
  },
},

Any idea why this could be happening?

Hmm, it's a pretty simple and dumb plugin, so I have to guess the events is being mangled somehow, none of the actions it's doing should be triggering a new esbuild build. Out of curiosity when it is removed (but nothing else has been changed) does the infinite loop still occur? I have to guess it's a file system related issue (esbuild is generating something that is then being picked up by esbuild, and repeats; or something similiar)

I'm also not sure why this was added, I can only expect it to break things 🤔 :

define: { 'require.resolve': undefined },, it will def break that plugin but it might break everything

When the plugin is removed, and after a single save the memory usage keeps incrementing, I have stopped the process when it reaches to 14 GB or so.

I removed the line you mentioned define: { 'require.resolve': undefined },, and ran serverless offline again, made a dummy code change and saved, the ram usage keeps incrementing but somehow stabilizes between the 5-9 GB, in the meanwhile, the plugin is in the infinite loop.

[matthew-gladman-oua]

@mattisdada I'm trying to implement your solution on a TypeScript project with around 8 functions, but I'm getting a weird issue, after running serverless offline, make one code change and save, the 'refreshServer' plugin runs in an infinite loop incrementing the ram usage until exhaustion.

// serverless.ts
custom: {
  'serverless-offline': {
    apiKey: API_KEY,
    allowCache: true,
  },
  esbuild: {
    bundle: true,
    minify: false,
    sourcemap: true,
    exclude: ['aws-sdk', 'pg-native'],
    target: 'node14',
    define: { 'require.resolve': undefined },
    platform: 'node',
    concurrency: 10,
    plugins: './esbuild/plugins.js',
  },
},

Any idea why this could be happening?

Hmm, it's a pretty simple and dumb plugin, so I have to guess the events is being mangled somehow, none of the actions it's doing should be triggering a new esbuild build. Out of curiosity when it is removed (but nothing else has been changed) does the infinite loop still occur? I have to guess it's a file system related issue (esbuild is generating something that is then being picked up by esbuild, and repeats; or something similiar)
I'm also not sure why this was added, I can only expect it to break things 🤔 :
define: { 'require.resolve': undefined },, it will def break that plugin but it might break everything

When the plugin is removed, and after a single save the memory usage keeps incrementing, I have stopped the process when it reaches to 14 GB or so.

I removed the line you mentioned define: { 'require.resolve': undefined },, and ran serverless offline again, made a dummy code change and saved, the ram usage keeps incrementing but somehow stabilizes between the 5-9 GB, in the meanwhile, the plugin is in the infinite loop.

I think you've got something else going wrong that's triggering the builds to re-trigger unrelated to the plugin:

Try just logging that esbuild build is being triggered:

// esbuild/plugins.js
let refreshServer = {
  name: "refresh-server",
  setup(build) {
    build.onEnd((result) => {
      if (build.initialOptions.incremental) {
        console.log(`refresh-server: Clearing Cache for ${build.initialOptions.entryPoints.join(", ")}...`);
      }
    });
  },
};
module.exports = [refreshServer];

Do you have anything that would be writing to your src directory? And what other plugins do you have installed?

[mrjavi91]

@mattisdada I'm trying to implement your solution on a TypeScript project with around 8 functions, but I'm getting a weird issue, after running serverless offline, make one code change and save, the 'refreshServer' plugin runs in an infinite loop incrementing the ram usage until exhaustion.

// serverless.ts
custom: {
  'serverless-offline': {
    apiKey: API_KEY,
    allowCache: true,
  },
  esbuild: {
    bundle: true,
    minify: false,
    sourcemap: true,
    exclude: ['aws-sdk', 'pg-native'],
    target: 'node14',
    define: { 'require.resolve': undefined },
    platform: 'node',
    concurrency: 10,
    plugins: './esbuild/plugins.js',
  },
},

Any idea why this could be happening?

Hmm, it's a pretty simple and dumb plugin, so I have to guess the events is being mangled somehow, none of the actions it's doing should be triggering a new esbuild build. Out of curiosity when it is removed (but nothing else has been changed) does the infinite loop still occur? I have to guess it's a file system related issue (esbuild is generating something that is then being picked up by esbuild, and repeats; or something similiar)
I'm also not sure why this was added, I can only expect it to break things 🤔 :
define: { 'require.resolve': undefined },, it will def break that plugin but it might break everything

When the plugin is removed, and after a single save the memory usage keeps incrementing, I have stopped the process when it reaches to 14 GB or so.
I removed the line you mentioned define: { 'require.resolve': undefined },, and ran serverless offline again, made a dummy code change and saved, the ram usage keeps incrementing but somehow stabilizes between the 5-9 GB, in the meanwhile, the plugin is in the infinite loop.

I think you've got something else going wrong that's triggering the builds to re-trigger unrelated to the plugin:

Try just logging that esbuild build is being triggered:

// esbuild/plugins.js
let refreshServer = {
  name: "refresh-server",
  setup(build) {
    build.onEnd((result) => {
      if (build.initialOptions.incremental) {
        console.log(`refresh-server: Clearing Cache for ${build.initialOptions.entryPoints.join(", ")}...`);
      }
    });
  },
};
module.exports = [refreshServer];

Do you have anything that would be writing to your src directory? And what other plugins do you have installed?

I don't think there is anything writing to the src dir. We have the following plugins: serverless-esbuild, serverless-offline, serverless-openapi-documentation-v2 and serverless-hooks.

This is a small sample of the logs (the plugin is just logging):

refresh-server: Clearing Cache for src/functions/internal/status/handler.ts...
refresh-server: Clearing Cache for src/functions/merchant/handler.ts...
refresh-server: Clearing Cache for src/functions/webhooks/handler.ts...
refresh-server: Clearing Cache for src/functions/hello/handler.ts...
refresh-server: Clearing Cache for src/functions/loanRequests/handler.ts...
refresh-server: Clearing Cache for src/functions/webhooks/handler.ts...
refresh-server: Clearing Cache for src/functions/webhooks/handler.ts...
refresh-server: Clearing Cache for src/functions/internal/status/handler.ts...
refresh-server: Clearing Cache for src/functions/hello/handler.ts...
refresh-server: Clearing Cache for src/functions/hello/handler.ts...

The ram usage never stops incrementing.

[mattisdada]

The revised plugin only does logging, and does nothing else, your problem is outside the plugin. I recommend copying the project elsewhere and just keep removing things until it works as expected. (plugins, functions, config, etc)

I suspect serverless-hooks might be the problem depending on what they are configured to do, but I cannot really say without looking at the complete project sorry. But you have something going on that is triggering esbuild to run continuously (almost certainly something writing to disk). I'd also make sure that the api doc generation is outside the directory that esbuild is looking at as well or ignore it for compilation.

Once you've discovered the source, modify the watch pattern and ignore, example:
https://github.com/floydspace/serverless-esbuild#serverless-offline

[mrjavi91]

The revised plugin only does logging, and does nothing else, your problem is outside the plugin. I recommend copying the project elsewhere and just keep removing things until it works as expected. (plugins, functions, config, etc)

I suspect serverless-hooks might be the problem depending on what they are configured to do, but I cannot really say without looking at the complete project sorry. But you have something going on that is triggering esbuild to run continuously (almost certainly something writing to disk). I'd also make sure that the api doc generation is outside the directory that esbuild is looking at as well or ignore it for compilation.

Once you've discovered the source, modify the watch pattern and ignore, example: https://github.com/floydspace/serverless-esbuild#serverless-offline

Thank very much for the guidance, I'll look into it and get back with the fix.

[svoitovych0218]

In my cause I had the same issue before I refactored packages import. When I have import like this:
import * as firebaseAdmin from 'firebase-admin' - memory increases exponentially on rebuild.
Once I rewrote import into following approach
import { initializeApp, getApp, getApps, cert } from 'firebase-admin/app'; - memory leaks fixed.

[SamPF53]

The revised plugin only does logging, and does nothing else, your problem is outside the plugin. I recommend copying the project elsewhere and just keep removing things until it works as expected. (plugins, functions, config, etc)
I suspect serverless-hooks might be the problem depending on what they are configured to do, but I cannot really say without looking at the complete project sorry. But you have something going on that is triggering esbuild to run continuously (almost certainly something writing to disk). I'd also make sure that the api doc generation is outside the directory that esbuild is looking at as well or ignore it for compilation.
Once you've discovered the source, modify the watch pattern and ignore, example: https://github.com/floydspace/serverless-esbuild#serverless-offline

Thank very much for the guidance, I'll look into it and get back with the fix.

@mrjavi91 Did you ever find a fix? I'm having the exact same issue

[mrjavi91]

@SamPF53 No, I haven't found a fix yet. I tried what @svoitovych0218 suggested, with no luck. Did you try that too? Please let me know if that works for you or not.

[SamPF53]

@mrjavi91 - I just upgraded to the latest esbuild (0.15.5) and latest serverless-esbuild (1.32.8) and am not getting the endless loop anymore! Hopefully that resolves it for you. I was having an issue with hot reloading as well, but realized the handler was never reloading. Note for anyone having issues with hot reloading to set the --reloadHandler flag. In package.json:

  "scripts": {
    "local": "set NODE_OPTIONS=\"--enable-source-maps\"&& serverless offline --stage local --reloadHandler",

[johnsonps08]

Anyone still experiencing this with Node20 or Node21? I'm using this pattern which seems to cause the memory to grow until it runs the executor out of resources. Removing the plugins does prevent the memory issue but I bump into other problems with the build regarding node internals and external dependencies.

[nsdhillon1]

We are also experiencing this issue on mac os with apple chips, not seeing on the intel based mac

[mschipperheyn]

I have this on Intel based Mac

[nsdhillon1]

Follow up to my post above, we found our issue to be a layer was not excluded in serverless.ts for esbuild. Once we added it to exclude list, the issue got resolved for us.

[coolcorexix]

Before a more official solution is in place, this is what we ended up using:

# serverless.yml
custom:
  serverless-offline:
    allowCache: true # Important: This will prevent serverless-offline from eating all of your memory
    # ...
  esbuild:
    plugins : ./esbuild/plugins.js
    # ....

// esbuild/plugins.js
let refreshServer = {
  name: "refresh-server",
  setup(build) {
    build.onEnd((result) => {
      if (build.initialOptions.incremental) {
        console.log(`refresh-server: Clearing Cache for ${build.initialOptions.entryPoints.join(", ")}...`);
        // Remove all items from the cache (this will force node to reload all of the built artifacts)
        Object.keys(require.cache).forEach(function (key) {
          const resolvedPath = require.resolve(key);
          if (resolvedPath.includes(build.initialOptions.outdir)) {
            delete require.cache[key];
          }
        });
      }
    });
  },
};
module.exports = [refreshServer];

By utilising the esbuild plugin system we just did the clearing there instead of in the serverless-esbuild side. We now have a watch that can run for a long time, that gets updated whenever we change code and our memory usage remains relatively static. Would strongly recommend for anyone running into any of these memory issues or having to restart serverless whenever code changes (because of attempts to resolve the memory issues).

Hope that helps someone else...

this SEEM to work for me, thank you so much <3