Automatically restart processes using more than a fixed amount of RAM #141

Ecco · 2013-11-01T21:24:28Z

It would be very valuable to get the option to kill and respawn processes that are using more memory than a configurable amount. This would help a lot in the case of a leaking app. Indeed, consider the following scenario:

No respawn: The leaking app ends up taking tons of memory, effectively bringing the server down
Auto respawn: It might indeed require a bit of CPU time, but the service can be kept up and running

I'm not saying that leaking apps are a good thing. But from a "host" point of view, killing and respawning seems like the best thing to do. This is what Heroku does, for example.

rodrigok · 2013-11-04T13:04:12Z

I need it too. Maybe restart based on amount of CPU usage too.

abelaska · 2013-11-18T09:26:37Z

+1

dandv · 2014-03-07T10:21:23Z

👍 Very useful feature for production while debugging the leak in dev.

tjwebb · 2014-03-27T15:31:17Z

+1

devsmart · 2014-07-05T08:33:11Z

+1 , since PM2 cluster_mode takes much memory. if it can be restart when memory reached 500MB like will be great. it takes around 1.5GB when we keep process 3,4 days :(

rlidwka · 2014-07-16T21:51:35Z

I think you could set a memory limit in node.js/v8 command-line arguments, so it'll crash when the limit is reached. And pm2 will restart it as usual. Would this work for you?

devsmart · 2014-07-17T05:11:13Z

Thanks @rlidwka you mean by V8_MAX_SEMISPACE_SIZE ? or any other way to set it on with PM2 ?
I usually use simple PM2 command "pm2 start app.js -i max" appreciate if you can explain more how I can set memory limit to my application.

rlidwka · 2014-07-17T09:50:50Z

I'm talking about node --max-old-space-size=XXX test.js, with which a process that eats too much will die with "FATAL ERROR: CALL_AND_RETRY_2 Allocation failed - process out of memory".

If you're using pm2, I suppose pm2 start --node-args="--max-old-space-size=XXX" test.js will give the same effect.

I'm not sure if it's the right way, but it seems worth exploring.

soyuka · 2014-07-17T12:26:53Z

That's nice and working.

Just tested with a small memory leak that tests the pidusage module. This test breaks when memory is up to 200 MB and with --max-old-space-size=100 it breaks at 100 MB:

FATAL ERROR: JS Allocation failed - process out of memory
[1]    5718 abort      node --max-old-space-size=100 test/stresstest.js

Of course when using pm2 the process will be restarted.

Thanks @rlidwka for this nice information!

Should it be added to pm2 anyway?

To restart on CPU or memory amount with pm2 we 'd have to check the process status based on an interval which would lead to more CPU usage, I don't think that it's a good idea. Suggestions are welcome.

devsmart · 2014-07-18T05:06:41Z

Thanks a lot @rlidwka !

Unitech · 2014-08-14T13:44:42Z

I've added the option --max-restart-memory: https://github.com/Unitech/pm2/tree/development#automatic-restart-process-based-on-memory

Thanks for that v8 trick @rlidwka !

yorkie · 2015-04-13T05:57:52Z

Hey @Unitech actually the max-restart-memory doesn't work well in my machine, the memory in my machine keeps under 100M, but my node instance will automatically restart every 30s.

soyuka · 2015-04-13T06:02:22Z

@yorkie please see #1159.
Could you show your pm2.log? This might be an issue in your app.

ablankenship10 · 2015-04-13T16:19:26Z

I have the same problem where after about 8-10 hours, the pm2 instance runs of out of memory and never recovers. When I log into the server and run pm2 list, its like pm2 was never running and it restarts the Daemonizer. Here's my log. This is running on a 512MB Debian digital ocean droplet. I run it with the command 'pm2 start -x index.js'

2015-04-10 21:02:42: Starting execution sequence in -fork mode- for app name:index id:0
2015-04-10 21:02:42: App name:index id:0 online
2015-04-11 06:50:00: ERROR CAUGHT BY DOMAIN:
Error: spawn ENOMEM
    at exports._errnoException (util.js:746:11)
    at ChildProcess.spawn (child_process.js:1155:11)
    at exports.spawn (child_process.js:988:9)
    at Object.exports.execFile (child_process.js:682:15)
    at exports.exec (child_process.js:642:18)
    at getUrl (/root/.nvm/versions/node/v0.12.0/lib/node_modules/pm2/node_modules/vizion/lib/git.js:15:5)
    at fn (/root/.nvm/versions/node/v0.12.0/lib/node_modules/pm2/node_modules/async/lib/async.js:641:34)
    at Immediate._onImmediate (/root/.nvm/versions/node/v0.12.0/lib/node_modules/pm2/node_modules/async/lib/async.js:557:34)
    at processImmediate [as _immediateCallback] (timers.js:358:17)
child_process.js:1155
    throw errnoException(err, 'spawn');
          ^
Error: spawn ENOMEM
    at exports._errnoException (util.js:746:11)
    at ChildProcess.spawn (child_process.js:1155:11)
    at exports.spawn (child_process.js:988:9)
    at Object.exports.execFile (child_process.js:682:15)
    at exports.exec (child_process.js:642:18)
    at getUrl (/root/.nvm/versions/node/v0.12.0/lib/node_modules/pm2/node_modules/vizion/lib/git.js:15:5)
    at fn (/root/.nvm/versions/node/v0.12.0/lib/node_modules/pm2/node_modules/async/lib/async.js:641:34)
    at Immediate._onImmediate (/root/.nvm/versions/node/v0.12.0/lib/node_modules/pm2/node_modules/async/lib/async.js:557:34)
    at processImmediate [as _immediateCallback] (timers.js:358:17)
2015-04-13 15:54:37: [PM2][WORKER] Started with refreshing interval: 30000
2015-04-13 15:54:37: [[[[ PM2/God daemon launched ]]]]
2015-04-13 15:54:37: BUS system [READY] on port /root/.pm2/pub.sock
2015-04-13 15:54:37: RPC interface [READY] on port /root/.pm2/rpc.sock
2015-04-13 15:58:24: Starting execution sequence in -fork mode- for app name:index id:0
2015-04-13 15:58:24: App name:index id:0 online

jshkurti · 2015-04-13T16:24:28Z

Maybe it is your app which takes large amounts of memory and thus PM2 doesn't have enough.

ablankenship10 · 2015-04-13T16:32:00Z

Sure but shouldn't pm2 be catching that error and restart itself? Is it because I'm using fork mode? I guess I don't have to since it is only 1 core.. Its using jsdom and jquery and reading web page, I make sure to close the session each time but I'm sure its possible the dom library has some leaks in it.

Idk if its a PM2 issue or a Node issue in general, but every app I've made I've noticed that memory use always constantly increases, its never dumped. Is there a fix for this that I've missed?

jshkurti · 2015-04-13T16:39:02Z

Well, the thing is, in order to update itself, PM2 calls spawn('pm2 update') but since there's not enough memory for spawn(), it doesn't work. That's why you see the error twice. The first time it gets triggered by a spawn() in the worker, the second time it gets triggered when pm2 tries to update itself.
I think 512MB of RAM is just not enough nowadays to run your apps + a process manager while both of them use Node.js (which means being run by a VM such as Google V8 Runtime therefore using a lot of memory) :/

yorkie · 2015-04-13T18:08:46Z

Thanks guys, my issue is addressed by restarted the pm2 process, because in my previous boot script I set max-memory-restart to 80m, but later i changed it to 150m, but I guess there is a cache problem, so the solution is to kill the pm2 and restart it by using the new setup.

By the way, could someone can point out how the max-memory-restart is working? what's the difference from the v8 flag: max-old-space-size?

jorge-d · 2015-04-14T08:54:49Z

@yorkie All the magic is happening here :)

yorkie · 2015-04-14T09:13:50Z

Okay thank you @jorge-d

jshkurti · 2015-04-14T13:54:51Z

It's just a worker which checks every 30sec the memory usage of your app and restarts it if it exceeds the limit

jshkurti · 2015-06-05T17:22:52Z

@ablankenship10 PM2 v0.12.16 is released. This bug shouldn't happen anymore. At least it won't crash PM2 if ENOMEM occurs.

https://github.com/Unitech/PM2/releases/tag/0.12.16

akamanocha · 2016-11-24T07:03:44Z

What is the default value of max-memory-restart?

We are seeing that in a nodejs app we have given the max-old-space to be 512M its not hitting that limit. Still pm2 is retsrating our app.

Does pm2 restart on cpu reaching 100% or anything too?

soyuka · 2016-11-24T07:30:49Z

Does pm2 restart on cpu reaching 100% or anything too?
Nope.

What is the default value of max-memory-restart?
There is none, it is the nodejs default max-old-space-size

Btw, in nodejs 7 I see:

  --max_old_space_size (max size of the old space (in Mbytes))
        type: int  default: 0

Unitech mentioned this issue Nov 12, 2013

Offer an option to automatically respawn apps every now and then #142

Closed

soyuka added enhancement and removed mid-priority labels Jun 13, 2014

soyuka mentioned this issue Jun 19, 2014

auto restart app by max request option #526

Closed

soyuka added mid-priority and removed low-priority labels Jul 17, 2014

soyuka closed this as completed Aug 13, 2014

soyuka added wontfix and removed enhancement labels Aug 13, 2014

Unitech removed the wontfix label Aug 27, 2015

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Automatically restart processes using more than a fixed amount of RAM #141

Automatically restart processes using more than a fixed amount of RAM #141

Ecco commented Nov 1, 2013

rodrigok commented Nov 4, 2013

abelaska commented Nov 18, 2013

dandv commented Mar 7, 2014

tjwebb commented Mar 27, 2014

devsmart commented Jul 5, 2014

rlidwka commented Jul 16, 2014

devsmart commented Jul 17, 2014

rlidwka commented Jul 17, 2014

soyuka commented Jul 17, 2014

devsmart commented Jul 18, 2014

Unitech commented Aug 14, 2014

yorkie commented Apr 13, 2015

soyuka commented Apr 13, 2015

ablankenship10 commented Apr 13, 2015

jshkurti commented Apr 13, 2015

ablankenship10 commented Apr 13, 2015

jshkurti commented Apr 13, 2015

yorkie commented Apr 13, 2015

jorge-d commented Apr 14, 2015

yorkie commented Apr 14, 2015

jshkurti commented Apr 14, 2015

jshkurti commented Jun 5, 2015

akamanocha commented Nov 24, 2016

soyuka commented Nov 24, 2016

Automatically restart processes using more than a fixed amount of RAM #141

Automatically restart processes using more than a fixed amount of RAM #141

Comments

Ecco commented Nov 1, 2013

rodrigok commented Nov 4, 2013

abelaska commented Nov 18, 2013

dandv commented Mar 7, 2014

tjwebb commented Mar 27, 2014

devsmart commented Jul 5, 2014

rlidwka commented Jul 16, 2014

devsmart commented Jul 17, 2014

rlidwka commented Jul 17, 2014

soyuka commented Jul 17, 2014

devsmart commented Jul 18, 2014

Unitech commented Aug 14, 2014

yorkie commented Apr 13, 2015

soyuka commented Apr 13, 2015

ablankenship10 commented Apr 13, 2015

jshkurti commented Apr 13, 2015

ablankenship10 commented Apr 13, 2015

jshkurti commented Apr 13, 2015

yorkie commented Apr 13, 2015

jorge-d commented Apr 14, 2015

yorkie commented Apr 14, 2015

jshkurti commented Apr 14, 2015

jshkurti commented Jun 5, 2015

akamanocha commented Nov 24, 2016

soyuka commented Nov 24, 2016