Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

PM2 daemon disappeared after certain time #1424

Closed
iKoala opened this issue Jul 9, 2015 · 22 comments
Closed

PM2 daemon disappeared after certain time #1424

iKoala opened this issue Jul 9, 2015 · 22 comments
Labels
S: In progress T: Bug

Comments

@iKoala
Copy link

iKoala commented Jul 9, 2015

I am using PM2 to run three node process but PM2 daemon was killed from time to time without any error or logs.
Are there log files which I can use it to debug for this situation?
@Tjatse
Copy link
Collaborator

Tjatse commented Jul 9, 2015

You can run tail -n 30 ~/.pm2/pm2.log and paste it here.

@iKoala
Copy link
Author

iKoala commented Jul 9, 2015 

tail -n 30 pm2.log
2015-07-07 14:51:20: BUS system [READY] on port /home/ec2-user/.pm2/pub.sock
2015-07-07 14:51:20: RPC interface [READY] on port /home/ec2-user/.pm2/rpc.sock
2015-07-07 14:51:29: Starting execution sequence in -fork mode- for app name:app-www-prod id:0
2015-07-07 14:51:29: App name:app-www-prod id:0 online
2015-07-07 14:51:40: Starting execution sequence in -fork mode- for app name:app-cms-prod id:1
2015-07-07 14:51:40: App name:app-cms-prod id:1 online
2015-07-07 14:51:48: Starting execution sequence in -fork mode- for app name:app-merchant-prod id:2
2015-07-07 14:51:48: App name:app-merchant-prod id:2 online
2015-07-08 01:03:46: Stopping app:app-cms-prod id:1
2015-07-08 01:03:46: App name:app-cms-prod id:1 exited with code SIGINT
2015-07-08 01:03:46: Process with pid 21272 killed
2015-07-08 01:03:46: Starting execution sequence in -fork mode- for app name:app-cms-prod id:1
2015-07-08 01:03:46: App name:app-cms-prod id:1 online
2015-07-08 18:11:38: [PM2][WORKER] Started with refreshing interval: 30000
2015-07-08 18:11:38: [[[[ PM2/God daemon launched ]]]]
2015-07-08 18:11:38: BUS system [READY] on port /home/ec2-user/.pm2/pub.sock
2015-07-08 18:11:38: RPC interface [READY] on port /home/ec2-user/.pm2/rpc.sock
2015-07-08 18:11:57: Starting execution sequence in -fork mode- for app name:app-www-prod id:0
2015-07-08 18:11:57: App name:app-www-prod id:0 online
2015-07-08 18:12:28: Starting execution sequence in -fork mode- for app name:app-cms-prod id:1
2015-07-08 18:12:28: App name:app-cms-prod id:1 online
2015-07-08 18:12:33: Starting execution sequence in -fork mode- for app name:app-merchant-prod id:2
2015-07-08 18:12:33: App name:app-merchant-prod id:2 online
2015-07-09 12:01:56: [PM2][WORKER] Started with refreshing interval: 30000
2015-07-09 12:01:56: [[[[ PM2/God daemon launched ]]]]
2015-07-09 12:01:56: BUS system [READY] on port /home/ec2-user/.pm2/pub.sock
2015-07-09 12:01:56: RPC interface [READY] on port /home/ec2-user/.pm2/rpc.sock
2015-07-09 12:02:13: PM2 is being killed via kill method
2015-07-09 12:02:13: RPC socket closed
2015-07-09 12:02:13: PUB socket closed

node -v
v0.12.6

Thanks in advance.

@Mickael-van-der-Beek
Copy link

I'm getting the same issue recently. Here's a dump of my PM2 logs:

2015-07-07 21:56:13: Starting execution sequence in -cluster mode- for app name:server id:0
2015-07-07 21:56:13: App name:server id:0 online
2015-07-07 22:01:48: App name:server id:0 disconnected
2015-07-07 22:01:48: App name:server id:0 exited with code 0
2015-07-07 22:01:48: Starting execution sequence in -cluster mode- for app name:server id:0
2015-07-07 22:01:48: App name:server id:0 online
2015-07-07 22:05:54: App name:server id:0 disconnected
2015-07-07 22:05:54: App name:server id:0 exited with code 0
2015-07-07 22:05:54: Starting execution sequence in -cluster mode- for app name:server id:0
2015-07-07 22:05:54: App name:server id:0 online
2015-07-07 22:06:28: App name:server id:0 disconnected
2015-07-07 22:06:28: App name:server id:0 exited with code 0
2015-07-07 22:06:28: Starting execution sequence in -cluster mode- for app name:server id:0
2015-07-07 22:06:28: App name:server id:0 online
2015-07-07 22:07:07: App name:server id:0 disconnected
2015-07-07 22:07:07: App name:server id:0 exited with code 0
2015-07-07 22:07:07: Starting execution sequence in -cluster mode- for app name:server id:0
2015-07-07 22:07:07: App name:server id:0 online
2015-07-07 22:07:48: App name:server id:0 disconnected
2015-07-07 22:07:48: App name:server id:0 exited with code 0
2015-07-07 22:07:48: Starting execution sequence in -cluster mode- for app name:server id:0
2015-07-07 22:07:48: App name:server id:0 online
2015-07-07 22:10:42: App name:server id:0 disconnected
2015-07-07 22:10:42: App name:server id:0 exited with code 0
2015-07-07 22:10:42: Starting execution sequence in -cluster mode- for app name:server id:0
2015-07-07 22:10:42: App name:server id:0 online
2015-07-07 22:13:02: App name:server id:0 disconnected
2015-07-07 22:13:02: App name:server id:0 exited with code 0
2015-07-07 22:13:02: Starting execution sequence in -cluster mode- for app name:server id:0
2015-07-07 22:13:02: App name:server id:0 online
2015-07-07 22:15:25: App name:server id:0 disconnected
2015-07-07 22:15:25: App name:server id:0 exited with code 0
2015-07-07 22:15:25: Starting execution sequence in -cluster mode- for app name:server id:0
2015-07-07 22:15:25: App name:server id:0 online
2015-07-07 22:18:21: App name:server id:0 disconnected
2015-07-07 22:18:21: App name:server id:0 exited with code 0
2015-07-07 22:18:21: Starting execution sequence in -cluster mode- for app name:server id:0
2015-07-07 22:18:21: App name:server id:0 online
2015-07-07 22:19:09: App name:server id:0 disconnected
2015-07-07 22:19:09: App name:server id:0 exited with code 0
2015-07-07 22:19:09: Starting execution sequence in -cluster mode- for app name:server id:0
2015-07-07 22:19:09: App name:server id:0 online
2015-07-07 22:23:16: App name:server id:0 disconnected
2015-07-07 22:23:16: App name:server id:0 exited with code 0
2015-07-07 22:23:16: Starting execution sequence in -cluster mode- for app name:server id:0
2015-07-07 22:23:16: App name:server id:0 online
2015-07-08 10:40:44: App name:server id:0 disconnected
2015-07-08 10:40:44: App name:server id:0 exited with code 0
2015-07-08 10:40:44: Starting execution sequence in -cluster mode- for app name:server id:0
2015-07-08 10:40:44: App name:server id:0 online
2015-07-08 10:42:12: App name:server id:0 disconnected
2015-07-08 10:42:12: App name:server id:0 exited with code 0
2015-07-08 10:42:12: Starting execution sequence in -cluster mode- for app name:server id:0
2015-07-08 10:42:12: App name:server id:0 online
2015-07-09 11:38:38: [PM2][WORKER] Started with refreshing interval: 30000
2015-07-09 11:38:38: [[[[ PM2/God daemon launched ]]]]
2015-07-09 11:38:38: BUS system [READY] on port /home/development/.pm2/pub.sock
2015-07-09 11:38:38: RPC interface [READY] on port /home/development/.pm2/rpc.sock

The last four lines are me running pm2 list which rebooted PM2. The logs themselves don't seem to show much information ...

Here's some additional information on my environment:

Node.js v0.12.2
NPM 2.7.4
PM2 0.14.3
Ubuntu 15.04
Linux Kernel 3.14.32-xxxx-grs-ipv6-64

@airs0urce
Copy link
Contributor

Same thing when using pm2 v0.14.3 (node.js v0.12.6).
I run app in cluster mode and use "pm2 list" to check running processes and checking if everything works sometimes. And now my app running for about 10 hours and then I run "pm2 list" and see empty list.

Downgraded to pm2 v0.12.6, will see if this will solve the issue.

@airs0urce
Copy link
Contributor

Seems like everything is ok after downgrade

@iKoala
Copy link
Author

iKoala commented Jul 11, 2015

I switched from forever to pm2 because it seems the pm2 features are much more promising.
However, now I am quite disappointed with this issue because I think pm2 is not production ready.
I think I will just switch back to forever because it could run for 363 days without interruption before I switched to pm2.

@membrive
Copy link

Is there any error in /var/log/kern.log or /var/log/syslog?

Your problem could be related to #1380?

@iKoala
Copy link
Author

iKoala commented Jul 11, 2015

I am not sure if it is the same issue because I just can't find similar error log on my server.
It may because I switched back to forever yesterday.
I will try it again and see if I get the same error logs.

@soyuka soyuka mentioned this issue Jul 13, 2015
Closed
@oscar608
Copy link

pm2 v0.14.3 have resolved this problem?

@iKoala
Copy link
Author

iKoala commented Jul 14, 2015

The problem still exists and I still can't find error logs related to this issue.

@airs0urce
Copy link
Contributor

If you have this issue - Install pm2 v0.12.6 - it works, tested last 4 days.
0.14.3 is buggy

@oscar608
Copy link

OK,have a try to down version to v0.12.6

@sbmaxx
Copy link

sbmaxx commented Jul 18, 2015

Same to me:

PM2 Starting execution sequence in -cluster mode- for app name:report id:0
PM2 App name:report id:0 online
PM2 App name:report id:0 disconnected
PM2 App name:report id:0 exited with code 1
PM2 Starting execution sequence in -cluster mode- for app name:report id:0
PM2 App name:report id:0 online
PM2 App name:report id:0 disconnected
PM2 App name:report id:0 exited with code 1
PM2 Starting execution sequence in -cluster mode- for app name:report id:0
PM2 App name:report id:0 online
PM2 App name:report id:0 disconnected
PM2 App name:report id:0 exited with code 1

pm2 --version
0.14.3
iojs --version
v2.3.4

Looks like it's ipv6 only problem. In other setups with the same pm2 & iojs versions but with ipv4&ipv6 address — its works fine. And it fails at ipv6 only in cluster mode.

This was referenced Jul 20, 2015
Closed
Closed
Closed
@airs0urce
Copy link
Contributor

I used ip4 setup only

@soyuka soyuka mentioned this issue Jul 21, 2015
Closed
@crispkiwi
Copy link

I've been running into this as well. Daemon disappears with no hints in the log files. Longest running instance I've had is a few days. Running Node v0.12.6, PM2 v0.14.3, on Amazon Linux.

Will swap out instances for ones running PM2 v0.12.6 until some stability is returned.

@Unitech
Copy link
Owner

Unitech commented Jul 22, 2015

Hello,

@airs0urce @crispkiwi @iKoala Have you linked PM2 to Keymetrics?
(@jshkurti Watchdog/auto reboot)

@iKoala
Copy link
Author

iKoala commented Jul 22, 2015

@Unitech Hello, I didn't link PM2 to Keymetrics (although that's my purpose switching to PM2).

I still have no idea when this instance happens. I am using pm2 v0.14.0 on my development server and it worked fine when there are loading on the server during working hours. However, PM2 had a great chance to disappear after the weekend or holiday, i.e. no loading.

@Unitech Unitech mentioned this issue Jul 22, 2015
Closed
14 tasks
@airs0urce
Copy link
Contributor

@Unitech I didn't use Keymetrics too

@Mickael-van-der-Beek
Copy link

@Unitech I don't use Keymetrics either.

My applications use quite a lot of memory and are usually rebooted if they go over 1.4Gb of used RAM memory by Node.js. Could it be that something changed in this case like the process's exit code for example and that PM2 doesn't restart the process as usual ?

@soyuka
Copy link
Collaborator

soyuka commented Jul 22, 2015

Can you try the development version? It may fix the segfault (#1380) and just might be related to this issue.

npm i -g Unitech/PM2#development

@Unitech
Copy link
Owner

Unitech commented Jul 22, 2015

PLEASE UPGRADE TO LATEST:

$ npm install pm2 -g
$ pm2 update

@Unitech Unitech closed this as completed Jul 22, 2015
@soyuka soyuka mentioned this issue Jul 24, 2015
Closed
@iKoala
Copy link
Author

iKoala commented Aug 17, 2015

I am now using pm2 version 0.14.6 and it has been working fine for a week now.

@jshkurti jshkurti mentioned this issue Aug 18, 2015
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
S: In progress T: Bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
10 participants
@airs0urce @sbmaxx @Unitech @soyuka @iKoala @Tjatse @membrive @Mickael-van-der-Beek @crispkiwi @oscar608