15 restarts, no logs

[dandv]

My script is in an errored status and none of the log files shown by pm2 desc contain anything (they don't exist). How can I debug what caused the 15 restarts in a row?

02:16:28 [0s] stockbase:~/stockbase$ pm2 desc 4
Describing process with pid 4 - name stockbase-frontend-prod
┌───────────────────┬────────────────────────────────────────────────────────────────────┐
│ status            │ errored                                                            │
│ name              │ stockbase-frontend-prod                                            │
│ id                │ 4                                                                  │
│ path              │ /home/stockbase/stockbase-prod/frontend/stockbase-frontend-prod.js │
│ exec cwd          │ /home/stockbase/stockbase-prod                                     │
│ error log path    │ /home/stockbase/stockbase-prod/stockbase-frontend-prod.err         │
│ out log path      │ /home/stockbase/stockbase-prod/stockbase-frontend-prod-4.log       │
│ pid path          │ /home/stockbase/.pm2/pids/stockbase-frontend-prod.pid              │
│ mode              │ cluster_mode                                                       │
│ watch & reload    │ ✘                                                                  │
│ interpreter       │ node                                                               │
│ restarts          │ 15                                                                 │
│ unstable restarts │ 0                                                                  │
│ uptime            │ 0                                                                  │
│ created at        │ 1970-01-01T00:00:00.000Z                                           │
└───────────────────┴────────────────────────────────────────────────────────────────────┘
02:16:45 [0s] stockbase:~/stockbase$ cat /home/stockbase/stockbase-prod/stockbase-frontend-prod-4.log
cat: /home/stockbase/stockbase-prod/stockbase-frontend-prod-4.log: No such file or directory
02:17:03 [0s] stockbase:~/stockbase$ cat /home/stockbase/stockbase-prod/stockbase-frontend-prod.err
cat: /home/stockbase/stockbase-prod/stockbase-frontend-prod.err: No such file or directory

Also, ~/pm2/logs is empty.

[soyuka]

• Are you sure that pm2 can write in /home/stockbase/stockbase-prod/ and in /home/stockbase/.pm2/pids/?
• Are you catching unhandled exceptions in your application?

Make sure that the cwd has the wanted impact on your application.
You could try pm2 start myapp.js --no-daemon to see if it gets started as intended.

[dandv]

It seems to be a permissions issue, though I'm trying to figure out exactly which one. The exact same deployment script works if I'm logged in as dandv, and launch it from /home/dandv/stockbase, but fails as above if I'm logged in as stockbase and launch it from /home/dandv/stockbase.

• Yes, the user stockbase can create files in both those directories, though there are no .pid files when I run the deployment script as stockbase (there's one when I run it as dandv, even after I stop the process).
• Probably, but you never know.

pm2 start frontend/stockbase-frontend-prod.js --no-daemon shows

>> pm2 is already daemonized ! You should kill it before launching it in no-daemon mode
fs.js:432
  return binding.open(pathModule._makeLong(path), stringToFlags(flags), mode);
                 ^
Error: ENOENT, no such file or directory '/home/stockbase/.pm2/pm2.pid'

The permissions of /home/stockbase/.pm2/ are 775.

I ran touch pm2.pid in that directory, then pm2 start frontend/stockbase-frontend-prod.js --no-daemon -f again. Got, among other things, PM2 dameon PID = 0 with /home/stockbase/.pm2/pm2.log file not found, and again, 15 restarts.

After touch /home/stockbase/.pm2/pm2.log and running pm2 start a 3rd time, I got process launched... PM2 dameon PID = [blank, no PID], and pm2 was hanging (no output). pm2 l in another shell showed again 15 restarts. Hit ^C.

/home/stockbase/.pm2/pm2.log and /home/stockbase/.pm2/pm2.pid are 0-byte.

Looks like pm2 doesn't quite support more than one user running commands? I've been running pm2 as dandv for months, created boot scripts, then yesterday we wanted to switch to a dedicated user (stockbase) to manage pm2.

[soyuka]

What user has started pm2 the first time?
For example

ps -ef | grep pm2

Could you try specifying logs (out and err) and pid paths through a json file?

[dandv]

That must be it - pm2 was first started by dandv and now trying to control it via stockbase results in the errors above. Is it a bug perhaps? Otherwise I could try using a JSON file for the logs and paths, but it seems like command line parameters should work as well. A similar issue is why I stopped using forever.

[soyuka]

Maybe linked to #329 (comment) (multi-user suggestions) ?

The link leads to a 404 error.

[dandv]

Fixed the link, it was missing http://.

[soyuka]

Ok thanks, could you try changing the user that is running pm2 to see if it works?

You could also try using --run-as-user stockbase on a process but it should be executed from the dandv account (i.e the user that runs pm2):

dandv@x:~/ pm2 start stockbase --run-as-user stockbase

It should resolve such things but this feature is not really working according to #329.

[soyuka]

We 'll try to fix those user-related things, closing feel free to reopen if it's not fixed.

[dandv]

Now I'm getting the following error:

Error: EPERM, Operation not permitted
  at /usr/lib/node_modules/pm2/lib/ProcessContainer.js:177:15
  at WriteStream.<anonymous> (/usr/lib/node_modules/pm2/lib/ProcessContainer.js:128:9)
  at WriteStream.emit (events.js:95:17)
  at WriteStream.<anonymous> (fs.js:1667:10)

I started the pm2 daemon as the same user as the one that runs pm2 start. The script is relatively complicated; any idea what information I need to provide to debug this?

[soyuka]

What error is this? You only gave a piece of stack trace.

[dandv]

Sorry, updated the trace above.

[soyuka]

Did you use --run-as-user or --run-as-group? Please don't use these atm.

[dandv]

I was using --run-as-user as indicated earlier. In the meantime I solved the error, while still using --run-as-user, though I'm not sure exactly how due to numerous config changes. One possibility is that I restarted the pm2 daemon as the same user as the one in the .json file.

[soyuka]

Multi-user configuration is not fully functionnal atm, we are looking for a solution to improve this but it's not an easy task (#329 (comment)).

[SuryaSankar]

I ran into the same issue and @soyuka's comment about checking the first user who started pm2 helped me. It was my personal user account and not the specific account I had created for the app ( ghost blog )
So I stopped pm2, did su - ghost and started pm2 again by just running
pm2 start index.js --name ghost
And it worked. This might be the way to do it until --run-as-user is fixed

[aamiryu]

This one just bothered me yesterday, I was trying to restart in cluster mode with an option of "-i" to start multiple apps for the same node-worker but I was unable to do it. Then I looked into the folder logs/ created inside "$userHome/.pm2/" by giving a command "ls -al $userHome/.pm2/logs/*" I found that there were files which were owned by root and when I deleted them and then restarted the Apps, they were just turning ON as it never happened before I meant to say the ISSUE of ERRORED. Hope this helps somebody.

[jkulak]

In my case the problem was trivial. I looked at $ tailf ~/.pm2/pm2.log to see:

vagrant@lol-slack-bot-local:~/.pm2$ tailf ~/.pm2/pm2.log
2016-10-19 14:42:53: Starting execution sequence in -fork mode- for app name:lol-slack-bot-TEST id:0
2016-10-19 14:42:53: App name:lol-slack-bot-TEST id:0 online
2016-10-19 14:42:54: App [lol-slack-bot-TEST] with id [0] and pid [14834], exited with code [0] via signal [SIGINT]
2016-10-19 14:42:54: Starting execution sequence in -fork mode- for app name:lol-slack-bot-TEST id:0
2016-10-19 14:42:54: App name:lol-slack-bot-TEST id:0 online
2016-10-19 14:42:55: App [lol-slack-bot-TEST] with id [0] and pid [14860], exited with code [0] via signal [SIGINT]
2016-10-19 14:42:55: Starting execution sequence in -fork mode- for app name:lol-slack-bot-TEST id:0
2016-10-19 14:42:55: App name:lol-slack-bot-TEST id:0 online
2016-10-19 14:42:56: App [lol-slack-bot-TEST] with id [0] and pid [14886], exited with code [0] via signal [SIGINT]
2016-10-19 14:42:56: Script /mnt/host/mount_dir/test.js had too many unstable restarts (16). Stopped. "errored"

The part exited with code [0] via signal [SIGINT] mans that execution of my script has finished and therefore it needs to be restarted.

I did not have any server/while loops inside my script - so it run from top to bottom and was exiting as soon as it run the last line of code. As expected... This was causing the 15 restarts.

I had logs - so it is not exactly the same issue - but it was the first result I found when looking for a solution - so I'm leaving it here in case anyone else runs into the same issue.

[ktrzeciaknubisa]

I started to have same problems with pm2 2.3.0, when trying to run cluster app with -i 2 option. The app logs didn't show anything, because the problem probably was related to pm2 itself:

$ pm2 logs --lines 200
[TAILING] Tailing last 200 lines for [all] processes (change the value with --lines option)
/home/ubuntu/.pm2/pm2.log last 200 lines:
...
PM2        | path.js:1144
PM2        |           cwd = process.cwd();
PM2        |                         ^
PM2        | 
PM2        | Error: ENOENT: no such file or directory, uv_cwd
PM2        |     at Error (native)
PM2        |     at Object.resolve (path.js:1144:25)
PM2        |     at Function.Module._resolveLookupPaths (module.js:361:17)
PM2        |     at Function.Module._resolveFilename (module.js:431:31)
PM2        |     at Function.Module._load (module.js:388:25)
PM2        |     at Module.require (module.js:468:17)
PM2        |     at require (internal/module.js:20:19)
PM2        |     at Object.<anonymous> (/usr/local/lib/node_modules/pm2/lib/ProcessContainer.js:12:15)
PM2        |     at Module._compile (module.js:541:32)
PM2        |     at Object.Module._extensions..js (module.js:550:10)
PM2        | 2017-02-20 14:16:08: App name:app id:0 disconnected
PM2        | 2017-02-20 14:16:08: App [app] with id [0] and pid [28520], exited with code [1] via signal [SIGINT]
PM2        | 2017-02-20 14:16:08: Script /.../app.js had too many unstable restarts (16). Stopped. "errored"
PM2        | 2017-02-20 14:16:08: App name:app id:1 disconnected
PM2        | 2017-02-20 14:16:08: App [app] with id [1] and pid [28538], exited with code [1] via signal [SIGINT]
PM2        | 2017-02-20 14:16:08: Script /.../app.js had too many unstable restarts (16). Stopped. "errored"

Running it in non-cluster mode worked. Also worked in cluster mode, but with --no-daemon.

However after I upgraded pm2 to 2.4.0 it started to work for me.