How to make socket.io work well with pm2 when start more than one instance?

[node-rookie]

No description provided.

[klinquist]

In cluster mode, things should work fine. In fork mode, you'd need to run your socket.io server on a different port for each instance and reverse proxy to them via nginx (with ip_hash set) or HAProxy...

[thewilli]

@klinquist I have the same question as the OP. socket.io requires that each client always connects to the same worker, but it doesn't seem that the pm2 loadbalancer does some internal magic to ensure it, right? So the only option would be to use some external load balancer (like nginx) and fixes ports. But how can I get pm2 to give each worker a fixed port number that persists even a crash and restart of a worker?

[klinquist]

@thewilli

Two things:

1. "socket.io requires that each client always connects to the same worker" - says who?
   A properly written nodejs application should be stateless! Clients should absolutely not have to connect to the same worker.

2. You could map ports to the NODE_INSTANCE environment variable (which I believe PM2 sets ... that name might not be right..... check process.env.NODE_ENV to verify). But once again, doing it this way is bad practice!!

[jshkurti]

@thewilli When you run an app in cluster mode with PM2 each worker will receive a value in process.env.NODE_APP_INSTANCE which goes from 0 to number of workers-1.
Let's say you start your app with pm2 start app.js -i 4, then your workers will receive 0 1 2 3 as NODE_APP_INSTANCE value.

Then in your code you can use that value to set the listen port such as
server.listen(8000 + process.env.NODE_APP_INSTANCE)

[thewilli]

@klinquist
the docs say it, and I tried it by myself:

If you plan to distribute the load of connections among different processes or machines, you have to make sure that requests associated with a particular session id connect to the process that originated them.

[thewilli]

@jshkurti works great, thank you!

[jiajianrong]

Just wondering if the only solution for socket.io is to use sticky session?

[karlpokus]

@jiajianrong I believe long-polling is the main culprit here and can be toggled. Instructions here

[mattbrunetti]

Thank you for your note @karlpokus .. I was under the impression that WebSockets themselves required a stateful server, but of course I was wrong

TL;DR

PM2 works fine with socket.io if you use 'websocket' transport only, which doesn't rely on server state. Other transports (i.e. 'polling') depend on the server to retain some state in-between requests, and will cause errors when requests are routed to different workers (that are unaware of the state of the other workers).

WebSockets are are supported by IE 11 and all normal/evergreen browsers. If you need to support IE < 11, you will need to use the 'polling' transport, and thus you will need to have a stateful server and cannot use PM2 clustering. If this is the case, I suggest you look into clustering using sticky-listen (a fork of sticky-session)

[qiulang]

I can confirm that except for the latest firefox (56), chrome/safari/opera all work when using 'websock' only with pm2 cluster.

I got following error message when using firefox:

Firefox can’t establish a connection to the server at ws://localhost:3000/socket.io/?EIO=3&transport=websocket.

The connection to ws://localhost:3000/socket.io/?EIO=3&transport=websocket was interrupted while the page was loading.

But I can't make sticky-session solution work!

[knoxcard]

thank you @jshkurti! I couldn't find your solution anywhere else on the web.

Here is what my end code looks like...

SERVER

  var http = require('http')
  http.globalAgent.maxSockets = Infinity
  app.server = http.Server(app)
  app.server.listen(parseInt(app.nconf.get('app:port')) + (process.env.NODE_APP_INSTANCE ? parseInt(process.env.NODE_APP_INSTANCE) : 0), () => {
    app.sio = require('socket.io').listen(app.server, {
      transports: [
        'websocket',
        'polling',
        'long-polling'
      ]
    })
    app.sio.adapter(require('socket.io-redis')({
      host: 'localhost',
      port: 6379
    }))
    app.sio.use((socket, next) => {
      app.sessionMiddleware(socket.request, socket.request.res, next)
    })
    app.sio.sockets.on('connection', socket => {
      var room = 'room_' + (socket.request.session.user_id ? socket.request.session.user_id : socket.request.session.id)
      socket.join(room)
      app.sio.sockets.in(room).emit('auth', {
        action: 'check'
      })
    })
    app.sio.sockets.on('disconnect', () => {
      console.log(app.colors.red('[socket disconnected]'))
    })
  })

CLIENT

	jQuery.socket = io()
	jQuery.socket.on('connect', function() {
		jQuery.socket.on('auth', function(ret) {
		})
	})
	jQuery.socket.on('disconnect', function() {
		jQuery.reconnect = null
		jQuery.reconnect = setInterval(function() {
			jQuery.socket = io.connect()
			console.log('Attempting server connection...')
		}, 10000)
	})

[subatomicglue]

Redis pub/sub should coordinate a shared state between all the pm2 workers... so shouldn't that allow polling to work?

So I see solution proposed above, recommending a unique port per nodejs process

How would this work with a proxy server like nginx that has only one port exposed to the client.

[tvvignesh]

@jshkurti If we use the instance env to generate one port for every node like what you said, how would we expose the same via a reverse proxy like nginx? What port number should we specify there?

UPDATE: Never mind. Looks like I have to set upstream servers to do this: http://nginx.org/en/docs/http/load_balancing.html

[Gby56]

@tvvignesh yeah you loose the loadbalancing from pm2 if you use @jshkurti 's solution, and end up doing it with nginx 😅

[ffflabs]

Thing is, if you want to keep an app leveraging e.g. Express and socket.io, you could receive a large file upload using multer, then transform the payload to another format, then upload it to Google drive, then sharing it with a friend and all along be notified about the operation progress, errors, events, etc, you'd like to store some kind of session info.

For example, if the visitor closed the tab, reopening it should let him retake the process where he left it. In express you would use a cookie to identify yourself no matter what worker answers the request. But in socket.io a connection opens a socket. If you communicate with four workers each one of them will try and keep a websocket connection with you.

The first one tells you "welcome, you joined channel x". You send a message back and it is received by the second worker, for whom the channel x doesn't exist. Even if the room had the same name, it's a different socket.

It's easy to propagate the express session to the websocket listener, but it's not trivial the other way around. You can't store a reference to the socket in the session because it's not serializable nor able to be rehydrated. Moreover, if socket.io rewrites a "socket" attribute of the express session, it may mutate during the request lifetime. Calling "emit" on said property might not reach the original user ever.

I've tried using sticky-listen and recluster, but then you have like 4 threads running jammed in one pm2 worker, where you can't benefit from graceful reloading or restarting workers independently given a memory threshold.

C'mon, trying to monkeypatch node clustering to mock what pm2 already does, just to make sure you always communicate with the same worker... It already led me to crashing due to malloc or mem leaks.

So... Everything seems to be pointing to redis.

[hdodov]

@thewilli When you run an app in cluster mode with PM2 each worker will receive a value in process.env.NODE_APP_INSTANCE which goes from 0 to number of workers-1.
Let's say you start your app with pm2 start app.js -i 4, then your workers will receive 0 1 2 3 as NODE_APP_INSTANCE value.

Then in your code you can use that value to set the listen port such as
server.listen(8000 + process.env.NODE_APP_INSTANCE)

@jshkurti thanks for the insight! I don't get one thing. My workers will listen on the respective ports (in this case, 8001, 8002...). However, I use nginx as a reverse proxy and how should I configure it to forward the requests to these ports? I mean, I forward /socket.io requests to http://localhost:8000/ and if I keep doing that, I'll still be using only one worker. The others would sit around and do nothing, right?

[PsyGik]

@hdodov, Maybe this way

http {
    upstream myapp1 {
        server srv1.example.com:8001;
        server srv1.example.com:8002;
        server srv1.example.com:8003;
    }

    server {
        listen 80;
        location / {
            proxy_pass http://myapp1;
        }
    }
}

[averri]

@jshkurti, what you have mentioned about changing the port numbers for pm2 cluster mode is not required. pm2 shares the same port across different process (thanks to Nodejs cluster mode) - Reference: https://pm2.keymetrics.io/docs/usage/cluster-mode.

If you are changing the ports of your application for using the pm2 cluster mode you are doing it wrong!