Latest Socket.io and pm2 causing issues when using -i max

[DPerkunas]

When I launch my app.js file via pm2 start app.js -i max I get the following error on client end
" failed: Connection closed before receiving a handshake response "

when I launch the app.js via "pm2 start app.js" it works fine.

I utilize socket.io in my application on apache. The error logs are empty, but I do notice that it emits a console.log("hello world"); twice upon start from each start up on each of the cores. The websocket then keeps connecting and disconnecting.

One thing I think it could be, is that it's supposed to be "stateless", and I hold a list of online users in users = {};

But why would it work when it's only one CPU vs 4?

[rlidwka]

if you're using more than one cpu, users can hit different processes with their requests

try to keep shared state somewhere, in redis for example

[DPerkunas]

So it WAS because of my not following of "stateless". Gotcha, gotta move that user list to redis. Silly me. Thanks.

Edit: Unitech, if you can also check this out, it'd be great. Now that I re-read my own post and tested, it still doesn't make sense to me why not being stateless would cause socket.io not to be found. It would make sense for the app to crash, but not to not find socket.io. Thanks.

[Unitech]

Cluster module doesn't play well with Socket.io 1.x

They ask to do sticky load balancing in order to redirect each user to the same process (http://socket.io/blog/introducing-socket-io-1-0/#scalability) and the Node cluster module does not permit this.

The first solution I see is to fork_mode processes and attributing them different port (via JSON conf). Then with Nginx to do the sticky load balancing (http://socket.io/docs/using-multiple-nodes/#nginx-configuration)

The second solution would be to use the https://github.com/indutny/sticky-session, but unfortunately I can't embed it in PM2.

In conclusion I don't really like this latest socket.io version, as it break the stateless paradigm.

[rlidwka]

sticky-session looks fishy, I mean you can't reconnect with different ip address for example.

But they do recommend redis as the second option, what's wrong with that?

[Unitech]

If I understood well, the redis stuff is about passing events between processes (http://socket.io/docs/using-multiple-nodes/#passing-events-between-nodes)

And as written on the scalability page:
Instead of storing and/or replicating data across nodes, Socket.IO is now only concerned with passing events around.
So each process is only concerned with his very own connections

[DPerkunas]

So we'll leave it at, Socket.IO is the issue, pm2 is fine and nothing we could do about it. Sticky-Session throws a random error after using it for ever so long according to some posts. This is something to be discussed with Socket.IO. Thanks guys.

[soyuka]

Please try to link socket.io issues to this thread so that we might follow them.

Closing this as a non-pm2 issue, feel free to comment or discuss here.

[handleror]

I got same issue! If I did try to run single instance it worked fine, but got a problem for multiple instances.

[lalith-b]

set the protocol to ws from client side while connecting

Client Side - somewhere in js

io.connect({transports: ['websocket']})

Server side - doing this is solving the problem

io.adapter(redis({ host:SETTINGS.redis.host, port: SETTINGS.redis.port }));

Note

It will be a problem with supporting IE and other old browsers as they don't support native websockets, if the application browser compatibility is not an issue this seems a solution.

[lucaswxp]

@deathlord87 answer helped, but I still get, occasionally, "failed: Error during WebSocket handshake: Unexpected response code: 502". Anyone knows how to fix it?

[lalith-b]

the 502 error comes when there's a requirement of the client when set to server to handle xhr-long polling or other protocols other than websockets or if the websocket protocol is not supported.

[fiora]

Hi, with a cluster requests may arrive to different workers, which will break handshake protocol. Can pm2 use the same way as indutny/sticky-session to passing request to child process?

[BrandonCopley]

I use a system where I store my sessions in redis, but load the sessions to the server on a connect. This way my system is "stateless" but faster than always having to talk to the redis server. What is pm2 doing in the background that makes it to where we can't create sticky sessions?

[brandonros]

This isn't necessarily a pm2 issue, but using pm2's command line options prevents all of the public code out there from being used (as they require you do the cluster calls yourself).

What would you suggest?

[houmark]

I know this one is old and closed, but I really didn't find a simple straight forward solution — or I miss it while searching around in docs anywhere (did not help that socket.io is currently down for days).

FWIW I wanted to share what seems to work for me. This solution might break old browsers that does not support true websockets, but this is IE9 and below. Our application does not support those anyways, so...

Swapping the transports order both on the client and server side removed the handshake error and random poll based errors with "session ID not found".

So on the server (need to use redis as an adaptor to socket.io):

var io = require('socket.io')(3000, { transports: ['websocket', 'polling'] });
var redis = require('socket.io-redis');
io.adapter(redis({ host: 'localhost', port: 6379 }));

And on the client:

io.connect('http://server/url', {'transports': ['websocket', 'polling']});

I hope it helps someone out there. Took me many hours of trial and error to figure out.

[http91]

@houmark, I've been trying to figure this out for two days, your solution works for me. caniuse shows that websockets support is ~87%, not bad, but I wish there was a prettier solution.

Thanks.

[esellier]

Hi,

I succeeded (it was a bit tricky... for me at least...) to implement an Apache Proxy/Balancer "load balancing" between two nodes managed with PM2. These nodes implement both an Express server and a Socket.IO server (polling or websocket - transports order has not been modified - thus fully compatible with older browsers not implementing websockets).

• PM2 (nodes in fork mode, but that works also with clusters using NODE_APP_INSTANCE "trick")
• Socket.IO 1.0
• NodeJS 4.4.3
• Apache 2.4.7 (Ubuntu/Trusty) - standard package for Trusty - fork mode (I also have old PHP running, so I couldn't easily switch to an hybrid or other worker mode - BTW, it's also why I've not switched to Nginx yet)

Client App (both HTTPs and Socket.io) ==> Apache Proxy (listening on 443 - SSL/TLS) ==> PM2/Nodes (listening on 3027 and 3028 for instance)

I didn't use sticky-session module, nor REDIS. But it's "sticky" from the Apache point of view.
1/ The client app makes an HTTPs request (to Apache Proxy 443 port)
2/ Apache affects a route for this new session (if sticky-session cookie is not set), and will proxy the request to the Node (port 3027 or 3028 regarding elected route)
3/ Node (Express) will reply with (my app specific) data AND a session cookie (format .)
4/ AFTER response to 1/, the client app connects using socket.io to Apache Proxy 443 port (the cookie data is only used by Apache, socket.io client still connects to the proxy on standard 443 port)
5/ Apache gets the request and the sticky session cookie set at 1/, the cookie also contains the route id for balancing
6/ Apache routes the connection to one of the nodes (to 3027 or 3028)
7/ Client App runs the polling and upgrade process for establishing websocket connection (when possible for the client browser)

If anyone is interested, please reply or contact me and I'll give more information.
It's quite simple in the end, and nearly obvious, but was tough for me to set up. So I hope this helps.
Any comment/remark on this configuration is welcome !!

Apache configuration:

<VirtualHost>
        ...
        <Proxy balancer://socket>
            BalancerMember ws://mybackend.mydomain.com:3027 route=3027 retry=1 acquire=1000
            BalancerMember ws://mybackend.mydomain.com:3028 route=3028 retry=1 acquire=1000
            ProxySet lbmethod=byrequests
            ProxySet nofailover=on
            ProxySet timeout=1000
            ProxySet stickysession=ROUTEPORT
            ProxySet forcerecovery=On
        </Proxy>
        <Proxy balancer://myapp>
            BalancerMember http://mybackend.mydomain.com:3027 route=3027
            BalancerMember http://mybackend.mydomain.com:3028 route=3028
            ProxySet lbmethod=byrequests
            ProxySet nofailover=on
            ProxySet timeout=1000
            ProxySet stickysession=ROUTEPORT
            ProxySet forcerecovery=On
        </Proxy>

        RewriteEngine On
        RewriteCond %{QUERY_STRING} transport=websocket  [NC]
        RewriteRule /(.*)           balancer://socket/$1 [P,L]

        RewriteEngine On
        RewriteCond %{QUERY_STRING} transport=polling   [NC]
        RewriteRule /(.*)           balancer://myapp/$1 [P,L]

        RewriteEngine On
        RewriteCond %{REQUEST_SCHEME} ws  [NC]
        RewriteRule /(.*)           balancer://socket/$1 [P,L]

        # Proxy also serves HTTP requests for another app (node listening on 3013)
        # multiple other apps could be configured the same way
        RewriteEngine On
        RewriteCond %{REQUEST_URI} ^/otherapp [NC]
        RewriteRule /(.*)           http://mybackend.mydomain.com:3013/$1 [P,L]

        RewriteEngine On
        RewriteCond %{REQUEST_SCHEME} http  [NC]
        RewriteCond %{REQUEST_URI} !^/otherapp [NC]
        RewriteRule /(.*)           balancer://myapp/$1 [P,L]
</VirtualHost>

Cookie management in the Node/Express server:

    app.get("/myapp/getinfo", function(req,res) {
        if (!req.cookies.ROUTEPORT) // dependency :  require('cookie-parser') set for Express
        {
            var sessionid=uid.sync(18); // dependency : uid=require('uid')
            res.cookie('ROUTEPORT', sessionid+'.'+socketPort, {
                expires: 0,
                httpOnly: false
            });
        }
        // reply...
    });

[efkan]

Hello @houmark and @deathlord87 ,

I wonder that do not I need to use sticky session in this way (specifying transports) ?

I tested your way in a sample chat app. (http://socket.io/get-started/chat/)

I ran the app cluster mode with 2 cores of my CPU by using PM2. So actually I ran 2 instances.
In the app I count the connected clients' connections and I log to console at each 15 seconds once.
Also I wrote the current connected instance name to the web page.

When I connect to the app on 2 different browser which are present at 2 different machines and if I connected to the instance 1 with two browsers, instances write to the console the outputs below;

instance 2 total connection = 2
instance 1 total connection = 2

If I use only one browser to connect to the app the console outputs to be as follows;

instance 2 total connection = 1
instance 1 total connection = 1

If I connected to instance 2 on both of browsers and if I refresh the web pages quickly several times then the console outputs are as follows;

instance 2 total connection = 8
instance 1 total connection = 4

I think that result shows me that PM2 and Socket.IO run properly!

On the other hand, although If I connect different instances with browsers (browser A on instance 1 and browser B on instance 2) and if I write hello and on a browser then the other one can receive the hello message. And when I wrote world message on the other one, I saw the message on the first one.

In normal circumstances I need to configure pub/sub methods to share Socket messages, right?
Also I need use sticky session to handle socket handshakes right?
So, I never done above missions but now all these things work properly as exciting.
Did I kill two birds with one stone in this way ?????????

Thanks gentlemen. I never encountered with this way nowhere on the net.

---

Edit

For an app that's run on only 1 server everything is OK.

However, when using an Nginx load balancer and a second App server, Redis store usage in a Socket.io app cannot save me :) And it throws the message as follows;

VM53:35 WebSocket connection to 'ws://192.168.1.100/socket.io/?EIO=3&transport=websocket' failed: Error during WebSocket handshake: Unexpected response code: 400

Actually this situation had already discussed on #1942 and its solution is over here socketio/socket.io#1942 (comment).

Also I divided ports of every App instance on Nginx as directed aPoCoMiLogin over here #389 (comment).

Now, my app runs without any error.

[cyrilchapon]

Socket.io websocket transport is a solution for client / server as long the "client" is a browser..

From a node.js script, this doesn't seem to work at all (which is pretty strange)

[nimatrazmjo]

https://medium.com/@codeandcam/clustering-an-express-socket-io-server-with-pm2-789d11865f95

[cyrilchapon]

@nimatullah thank god someone did this.

Insane ❤️