"Error: failed [502] Bad Gateway" then Push stops working

[k-ahn]

Description:

I am getting "Error: failed [502] Bad Gateway" in syslog, following which Push notifications stop working until i restart the server. This has happened three times in the last 2 weeks at different times.

Steps to reproduce:

Unknown

Expected behavior:

Push works

Actual behavior:

Push stops working

Server Setup Information:

Version of Rocket.Chat Server: 3.2.2
Operating System: Ubuntu 18.04.5 LTS (Bionic Beaver)
Deployment Method: snap
Number of Running Instances: 1
DB Replicaset Oplog: Enabled
NodeJS Version: 12.16.1 - arm64
MongoDB Version: 3.6.17

Client Setup Information

Affects both Android and iOS clients

Additional context

None

Relevant logs:

Sep 13 05:12:04 server rocketchat-server.rocketchat-server[19437]: Error: failed [502] Bad Gateway
Sep 13 05:12:04 server rocketchat-server.rocketchat-server[19437]: at Object.exports.makeErrorByStatus (packages/http.js:176:10)
Sep 13 05:12:04 server rocketchat-server.rocketchat-server[19437]: at Request._callback (packages/http.js:140:24)
Sep 13 05:12:04 server rocketchat-server.rocketchat-server[19437]: at Request.self.callback (/snap/rocketchat-server/1437/programs/server/npm/node_modules/meteor/http/node_modu
Sep 13 05:12:04 server rocketchat-server.rocketchat-server[19437]: at Request.emit (events.js:311:20)
Sep 13 05:12:04 server rocketchat-server.rocketchat-server[19437]: at Request.EventEmitter.emit (domain.js:482:12)
Sep 13 05:12:04 server rocketchat-server.rocketchat-server[19437]: at Request. (/snap/rocketchat-server/1437/programs/server/npm/node_modules/meteor/http/node_module
Sep 13 05:12:04 server rocketchat-server.rocketchat-server[19437]: at Request.emit (events.js:311:20)
Sep 13 05:12:04 server rocketchat-server.rocketchat-server[19437]: at Request.EventEmitter.emit (domain.js:482:12)
Sep 13 05:12:04 server rocketchat-server.rocketchat-server[19437]: at IncomingMessage. (/snap/rocketchat-server/1437/programs/server/npm/node_modules/meteor/http/nod
Sep 13 05:12:04 server rocketchat-server.rocketchat-server[19437]: at Object.onceWrapper (events.js:417:28)
Sep 13 05:12:04 server rocketchat-server.rocketchat-server[19437]: at IncomingMessage.emit (events.js:323:22)
Sep 13 05:12:04 server rocketchat-server.rocketchat-server[19437]: at IncomingMessage.EventEmitter.emit (domain.js:482:12)
Sep 13 05:12:04 server rocketchat-server.rocketchat-server[19437]: at endReadableNT (_stream_readable.js:1204:12)
Sep 13 05:12:04 server rocketchat-server.rocketchat-server[19437]: at processTicksAndRejections (internal/process/task_queues.js:84:21) {
Sep 13 05:12:04 server rocketchat-server.rocketchat-server[19437]: response: {
Sep 13 05:12:04 server rocketchat-server.rocketchat-server[19437]: statusCode: 502,
Sep 13 05:12:04 server rocketchat-server.rocketchat-server[19437]: content: 'Bad Gateway',
Sep 13 05:12:04 server rocketchat-server.rocketchat-server[19437]: headers: {
Sep 13 05:12:04 server rocketchat-server.rocketchat-server[19437]: date: 'Sun, 13 Sep 2020 05:12:04 GMT',
Sep 13 05:12:04 server rocketchat-server.rocketchat-server[19437]: 'content-length': '11',
Sep 13 05:12:04 server rocketchat-server.rocketchat-server[19437]: 'content-type': 'text/plain; charset=utf-8',
Sep 13 05:12:04 server rocketchat-server.rocketchat-server[19437]: connection: 'close'
Sep 13 05:12:04 server rocketchat-server.rocketchat-server[19437]: },
Sep 13 05:12:04 server rocketchat-server.rocketchat-server[19437]: data: null
Sep 13 05:12:04 server rocketchat-server.rocketchat-server[19437]: }
Sep 13 05:12:04 server rocketchat-server.rocketchat-server[19437]: }

[plociu84]

Maybe it's related whit this node problem: nodejs/node#5436

[k-ahn]

Any update on this?

[davidmailreaderio]

This is also happening to me. Can someone please advise what to do? Happened 2 times, 2 days in a row.

[davidmailreaderio]

Just happened again. All I do to rectify is run "sudo systemctl restart rocketchat" on my Ubuntu installation. Any help would be greatly appreciated, the missed notifications are causing a lot of pain.

[k-ahn]

Any update on this? Still happening

[mohmyo]

I have the same issue, rocket gets down suddenly for no reason, here a sample of my logs:

May 18 12:00:08  rocketchat[657]: Error: failed [502] Bad Gateway
May 18 12:00:08  rocketchat[657]:     at Object.exports.makeErrorByStatus (packages/http.js:182:10)
May 18 12:00:08  rocketchat[657]:     at Request._callback (packages/http.js:142:24)
May 18 12:00:08  rocketchat[657]:     at Request.self.callback (/opt/rocket/Rocket.Chat/programs/server/npm/node_modules/meteor/http/node_modules/request/request.js:185:22)
May 18 12:00:08  rocketchat[657]:     at Request.emit (events.js:314:20)
May 18 12:00:08  rocketchat[657]:     at Request.EventEmitter.emit (domain.js:483:12)
May 18 12:00:08  rocketchat[657]:     at Request.<anonymous> (/opt/rocket/Rocket.Chat/programs/server/npm/node_modules/meteor/http/node_modules/request/request.js:1161:10)
May 18 12:00:08  rocketchat[657]:     at Request.emit (events.js:314:20)
May 18 12:00:08  rocketchat[657]:     at Request.EventEmitter.emit (domain.js:483:12)
May 18 12:00:08  rocketchat[657]:     at IncomingMessage.<anonymous> (/opt/rocket/Rocket.Chat/programs/server/npm/node_modules/meteor/http/node_modules/request/request.js:1083:12)
May 18 12:00:08  rocketchat[657]:     at Object.onceWrapper (events.js:420:28)
May 18 12:00:08  rocketchat[657]:     at IncomingMessage.emit (events.js:326:22)
May 18 12:00:08  rocketchat[657]:     at IncomingMessage.EventEmitter.emit (domain.js:483:12)
May 18 12:00:08  rocketchat[657]:     at endReadableNT (_stream_readable.js:1241:12)
May 18 12:00:08  rocketchat[657]:     at processTicksAndRejections (internal/process/task_queues.js:84:21) {
May 18 12:00:08  rocketchat[657]:   response: {
May 18 12:00:08  rocketchat[657]:     statusCode: 502,
May 18 12:00:08  rocketchat[657]:     content: 'Bad Gateway',
May 18 12:00:08  rocketchat[657]:     headers: {
May 18 12:00:08  rocketchat[657]:       date: 'Tue, 18 May 2021 12:00:08 GMT',
May 18 12:00:08  rocketchat[657]:       'content-length': '11',
May 18 12:00:08  rocketchat[657]:       'content-type': 'text/plain; charset=utf-8',
May 18 12:00:08  rocketchat[657]:       connection: 'close'
May 18 12:00:08  rocketchat[657]:     },
May 18 12:00:08  rocketchat[657]:     data: null
May 18 12:00:08  rocketchat[657]:   }
May 18 12:00:08  rocketchat[657]: }