Websocket zmq not working with reverse proxy

[kennhardy]

When using docker, either configuring everything manually with a regular setup or when using the install script, websocket is not working. Something seems to be missing from a regular setup using php. It seems nginx are not able to understans or translate the zmq and hence not becoming a socket. Instead only reaching the endpoint over http(s).

After spending countless hours on this I decided to cash up for a VM to try it outside my current hosting setup and came to the same conclusion result. Or I might be missing something fundamental, if so I'd love some help.

"auto-install" with ssh:

# MODE: letsencrypt
version: '3'
services:
 espocrm-mysql:
   image: mysql:8
   container_name: espocrm-mysql
   command: --default-authentication-plugin=mysql_native_password
   restart: always
   environment:
     MYSQL_ROOT_PASSWORD: pass
     MYSQL_DATABASE: espocrm
     MYSQL_USER: espocrm
     MYSQL_PASSWORD: pass
   volumes:
     - ./data/mysql/data:/var/lib/mysql
   networks:
     - internal

 espocrm-nginx:
   image: nginx
   container_name: espocrm-nginx
   restart: always
   depends_on:
     - espocrm
   volumes:
     - ./data/nginx/conf.d/:/etc/nginx/conf.d/
     - ./data/espocrm:/var/www/html
     - ./data/nginx/espocrm.conf:/etc/nginx/espocrm.conf
     - ./data/nginx/logs:/var/log/nginx
     - ./data/nginx/ssl:/etc/nginx/ssl
     - ./data/nginx/certbot:/var/www/certbot
   ports:
     - "80:80"
     - "443:443"
   command: "/bin/sh -c 'while :; do sleep 6h & wait $${!}; nginx -s reload; done & nginx -g \"daemon off;\"'"
   networks:
     - external

 espocrm:
   image: espocrm/espocrm:fpm
   container_name: espocrm
   environment:
     ESPOCRM_DATABASE_HOST: espocrm-mysql
     ESPOCRM_DATABASE_USER: espocrm
     ESPOCRM_DATABASE_PASSWORD: pass
     ESPOCRM_ADMIN_USERNAME: admin
     ESPOCRM_ADMIN_PASSWORD: pass
     ESPOCRM_SITE_URL: "https://espocrm.domain.tld"
     ESPOCRM_CONFIG_WEB_SOCKET_ZERO_M_Q_SUBMISSION_DSN: "tcp://espocrm-websocket:7777"
   restart: always
   depends_on:
     - espocrm-mysql
   volumes:
    - ./data/espocrm:/var/www/html
   networks:
     - external
     - internal

 espocrm-daemon:
   image: espocrm/espocrm:fpm
   container_name: espocrm-daemon
   volumes:
     - ./data/espocrm:/var/www/html
   restart: always
   depends_on:
     - espocrm
   entrypoint: docker-daemon.sh
   networks:
     - external
     - internal

 espocrm-websocket:
   container_name: espocrm-websocket
   image: espocrm/espocrm:fpm
   environment:
     ESPOCRM_CONFIG_WEB_SOCKET_ZERO_M_Q_SUBSCRIBER_DSN: "tcp://*:7777"
     ESPOCRM_CONFIG_USE_WEB_SOCKET: "true"
   volumes:
    - ./data/espocrm:/var/www/html
   restart: always
   depends_on:
     - espocrm
   entrypoint: docker-websocket.sh
   ports:
     - 8080:8080
   networks:
     - external
     - internal

 espocrm-certbot:
   image: certbot/certbot
   container_name: espocrm-certbot
   restart: always
   volumes:
     - ./data/nginx/ssl:/etc/letsencrypt
     - ./data/nginx/certbot:/var/www/certbot
   entrypoint: "/bin/sh -c 'trap exit TERM; while :; do certbot renew; sleep 12h & wait $${!}; done;'"
   networks:
     - external

networks:
 external:
   external: true
 internal:
   external: false

[kennhardy]

Has anyone else gotten the websocket's working together with docker and nginx as a reverse proxy?

[frauhottelmann]

I haven't gotten it to work with HaProxy either. Not sure about php-zmq, it doesn't seem to be maintained any more. At least not compared to the other zmq variants.

[kennhardy]

Yes. Not sure what breaks it. I have two instances running in docker, and another two running in kubernetes. Same with both. No matter what I try, websockets does not work.

[digitalextremist]

@kennhardy I have a nginx reverse proxy, docker containers, and websockets working. Will come back to chime in on this.

[kennhardy]

@digitalextremist oh, that sounds promising. Would you be able to share your config? Are you running out in docker as well?

[digitalextremist]

By the way, first impression: My configuration is very different because I have nginx running on the host machine, and that provides individual reverse proxies to each instance running. So there is no nginx in my docker-compose.yml at all. Therefore also no certbot in my docker configuration.

But looking closer at what you have, I am actually missing ESPOCRM_CONFIG_WEB_SOCKET_ZERO_M_Q_SUBMISSION_DSN completely. I am going to add that back with a new instance and double check if I have ZMQ working as I thought. I am having no websocket problems with my configuration right now that I know of though!

Will check back.

[digitalextremist]

@kennhardy I have since been able to get the part you are working with running with a reverse proxy, but I am starting to wonder if the background communications are either being dropped, or not happening. But the websocket itself is connected :)

Still have this on my mind and returning with more soon hopefully. Anything change for you?

[digitalextremist]

I do see there is ICMP traffic possible between containers though...

And when I turned on webSocketDebugMode I do see these:

websocket  | [2023-09-27 01:09:58] message newNotification for user 6501a8b5bdc913270
websocket  | [2023-09-27 01:09:58] message newNotification for user 65137fa3ec7b8ac51
websocket  | [2023-09-27 01:09:58] message streamUpdate.Case.651371e675bf83170 for all

Now it seems like the issue is that they never show up on the UX end.

[digitalextremist]

Wait, there's the error:

[Shahin-rmz]

@digitalextremist my config is much like yours. do you get notifications and reminders or you also have problem?

[digitalextremist]

I was able to get this worked out @Shahin-rmz

Would need to recall, but I think I completely redid my configuration after this point.

[Shahin-rmz]

I was able to get this worked out @Shahin-rmz

Would need to recall, but I think I completely redid my configuration after this point.

It would be awesome if you share your docker compose, nginx Proxy manager config whenever you have time.
Thanks

[FrankSchoene]

Did anybody solve the websocket issue? I'm experiencing the same behind traefik. Would be great to get feedback.

[arkadiyasuratov]

Example Traefik labels for Websocket container:

# Traefik
traefik.enable: 'true'
traefik.docker.network: traefik-proxy  # Use your own network

# Traefik for EspoCRM Websocket
traefik.http.routers.espocrm-ws.rule: Host(`example.com`) && PathPrefix(`/ws`)  # Use your own domain
traefik.http.routers.espocrm-ws.entrypoints: websecure  # Use your own entrypoint
traefik.http.routers.espocrm-ws.tls: 'true'
traefik.http.routers.espocrm-ws.tls.certresolver: cloudflare  # Use your own certresolver
traefik.http.routers.espocrm-ws.service: espocrm-ws-svc
traefik.http.services.espocrm-ws-svc.loadbalancer.server.port: '8080'

And in the environment:

ESPOCRM_CONFIG_USE_WEB_SOCKET: 'true'
ESPOCRM_CONFIG_WEB_SOCKET_URL: wss://example.com/ws  # Use your own domain
ESPOCRM_CONFIG_WEB_SOCKET_ZERO_M_Q_SUBSCRIBER_DSN: tcp://*:7777
ESPOCRM_CONFIG_WEB_SOCKET_ZERO_M_Q_SUBMISSION_DSN: tcp://websocket:7777  # Assuming the service name is 'websocket'

[FrankSchoene]

Thanks a lot!
I additionally had to map port 8080 to 8443 explicitly -- what should be standard -- but seems to works now.
It took me a serious number of attempts to convince Google Chrome to accept the connection (even in Incognito Mode) -- but since Firefox did not throw errors I kept trying -- and all of a sudden it worked.
I will observe it now and get back in a couple of days.