Move towards mosquitto / rabbitmq ?

[skx]

I'm increasingly thinking that I should move towards using MQ / mosquitto as the transport for communication, so I'll file this issue to sollicate feedback.

In my view the point of this software is not to offer a service that people can use, but instead to share software that users can host themselves. If people wanted something "free" and seamless they could use ngrok itself. It offers a nice console-based GUI, a decent set of stats, and similar.

I'm running my own instance here so that users can demo the tool, and try it out in a simple fashion. Not because I want to run a service that users can rely upon directly. If I'm committing to run a service myself it will become a support-burden, and the only way that will be eased is if people throw piles of shiny notes at me.

If I use MQ then setup for self-hosting users is still simple; instead of using apache/nginx they just apt-get install mosquitto (or the local equivilent).

I can stil allow users to demo it, by defaulting to the test install of mosquitto, available online

• http://test.mosquitto.org/

Of course anybody could be listening ..

[skx]

Yes I think this is the way to go; it will be significantly more easy to scale.

[fabfurnari]

You could setup user and password on mosquitto, and also certificates.
IMHO MQ are excellent for this kind of transport, but give a look to ZeroMQ (0MQ).
Maybe MQTT could be a little feature overkill for this...

[fabfurnari]

Consider also that a webserver in front of the whole server (for certificat/password/ecc...) would be still necessary. Adding a MQ is another component to install and configure.
In my case on the server side all I had to configure is an apache VHost (one file) and DNS wildcard (plus the certificate). Very simple.

[skx]

I realize there's extra complexity, and another service to run, but when I've been heavily testing multiple concurrent users there have been issues with the blocking nature of the websockets.

I'm very certain those would be resolved entirely by the use of the queue; but I think despite the extra-service being required this would still be simple to deploy; a pair of dockerfiles and a docker-composer file to link them together should be simple.

I guess it is all a bit of a trade-off, but I think the simplification and scalability will make it worthwhile.

[fabfurnari]

Check out ZeroMQ as I said, is simple enough to be integrated into the server and client part and doesn't require running another process because is brokerless

[skx]

Currently looking at it, and am writing some test code.

Thanks for the pointer, and your feedback. :)

Struggling to find a decent example that does pub & sub, with subscribe / unsubscribe cleanly. There's a lot of new concepts and reading to be done. Will finish the half-workign server-port I've got then come back to it I think; rather than leaving this pull request in a semi-broken state.

[skx]

So there are a few choices:

• zmq4
  • https://github.com/go-zeromq/zmq4
• magos-v1
  • https://github.com/nanomsg/mangos

The latter has a simple example which seems clear and readable; need to see if the polling blocks and how much overhead there is. By contrast I don't understand how this works.

I guess the end result would be that I'd present a HTTP interface on one port, and a Queue interface on another.

[skx]

• Merged MQ request since it is an improvement.
  • Though I recognise requiring an extra service is a pain.
• TODO
  • Look at embedded queue tools.
  • I spent a few hours on that last night and didn't have success with any experiments will try again over the next few days.

[skx]

So I stalled on making visible progress here. To recap, briefly:

• Originally the server was standalone and served both HTTP, for proxying, and websockets for the tunnelling.
• That became unreliable at scale because .. reasons.

So I wanted to use a queue instead, so that the threading/bottleneck would be outside this code. This does complicate the installation because it means installing a second service and exposing it to the world. The ideal would be to use an embedded queue-library. However I could find none useful for my needs.

So right now I'm torn between declaring this done, and leaving as-is, or switching to using long-lived GRPC connections. That would allow the tunneling to work sanely, and would remove the MQ need. I need to explore that further.