Electrum over Tor disconnects from Electrumx after about 5 minutes

[ghost]

I run an Electrumx (1.11.0) server (also) over Tor and sometimes test it by starting Electrum, also running over Tor, from my laptop. Electrum wouldn't connect to my server anymore, while Electrum without Tor has no problems connecting to my server.

I found out that if I restart Electrumx, my Electrum over Tor reconnects and shows the right amounts but after about 5 minutes (maybe it's doing a resync) it disconnects and won't connect anymore. Even restarting Electrum doesn't help: it simply won't connect to my server anymore (while Electrum without Tor has no problems, even running it longer (about 10 minutes)).

The Electrum wallet (.electrum) is created via my Trezor hardware wallet and I use it while the Trezor is not connected but that shouldn't be a problem, it shows the correct amounts initially.

I see the following Electrum error message (probably) when it disconnects:

  File "/home/user/Electrum-3.3.4/electrum/util.py", line 910, in wrapper
    return await func(*args, **kwargs)
  File "/home/user/Electrum-3.3.4/electrum/interface.py", line 289, in wrapper_func
    return await func(self, *args, **kwargs)
  File "/home/user/Electrum-3.3.4/electrum/interface.py", line 307, in run
    await self.open_session(ssl_context)
  File "/home/user/Electrum-3.3.4/electrum/interface.py", line 407, in open_session
    await group.spawn(self.monitor_connection)
  File "/home/user/Electrum-3.3.4/packages/aiorpcx/curio.py", line 240, in __aexit__
    await self.join()
  File "/home/user/Electrum-3.3.4/packages/aiorpcx/curio.py", line 207, in join
    raise task.exception()
  File "/home/user/Electrum-3.3.4/electrum/interface.py", line 419, in ping
    await self.session.send_request('server.ping')
  File "/home/user/Electrum-3.3.4/electrum/interface.py", line 107, in send_request
    timeout)
  File "/usr/lib/python3.6/asyncio/tasks.py", line 358, in wait_for
    return fut.result()
  File "/home/user/Electrum-3.3.4/packages/aiorpcx/session.py", line 523, in send_request
    raise result
aiorpcx.jsonrpc.RPCError: (-101, 'excessive resource usage')
Traceback (most recent call last):
  File "/home/user/Electrum-3.3.4/electrum/util.py", line 910, in wrapper
    return await func(*args, **kwargs)
  File "/home/user/Electrum-3.3.4/electrum/interface.py", line 289, in wrapper_func
    return await func(self, *args, **kwargs)
  File "/home/user/Electrum-3.3.4/electrum/interface.py", line 307, in run
    await self.open_session(ssl_context)
  File "/home/user/Electrum-3.3.4/electrum/interface.py", line 407, in open_session
    await group.spawn(self.monitor_connection)
  File "/home/user/Electrum-3.3.4/packages/aiorpcx/curio.py", line 240, in __aexit__
    await self.join()
  File "/home/user/Electrum-3.3.4/packages/aiorpcx/curio.py", line 207, in join
    raise task.exception()
  File "/home/user/Electrum-3.3.4/electrum/synchronizer.py", line 72, in _start_tasks
    await group.spawn(self.main())
  File "/home/user/Electrum-3.3.4/packages/aiorpcx/curio.py", line 240, in __aexit__
    await self.join()
  File "/home/user/Electrum-3.3.4/packages/aiorpcx/curio.py", line 207, in join
    raise task.exception()
  File "/home/user/Electrum-3.3.4/electrum/synchronizer.py", line 94, in subscribe_to_address
    await self.session.subscribe('blockchain.scripthash.subscribe', [h], self.status_queue)
  File "/home/user/Electrum-3.3.4/electrum/interface.py", line 122, in subscribe
    result = await self.send_request(method, params)
  File "/home/user/Electrum-3.3.4/electrum/interface.py", line 107, in send_request
    timeout)
  File "/usr/lib/python3.6/asyncio/tasks.py", line 358, in wait_for
    return fut.result()
  File "/home/user/Electrum-3.3.4/packages/aiorpcx/session.py", line 523, in send_request
    raise result
aiorpcx.jsonrpc.RPCError: (-102, 'server busy - request timed out')

[SomberNight]

I guess all resource usage over the .onion host is accounted against the private IP of the Tor gateway (e.g. localhost).

[peleion]

Unfortunately this seems to be the current state of ElectrumX - resource limits implemented to help counter the DDoS abuse tallies usage per discrete IP so once that limit is reached no further connections are allowed from that IP until server reboot i.e. stops serving Tor (Tails) clients. The server OP has the choice of a) disabling resource limits or b) disallowing Tor users after a short period of time, at least until something like #303, #790 or #805 is implemented although the first two would likely require a change on the client side.

[ghost]

@SomberNight is that because I also act as a Tor relay node?

@peleion

disabling resource limits

How can I do that (maybe just for testing)?

[SomberNight]

@sumBTC No, that is unrelated.

It's because recent commits introduced limiting of resource usage of sessions. Resource usage is totalled over IP address. The problem is that for the sessions connecting to your .onion host, ElectrumX sees the source IP as the local IP of your Tor process that runs the hidden service. So e.g. localhost in many cases.

This means that all users who use your server will increase the resource usage of that single IP, and then that IP will effectively end up banned.

@kyuupichan I think there could be an option to disable the IP buckets for onion only (for sessions where session.remote_address() == env.tor_proxy_host) -- or perhaps there could be a configurable whitelist of IPs

[ghost]

@SomberNight Thanks a lot for the explanation. As I understand it now, all Tor connections will be unable to connect eventually or Electrumx will be vulnerable to DDOS attacks over Tor. So this basically leaves whitelisting onion addresses as the only option.

[peleion]

@peleion

disabling resource limits

How can I do that (maybe just for testing)?

Environment variables:
COST_SOFT_LIMIT=0
BANDWIDTH_UNIT_COST=0
Should disable resource limiting

Environment variables documentation

[ghost]

@peleion So there is not 1 parameter to turn it on or off.

Setting BANDWIDTH_UNIT_COST=0 gives a division by zero error: .

:  session_class.bw_cost_per_byte = 1.0 / self.env.bw_unit_cost
:  ZeroDivisionError: float division by zero

I think that parameter should be really high to allow onion addresses (and also COST_SOFT_LIMIT, COST_HARD_LIMIT really high)

[SomberNight]

Try

COST_SOFT_LIMIT=0
COST_HARD_LIMIT=0

Don't set BANDWIDTH_UNIT_COST

[ghost]

@SomberNight High values work also but this is easier and till now everything seems fine.

EDIT: yep, this works. Thanks a lot to all.Not closing this issue myself because it's still a workaround in my opinion.

[ghost]

Setting soft and hard limit costs to zero works well enough to solve my issue. Closing.