Event subscription handler hangs after a while

[snarfed]

At least a few hours. Not sure why, probably websocket subscriptions something or other.

[snarfed]

Couldn't find anything relevant in logs, surprisingly. Still don't know what the root cause here is, but I may experiment with multiple workers vs eventlet vs gevent, etc: https://flask-sock.readthedocs.io/en/latest/web_servers.html

[snarfed]

Looking at this a bit more now. Here's an example from today, two things happened:

1. websocket request to /xrpc/com.atproto.sync.subscribeRepos that succeeded, served for 1h+10s, then returned HTTP 101 continue
2. another websocket request to /xrpc/com.atproto.sync.subscribeRepos that did nothing, hung for 1h, then returned HTTP 504

After that, the Bluesky relay kept making subscribeRepos requests that did the same thing as in snarfed/arroba#2.

I suspect we're not handling something in the initial HTTP 101 code path, and that's killing the serving thread or something similar.

2024-03-06 06:51:48.496 - 07:51:58.540
GET 101 /xrpc/com.atproto.sync.subscribeRepos?cursor=1864

New websocket client for com.atproto.sync.subscribeRepos
com.atproto.sync.subscribeRepos: {'cursor': 1864} None
Running method
subscribeRepos: fetching existing commits from seq 1864
Sending to com.atproto.sync.subscribeRepos websocket client: {'op': 1, 't': '#commit'} {'repo': 'did:plc:sp6aj7mvt7dw26xp6wbvbv6i', 'ops': [{'action': 'update', 'path': 'app.bsky.actor.profile/self', 'cid': CID('base32', 1, 'dag-cbor', '12202e0b1c21d9f0ebeffda410546889e3f4d22d91d2baef7996c9e920cb4c1f2340')}], 'commit': CID('base32', 1, 'dag-cbor', '1220b1ccee20ec7e5ef7aa55752834791e7214bdc7e9ed91ea26351bf72700fb6188'), 'blocks': b":\xa2eroots\x81\xd8*X%\x00\x01q\x12 \xb1\xcc\xee \xec~^\xf7\xaaUu(4y\x1er\x14\xbd\xc7\xe9\xed\x91\xea&5\x1b\xf7'\x00\xfba\x88gversion\x01\x88\x02\x01q\x...
subscribeRepos: serving new commits

2024-03-06 07:51:58.945 - 08:51:58.946
GET 504 /xrpc/com.atproto.sync.subscribeRepos?cursor=1864

[snarfed]

miguelgrinberg/flask-sock#7 ?

[snarfed]

I suspect we're not handling something in the initial HTTP 101 code path, and that's killing the serving thread or something similar.

Looking more and more likely. Looking at logs from the most recent time this happened, we served 20 101s and then started serving 504s, and hub.yaml runs gunicorn with --threads 20.

[snarfed]

Filed miguelgrinberg/flask-sock#78

[snarfed]

snarfed/arroba@9e1a356

[snarfed]

Deployed. If it stays up past ~1:30p PT tomorrow, I'll declare victory.

[snarfed]

Confirmed fixed!