Network adapter going down does not disconnect the websockets

[Insoleet]

Long story short

I'm working on a software which uses aiohttp websockets to connect to multiple network nodes. There is a bug with aiohttp and its websockets. When the users are disconnecting from wifi, or set their computers on standby and waking their computers later on, the software does not detect the disconnection.

Expected behaviour

When the network adapter is disconnected, the websockets should raise an exception and close themselves.

Actual behaviour

The websockets do not close.

Steps to reproduce

1- When connected to the internet, run the following code :

import aiohttp
import asyncio

async def connect():
    try:
        async with aiohttp.ws_connect('ws://metab.ucoin.io:9201/ws/block') as the_ws:
            async for msg in the_ws:
                if msg.tp == aiohttp.MsgType.text:
                    print(msg.data)
                elif msg.tp == aiohttp.MsgType.closed:
                    print("Closed")
                    break
                elif msg.tp == aiohttp.MsgType.error:
                    print("Error")
                    break
    except Exception as e:
        print(str(e))
    finally:
        print("Disconnected")

loop = asyncio.get_event_loop()
loop.run_until_complete(connect())
print("Finished")

2- It will print json data from an ucoin blockchain. Disconnect your internet
3- It never prints the exception, neither "Disconnected" nor "Finished" messages.

Your environment

I'm using python3 with archlinux 64 bits.

[asvetlov]

Thanks for report.

[asvetlov]

The patch is welcome :)

[Insoleet]

So I looked at it all this afternoon, but I'm not sure how to fix this bug.

• It seems to come from yield from reader.read() https://github.com/KeepSafe/aiohttp/blob/master/aiohttp/websocket_client.py#L144 not raising anything.
• It should raise when an exception is set on the DataQueue ( https://github.com/KeepSafe/aiohttp/blob/master/aiohttp/streams.py#L410 ). But this never happens.
• This happens if the StreamParser is being set an exception ( https://github.com/KeepSafe/aiohttp/blob/master/aiohttp/parsers.py#L119 ), but once again, this never happens.
• This happens if the StreamProtocol callback method connection_lost is being called ( https://github.com/KeepSafe/aiohttp/blob/master/aiohttp/parsers.py#L306 ), but it is never called.

This is where I'm stuck : I'm not sure what to do now. The asyncio documentation describes that

BaseProtocol.connection_lost(exc)
Called when the connection is lost or closed.
The argument is either an exception object or None. The latter means a regular EOF is received, or the connection was aborted or closed by this side of the connection.

So I'm not sure why it is never called. Any help would be greatly appreciated !

[fafhrd91]

If connection_lost is not being called aiohttp can not do anything. But that might be buggy device driver

[fafhrd91]

also you can put some print statements into asyncio event loop and see if asyncio received info from kernel for specific socket.

[Insoleet]

Thanks for the info. I'm just wondering, can you reproduce the error I described in the first post of this issue ? Just to check if it's a bug happening on some platforms only.

Where should I put trace logging in asyncio loop ?

[Insoleet]

After reading this : http://superuser.com/questions/1021988/connection-remains-flagged-as-established-even-if-host-is-unconnected
I supposed that enabling autoping would help detecting the lost connection. But autoping does not send ping automatically. Is this a bug or a feature ?

[Insoleet]

Ok I found a way to detect disconnections by using a heartbeat mecanism :

    async def heartbeat(ws):
        """

        :param asyncio.WebsocketClientResponse ws: the ws to do heartbeat with
        :return:
        """
        future_pong = None

        def pong_callback():
            if future_pong:
                future_pong.set_result(True)

        async def ping_loop():
            nonlocal future_pong
            future_pong = asyncio.Future()
            missed_heartbeats = 0
            while not ws.closed:
                await asyncio.sleep(15)
                if not ws.closed:
                    try:
                        future_pong = asyncio.Future()
                        ws.ping()
                        await asyncio.wait_for(future_pong, 15)
                        missed_heartbeats = 0
                    except asyncio.TimeoutError:
                        missed_heartbeats += 1
                    finally:
                        if missed_heartbeats > 3:
                            await ws.close()

        asyncio.ensure_future(ping_loop())
        return pong_callback

I use it this way :

import aiohttp
import asyncio

async def connect():
    try:
        async with aiohttp.ws_connect('ws://metab.ucoin.io:9201/ws/block') as the_ws:
            heartbeat_cb = heartbeat(the_ws)
            async for msg in the_ws:
                if msg.tp == aiohttp.MsgType.text:
                    print(msg.data)
                elif msg.tp == aiohttp.MsgType.closed:
                    print("Closed")
                    break
                elif msg.tp == aiohttp.MsgType.error:
                    print("Error")
                    break
                elif msg.tp == aiohttp.MsgType.pong:
                    heartbeat_cb()
    except Exception as e:
        print(str(e))
    finally:
        print("Disconnected")

loop = asyncio.get_event_loop()
loop.run_until_complete(connect())
print("Finished")

I suppose it could be used for the following issue : #777

What do you think ?

[asvetlov]

Heartbeat should be implemented by aiohttp.
Unfortunately not all servers support ping/pong game :(

[fafhrd91]

added auto ping_interval parameter for client and server, should solve this problem.

[lock]

This thread has been automatically locked since there has not been
any recent activity after it was closed. Please open a new issue for
related bugs.

If you feel like there's important points made in this discussion,
please include those exceprts into that new issue.