ClientSession CancelledError in old versions

[argaen]

DISCLAIMER: This is related with older versions of aiohttp. 2.1.0 seems not to be affected but all previous are.

Hi, I've been playing a bit due to a bug we had in production and finally found out why. It's related with ClientSession timeouts and how we capture exceptions. This is a small snippet reproducing what we have:

import logging
import asyncio
import aiohttp
import concurrent


logger = logging.getLogger(__name__)


class Service:
    def __init__(self):
        self.session = aiohttp.ClientSession()

    async def do(self, timeout):
        async with self.session.get('https://api.github.com/events', timeout=timeout) as resp:
            logger.warning("%s, %s", timeout, resp.status)


async def view():
    service1 = Service()
    service2 = Service()
    try:
        await service1.do(0.01)
    except asyncio.TimeoutError:
        logger.exception("Caught TimeoutError")

    await service2.do(20)   # This in some occasions will raise concurrent.futures.CancelledError??????

loop = asyncio.get_event_loop()
loop.run_until_complete(view())

Now this is the output I receive:

Caught TimeoutError
Traceback (most recent call last):
  File "aiohttp_cancellation.py", line 23, in view
    await service1.do(0.01)
  File "aiohttp_cancellation.py", line 15, in do
    async with self.session.get('https://api.github.com/events', timeout=timeout) as resp:
  File "/home/blck/.pyenv/versions/aiocache/lib/python3.5/site-packages/aiohttp/client.py", line 593, in __aenter__
    self._resp = yield from self._coro
  File "/home/blck/.pyenv/versions/aiocache/lib/python3.5/site-packages/aiohttp/client.py", line 222, in _request
    yield from resp.start(conn, read_until_eof)
  File "/home/blck/.pyenv/versions/aiocache/lib/python3.5/site-packages/aiohttp/client_reqrep.py", line 622, in start
    self._continue = None
  File "/home/blck/.pyenv/versions/aiocache/lib/python3.5/site-packages/aiohttp/helpers.py", line 807, in __exit__
    raise asyncio.TimeoutError from None
concurrent.futures._base.TimeoutError
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7fb56bd53b38>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7fb56bd53e10>
Traceback (most recent call last):
  File "aiohttp_cancellation.py", line 30, in <module>
    loop.run_until_complete(view())
  File "/home/blck/.pyenv/versions/3.5.2/lib/python3.5/asyncio/base_events.py", line 387, in run_until_complete
    return future.result()
  File "/home/blck/.pyenv/versions/3.5.2/lib/python3.5/asyncio/futures.py", line 266, in result
    raise CancelledError
concurrent.futures._base.CancelledError

Note that the TimeoutError exception is being caught but apparently CancelledError is also raised in the second call??

I was able to reproduce this with the following versions of aiohttp:

• 1.3.4 (current version in PROD, I know its old and we should upgrade)
• 2.0.0
• 2.0.1
• 2.0.5
• 2.0.7

Now, I couldn't reproduce this in 2.1.0 but going through the release notes, I haven't seen anything related. I'm trying to understand if this was fixed somewhere or was just fixed as a side effect of another issue? or maybe the issue is still there but much harder to reproduce?

Thanks

[argaen]

New script that will iterate until a CancelledError is raised. Useful to check which versions are affected:

import sys
import logging
import asyncio
import aiohttp
import concurrent


logger = logging.getLogger(__name__)


class Service:
    def __init__(self):
        self.session = aiohttp.ClientSession()

    async def do(self, timeout):
        async with self.session.get('https://api.github.com/events', timeout=timeout) as resp:
            logger.warning("%s, %s", timeout, resp.status)


async def view():
    service1 = Service()
    service2 = Service()
    try:
        await service1.do(0.0000001)
    except asyncio.TimeoutError:
        logger.exception("Caught TimeoutError")

    try:
        await service2.do(20)
    except concurrent.futures.CancelledError:
        logger.exception("GOT CANCELLED")
        sys.exit(0)


async def iterate_view():
    for x in range(100):
        await view()

loop = asyncio.get_event_loop()
loop.run_until_complete(iterate_view())

[fafhrd91]

this was bug in old versions

[fafhrd91]

I can find commit at the moment

[argaen]

If you remember the issue or the commit let me know. Good to know it was identified and fixed. Thanks!

[fafhrd91]

There was issue, but I can not find it. Will look later

[fafhrd91]

#1879

[argaen]

Amazing, thanks a lot!

[lock]

This thread has been automatically locked since there has not been
any recent activity after it was closed. Please open a new issue for
related bugs.

If you feel like there's important points made in this discussion,
please include those exceprts into that new issue.