Lots of CancelledError when requests are interupted #1565

samuelcolvin · 2017-01-30T12:31:25Z

Long story short

aiohttp barfs out hundreds of concurrent.futures._base.CancelledError when requests are interrupted.

Expected behaviour

Not sure, I guess we can't just ignore CancelledError in requests but it would be good if there was some way to stop loads of errors being printed.

This is annoying in the terminal but could cause a major backlog if sentry etc. were connected and processing exceptions.

You can catch and ignore this error with middleware:

async def cancelled_error_middleware(app, handler):
    async def _handler(request):
        try:
            return await handler(request)
        except CancelledError:
            raise HTTPBadRequest()
    return _handler

But I'm not sure how safe this is?

Actual behaviour

[2017-01-30 12:22:19 +0000] [30723] [ERROR] Error handling request
Traceback (most recent call last):
  File "/home/vagrant/FrameworkBenchmarks/installs/py3/lib/python3.5/site-packages/aiohttp/web_server.py", line 61, in handle_request
    resp = yield from self._handler(request)
  File "/home/vagrant/FrameworkBenchmarks/installs/py3/lib/python3.5/site-packages/aiohttp/web.py", line 249, in _handle
    resp = yield from handler(request)
  File "/home/vagrant/FrameworkBenchmarks/frameworks/Python/aiohttp/app/views.py", line 142, in updates
    .values(randomnumber=randint(1, 10000))
  File "/home/vagrant/FrameworkBenchmarks/installs/py3/lib/python3.5/site-packages/aiopg/utils.py", line 72, in __await__
    resp = yield from self._coro
  File "/home/vagrant/FrameworkBenchmarks/installs/py3/lib/python3.5/site-packages/aiopg/sa/connection.py", line 110, in _execute
    yield from cursor.execute(str(compiled), post_processed_params[0])
  File "/home/vagrant/FrameworkBenchmarks/installs/py3/lib/python3.5/site-packages/aiopg/cursor.py", line 113, in execute
    yield from self._conn._poll(waiter, timeout)
  File "/home/vagrant/FrameworkBenchmarks/installs/py3/lib/python3.5/site-packages/aiopg/connection.py", line 240, in _poll
    raise exc
  File "/home/vagrant/FrameworkBenchmarks/installs/py3/lib/python3.5/site-packages/aiopg/connection.py", line 237, in _poll
    yield from asyncio.wait_for(self._waiter, timeout, loop=self._loop)
  File "/home/vagrant/FrameworkBenchmarks/installs/py3/lib/python3.5/asyncio/tasks.py", line 379, in wait_for
    yield from waiter
  File "/home/vagrant/FrameworkBenchmarks/installs/py3/lib/python3.5/asyncio/futures.py", line 358, in __iter__
    yield self  # This tells Task to wait for completion.
  File "uvloop/future.pyx", line 434, in uvloop.loop.BaseTask._fast_wakeup (uvloop/loop.c:114015)
  File "/home/vagrant/FrameworkBenchmarks/installs/py3/lib/python3.5/asyncio/futures.py", line 266, in result
    raise CancelledError
concurrent.futures._base.CancelledError

Coming through thousands of times as I run wrk load tests. I suspect the same would occur with ajax interrupted requests or even browsers stopping a request if a new one is requested.

Steps to reproduce

wrk -t12 -c100 -d10s http://localhost:8000/whatever

While running your aiohttp application with gunicorn.

Your environment

Ubuntu.

The text was updated successfully, but these errors were encountered:

fafhrd91 · 2017-01-30T17:20:11Z

here is code that handle cancelation https://github.com/KeepSafe/aiohttp/blob/master/aiohttp/server.py#L292

could you confirm same behavior with default event loop?

samuelcolvin · 2017-01-30T17:42:29Z

I can confirm it's happening with the standard event loop:

[2017-01-30 17:39:13 +0000] [28963] [INFO] Starting gunicorn 19.6.0
[2017-01-30 17:39:13 +0000] [28963] [INFO] Listening at: http://127.0.0.1:8000 (28963)
[2017-01-30 17:39:13 +0000] [28963] [INFO] Using worker: aiohttp.worker.GunicornWebWorker
[2017-01-30 17:39:13 +0000] [28972] [INFO] Booting worker with pid: 28972
[2017-01-30 17:39:21 +0000] [28972] [ERROR] Error handling request
Traceback (most recent call last):
  File "/path/to/code/env/lib/python3.6/site-packages/aiohttp/web_server.py", line 61, in handle_request
    resp = yield from self._handler(request)
  File "/path/to/code/env/lib/python3.6/site-packages/aiohttp/web.py", line 249, in _handle
    resp = yield from handler(request)
  File "/path/to/code/app/middleware.py", line 77, in _handler
    return await handler(request)
  File "/path/to/code/app/middleware.py", line 90, in _handler
    conn = await request['conn_manager'].get_connection()
  File "/path/to/code/app/middleware.py", line 69, in get_connection
    self._conn = await self._engine._acquire()
  File "/path/to/code/env/lib/python3.6/site-packages/aiopg/sa/engine.py", line 161, in _acquire
    raw = yield from self._pool.acquire()
  File "/path/to/code/env/lib/python3.6/site-packages/aiopg/utils.py", line 67, in __iter__
    resp = yield from self._coro
  File "/path/to/code/env/lib/python3.6/site-packages/aiopg/pool.py", line 168, in _acquire
    with (yield from self._cond):
  File "/usr/lib/python3.6/asyncio/locks.py", line 67, in __iter__
    yield from self.acquire()
  File "/usr/lib/python3.6/asyncio/locks.py", line 176, in acquire
    yield from fut
concurrent.futures._base.CancelledError

I got this on a different project as running the benchmark tests is a bit involved but it's the same problem.

Seems to always relate to aiopg, but I don't know if that's significant.

fafhrd91 · 2017-01-30T17:43:43Z

i see the problem

samuelcolvin · 2017-01-30T17:44:41Z

Looking at it, the second case is slightly different as the connection is being acquired in middleware. Is that significant?

fafhrd91 · 2017-01-30T17:45:48Z

it is related to web_server.Server implementation, it was introduced in 1.2.0 version. it just swallows exceptions

fafhrd91 · 2017-01-30T18:20:21Z

should be fixed in master

samuelcolvin · 2017-02-04T15:24:20Z

This is indeed fixed in master. Thanks a lot, it would be great if there was a new release so I could use it on FrameworkBenchmarks currently every time a test is run aiohttp is printing about 10k lines of errors.

fafhrd91 · 2017-02-04T20:04:35Z

I will make new release early next week. I need to add timeouts to client close

AraHaan · 2017-02-04T21:00:16Z

@fafhrd91 Will it not conflict with #1586? because that would also be something to merge before the release as well as it fixes an issue with the warnings about initializing client session outside of an coroutine with an event loop. Because the original one was bugged because some things was flipped around.

niteeshm · 2017-03-06T13:20:04Z

Even with aiohttp 1.3.2 I am facing the same issue , if a request gets cancelled I get tonnes of errors in my error log saying TypeError: CancelledError() is not JSON serializable

samuelcolvin · 2017-03-06T13:23:26Z

This sounds like a different issue. aiohttp is correctly dealing with CancelledError, this is an error with your application code.

You'll need to do something like (in your view handler)

try:
    obj = await get_the_data()
except CancelledError:
    raise  # aiohttp will cope with error
else:
    return json_response(obj)

niteeshm · 2017-03-07T06:43:02Z

@samuelcolvin Even after adding the above handler it still gives me a bunch of CancelledError

Added this to my handler :

        except CancelledError:
            raise
        except Exception as e:
            return self._send_response({'status': {'type': 'ERROR', 'message': e}, 'data': {}, 'code': 500})

ERROR 2017-03-07 11:36:18,325 Module:web_server --- Error handling request
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/site-packages/aiohttp/web_server.py", line 61, in handle_request
    resp = yield from self._handler(request)
  File "/usr/local/lib/python3.5/site-packages/aiohttp/web.py", line 249, in _handle
    resp = yield from handler(request)
  File "/usr/local/lib/python3.5/site-packages/aiohttp/web_urldispatcher.py", line 113, in handler_wrapper
    result = yield from result
  File "/Users/6332/git/sizing_service/service/app/handler/size_handler.py", line 84, in get_recommended_size
    return_percentage = await self.redis_get('return_percentage:' + brand)
  File "/Users/6332/git/myservice/service/app/handler.py", line 20, in redis_get
    return await self.redis_conn.get(key)
  File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/asyncio/futures.py", line 358, in __iter__
    yield self  # This tells Task to wait for completion.
  File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/asyncio/tasks.py", line 290, in _wakeup
    future.result()
  File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/asyncio/futures.py", line 266, in result
    raise CancelledError
concurrent.futures._base.CancelledError

niteeshm · 2017-03-10T07:33:29Z

Yeah can confirm this has still not been fixed.

samuelcolvin · 2017-03-10T15:38:53Z

I'm pretty sure the issue I originally submitted the issue about has been fixed.

But that doesn't mean you can't still have issues in your application code with CancelledError.

However, I could be wrong. Please provide a simple example I can run which demonstrates the issue, eg. via a gist.

fafhrd91 · 2017-03-14T18:32:46Z

@niteeshm what version do you use? Did you try master?

niteeshm · 2017-03-20T08:05:48Z

@fafhrd91 @samuelcolvin I tried this on 1.3.3. I am using wrk to send requests to my server and it gives this CancelledError for a bunch of the requests when wrk benchmark completes.

samuelcolvin · 2017-03-20T10:01:27Z

As i said before you can still get cancellederror from upstream connections, eg. Database. This is not aiohttp at fault.

Please provide an example yo demonstrate the problem.

fafhrd91 · 2017-03-22T17:23:01Z

@niteeshm please provide an example, will reopen then

lock · 2019-10-28T22:02:49Z

This thread has been automatically locked since there has not been
any recent activity after it was closed. Please open a new issue for
related bugs.

If you feel like there's important points made in this discussion,
please include those exceprts into that new issue.

fafhrd91 pushed a commit that referenced this issue Jan 30, 2017

Lots of CancelledError when requests are interupted #1565

7ab213b

fafhrd91 closed this as completed Jan 30, 2017

samuelcolvin mentioned this issue Feb 9, 2017

upgrading aiohttp and correcting README TechEmpower/FrameworkBenchmarks#2561

Merged

fafhrd91 reopened this Mar 14, 2017

fafhrd91 closed this as completed Mar 22, 2017

lock bot added the outdated label Oct 28, 2019

lock bot locked as resolved and limited conversation to collaborators Oct 28, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Lots of CancelledError when requests are interupted #1565

Lots of CancelledError when requests are interupted #1565

samuelcolvin commented Jan 30, 2017

fafhrd91 commented Jan 30, 2017

samuelcolvin commented Jan 30, 2017

fafhrd91 commented Jan 30, 2017

samuelcolvin commented Jan 30, 2017

fafhrd91 commented Jan 30, 2017

fafhrd91 commented Jan 30, 2017

samuelcolvin commented Feb 4, 2017

fafhrd91 commented Feb 4, 2017

AraHaan commented Feb 4, 2017 •

edited

Loading

niteeshm commented Mar 6, 2017

samuelcolvin commented Mar 6, 2017

niteeshm commented Mar 7, 2017

niteeshm commented Mar 10, 2017

samuelcolvin commented Mar 10, 2017

fafhrd91 commented Mar 14, 2017

niteeshm commented Mar 20, 2017

samuelcolvin commented Mar 20, 2017

fafhrd91 commented Mar 22, 2017

lock bot commented Oct 28, 2019

Lots of CancelledError when requests are interupted #1565

Lots of CancelledError when requests are interupted #1565

Comments

samuelcolvin commented Jan 30, 2017

Long story short

Expected behaviour

Actual behaviour

Steps to reproduce

Your environment

fafhrd91 commented Jan 30, 2017

samuelcolvin commented Jan 30, 2017

fafhrd91 commented Jan 30, 2017

samuelcolvin commented Jan 30, 2017

fafhrd91 commented Jan 30, 2017

fafhrd91 commented Jan 30, 2017

samuelcolvin commented Feb 4, 2017

fafhrd91 commented Feb 4, 2017

AraHaan commented Feb 4, 2017 • edited Loading

niteeshm commented Mar 6, 2017

samuelcolvin commented Mar 6, 2017

niteeshm commented Mar 7, 2017

niteeshm commented Mar 10, 2017

samuelcolvin commented Mar 10, 2017

fafhrd91 commented Mar 14, 2017

niteeshm commented Mar 20, 2017

samuelcolvin commented Mar 20, 2017

fafhrd91 commented Mar 22, 2017

lock bot commented Oct 28, 2019

AraHaan commented Feb 4, 2017 •

edited

Loading