Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GeneratorExit in version 20.6.2 #1525

Closed
atemate opened this issue Jun 2, 2020 · 5 comments
Closed

GeneratorExit in version 20.6.2 #1525

atemate opened this issue Jun 2, 2020 · 5 comments
Assignees
Labels
bug Something isn't working

Comments

@atemate
Copy link
Contributor

atemate commented Jun 2, 2020

Task exception was never retrieved
future: <Task finished coro=<<async_generator_athrow without __name__>()> exception=RuntimeError("generator didn't stop after throw()")>
Traceback (most recent call last):
  File "/home/circleci/.local/lib/python3.7/site-packages/neuromation/api/core.py", line 170, in request
    yield resp
  File "/home/circleci/.local/lib/python3.7/site-packages/neuromation/api/jobs.py", line 227, in list
    yield _job_description_from_api(j, self._parse)
GeneratorExit
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/home/circleci/.local/lib/python3.7/site-packages/neuromation/api/jobs.py", line 231, in list
    yield _job_description_from_api(j, self._parse)
  File "/usr/local/lib/python3.7/contextlib.py", line 189, in __aexit__
    raise RuntimeError("generator didn't stop after throw()")
RuntimeError: generator didn't stop after throw()

see https://circleci.com/gh/neuromation/ml-recipe-object-detection/38

@atemate
Copy link
Contributor Author

atemate commented Jun 2, 2020

Finally got this with my own neuro client, not the one running on CircleCI:

$ n status airflow-inference
curl ERROR: Task exception was never retrieved
future: <Task finished coro=<<async_generator_athrow without __name__>()> exception=RuntimeError("generator didn't stop after throw()")>
Traceback (most recent call last):
  File "/home/ay/.pyenv/versions/3.7.6/lib/python3.7/site-packages/neuromation/api/core.py", line 170, in request
    yield resp
  File "/home/ay/.pyenv/versions/3.7.6/lib/python3.7/site-packages/neuromation/api/jobs.py", line 227, in list
    yield _job_description_from_api(j, self._parse)
GeneratorExit

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ay/.pyenv/versions/3.7.6/lib/python3.7/site-packages/neuromation/api/jobs.py", line 231, in list
    yield _job_description_from_api(j, self._parse)
  File "/home/ay/.pyenv/versions/3.7.6/lib/python3.7/contextlib.py", line 189, in __aexit__
    raise RuntimeError("generator didn't stop after throw()")
RuntimeError: generator didn't stop after throw()
Job: job-bf7c07ce-e064-4871-b236-4b8b346bed66
Name: airflow-inference
Tags: kind:airflow, preset:gpu, step:inference
Owner: artemyushkovskiy
Cluster: onprem-poc
Status: running
Image: image:project-zero:latest
...

cluster onprem-poc this time.

@serhiy-storchaka serhiy-storchaka added the bug Something isn't working label Jun 2, 2020
@serhiy-storchaka serhiy-storchaka self-assigned this Jun 2, 2020
@shagren
Copy link
Contributor

shagren commented Jun 10, 2020

Repeated for debian 10 with system python(3.7.3) and neuro installed by pip3 install neuromation

@waytobehigh
Copy link

Reproduced on Mac OS with the latest neuro-cli update

@serhiy-storchaka
Copy link
Contributor

It has been fixed by @asvetlov in #1497.
https://github.com/neuromation/platform-client-python/blob/dfe74a9614732a02ffdd1a8652db37dedc23e94a/neuromation/api/core.py#L171-L178

We do not know the root cause yet, but at least it is no longer manifested to users. Just wait the next release.

Would be nice to fix the root cause, but it is not critical now.

@asvetlov
Copy link
Contributor

The root case is described (and fixed) by https://bugs.python.org/issue33786

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

5 participants