Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

/api/tasks/TASK_ID/state gives 500 #17878

Closed
bernt-matthias opened this issue Apr 2, 2024 · 1 comment · Fixed by #17949
Closed

/api/tasks/TASK_ID/state gives 500 #17878

bernt-matthias opened this issue Apr 2, 2024 · 1 comment · Fixed by #17949

Comments

@bernt-matthias
Copy link
Contributor

Describe the bug

Just tried https://github.com/galaxyproject/gxyarchiver and got 2 problems

  • No export is created (I also tried via web UI and it works)
  • Got a 500 and a long traceback in the logs:
uvicorn.access INFO 2024-04-02 13:22:27,620 [pN:main.2,p:4131762,tN:MainThread] 141.65.254.225:0 - "POST /api/histories/86e50fc04e8f1350/write_store HTTP/1.0" 200
uvicorn.access INFO 2024-04-02 13:22:28,344 [pN:main.2,p:4131762,tN:MainThread] 141.65.254.225:0 - "GET /history/current_history_json?since=2024-04-02T10:15:29.724974 HTTP/1.0" 200
uvicorn.access INFO 2024-04-02 13:22:28,874 [pN:main.1,p:4131758,tN:MainThread] 141.65.254.225:0 - "GET /api/tasks/8b720767-eb55-4df8-8a8e-ea20e2912b4d/state HTTP/1.0" 500
[2024-04-02 13:22:28 +0200] [4131758] [ERROR] Exception in ASGI application
Traceback (most recent call last):
  File "/gpfs1/data/galaxy_server/galaxy-dev/.venv/lib/python3.11/site-packages/anyio/streams/memory.py", line 98, in receive
    return self.receive_nowait()
           ^^^^^^^^^^^^^^^^^^^^^
  File "/gpfs1/data/galaxy_server/galaxy-dev/.venv/lib/python3.11/site-packages/anyio/streams/memory.py", line 93, in receive_nowait
    raise WouldBlock
anyio.WouldBlock

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/gpfs1/data/galaxy_server/galaxy-dev/.venv/lib/python3.11/site-packages/starlette/middleware/base.py", line 78, in call_next
    message = await recv_stream.receive()
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfs1/data/galaxy_server/galaxy-dev/.venv/lib/python3.11/site-packages/anyio/streams/memory.py", line 118, in receive
    raise EndOfStream
anyio.EndOfStream

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/gpfs1/data/galaxy_server/galaxy-dev/.venv/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 428, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfs1/data/galaxy_server/galaxy-dev/.venv/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 78, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfs1/data/galaxy_server/galaxy-dev/.venv/lib/python3.11/site-packages/fastapi/applications.py", line 284, in __call__
    await super().__call__(scope, receive, send)
  File "/gpfs1/data/galaxy_server/galaxy-dev/.venv/lib/python3.11/site-packages/starlette/applications.py", line 122, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/gpfs1/data/galaxy_server/galaxy-dev/.venv/lib/python3.11/site-packages/starlette/middleware/errors.py", line 184, in __call__
    raise exc
  File "/gpfs1/data/galaxy_server/galaxy-dev/.venv/lib/python3.11/site-packages/starlette/middleware/errors.py", line 162, in __call__
    await self.app(scope, receive, _send)
  File "/gpfs1/data/galaxy_server/galaxy-dev/.venv/lib/python3.11/site-packages/starlette_context/middleware/raw_middleware.py", line 93, in __call__
    await self.app(scope, receive, send_wrapper)
  File "/gpfs1/data/galaxy_server/galaxy-dev/.venv/lib/python3.11/site-packages/starlette/middleware/base.py", line 108, in __call__
    response = await self.dispatch_func(request, call_next)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfs1/data/galaxy_server/galaxy-dev/lib/galaxy/webapps/galaxy/fast_app.py", line 107, in add_x_frame_options
    response = await call_next(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfs1/data/galaxy_server/galaxy-dev/.venv/lib/python3.11/site-packages/starlette/middleware/base.py", line 84, in call_next
    raise app_exc
  File "/gpfs1/data/galaxy_server/galaxy-dev/.venv/lib/python3.11/site-packages/starlette/middleware/base.py", line 70, in coro
    await self.app(scope, receive_or_disconnect, send_no_error)
  File "/gpfs1/data/galaxy_server/galaxy-dev/.venv/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 79, in __call__
    raise exc
  File "/gpfs1/data/galaxy_server/galaxy-dev/.venv/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 68, in __call__
    await self.app(scope, receive, sender)
  File "/gpfs1/data/galaxy_server/galaxy-dev/.venv/lib/python3.11/site-packages/fastapi/middleware/asyncexitstack.py", line 20, in __call__
    raise e
  File "/gpfs1/data/galaxy_server/galaxy-dev/.venv/lib/python3.11/site-packages/fastapi/middleware/asyncexitstack.py", line 17, in __call__
    await self.app(scope, receive, send)
  File "/gpfs1/data/galaxy_server/galaxy-dev/.venv/lib/python3.11/site-packages/starlette/routing.py", line 718, in __call__
    await route.handle(scope, receive, send)
  File "/gpfs1/data/galaxy_server/galaxy-dev/.venv/lib/python3.11/site-packages/starlette/routing.py", line 276, in handle
    await self.app(scope, receive, send)
  File "/gpfs1/data/galaxy_server/galaxy-dev/.venv/lib/python3.11/site-packages/starlette/routing.py", line 66, in app
    response = await func(request)
               ^^^^^^^^^^^^^^^^^^^
  File "/gpfs1/data/galaxy_server/galaxy-dev/.venv/lib/python3.11/site-packages/fastapi/routing.py", line 241, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfs1/data/galaxy_server/galaxy-dev/.venv/lib/python3.11/site-packages/fastapi/routing.py", line 169, in run_endpoint_function
    return await run_in_threadpool(dependant.call, **values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfs1/data/galaxy_server/galaxy-dev/.venv/lib/python3.11/site-packages/starlette/concurrency.py", line 41, in run_in_threadpool
    return await anyio.to_thread.run_sync(func, *args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfs1/data/galaxy_server/galaxy-dev/.venv/lib/python3.11/site-packages/anyio/to_thread.py", line 33, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfs1/data/galaxy_server/galaxy-dev/.venv/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 877, in run_sync_in_worker_thread
    return await future
           ^^^^^^^^^^^^
  File "/gpfs1/data/galaxy_server/galaxy-dev/.venv/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 807, in run
    result = context.run(func, *args)
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfs1/data/galaxy_server/galaxy-dev/lib/galaxy/webapps/galaxy/api/tasks.py", line 31, in state
    return self.manager.get_state(task_id)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfs1/data/galaxy_server/galaxy-dev/lib/galaxy/managers/tasks.py", line 76, in get_state
    return cast(TaskState, str(self._get_result(task_uuid).state))
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfs1/data/galaxy_server/galaxy-dev/.venv/lib/python3.11/site-packages/celery/result.py", line 503, in state
    return self._get_task_meta()['status']
           ^^^^^^^^^^^^^^^^^^^^^
  File "/gpfs1/data/galaxy_server/galaxy-dev/.venv/lib/python3.11/site-packages/celery/result.py", line 442, in _get_task_meta
    return self._maybe_set_cache(self.backend.get_task_meta(self.id))
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfs1/data/galaxy_server/galaxy-dev/.venv/lib/python3.11/site-packages/celery/backends/base.py", line 608, in get_task_meta
    meta = self._get_task_meta_for(task_id)
           ^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'DisabledBackend' object has no attribute '_get_task_meta_for'

I have no celery setup on my Galaxy. Or better: I have not changed the default setup (I have a celery process but no idea what it is doing).

Galaxy Version and/or server at which you observed the bug
Galaxy Version: 23.2
Commit: a8727a3

To Reproduce
Steps to reproduce the behavior:

Not sure yet. So far I just followed the

Expected behavior

No error :) .. or more useful error message.
@davelopez
Copy link
Contributor

We briefly discuss this in the backend meeting. This is the expected behavior because gxyarchiver relies on Celery tasks to check when the export has finished.

We will try to document better how to set Celery in #17879

We can also try to catch this error and log a more informative message telling you to configure the Celery backend explicitly.

@davelopez davelopez mentioned this issue Apr 9, 2024
Merged
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Replace sample Celery result_backend in config davelopez/galaxy
2 participants
@bernt-matthias @davelopez