ERROR: Received fragmented control frame when accessing job logs via neuro logs

[andriihomiak]

Description:

Sometimes, when trying to access logs via neuro logs, the logs are truncated by the following message:

$ neuro logs job-f08dd883-5675-4b92-9e37-9b9ea544c909
<OMITTED>
ERROR: Received fragmented control frame
Full logs are available under /home/andrii/.neuro/logs/neuro-run-2022-07-15_11-24-54.txt

Full logs show the following traceback:

<OMITTED>
Traceback (most recent call last):
  File "/home/andrii/.local/pipx/venvs/neuro-all/lib/python3.8/site-packages/neuro_cli/main.py", line 557, in main
    cli.main(
  File "/home/andrii/.local/pipx/venvs/neuro-all/lib/python3.8/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/home/andrii/.local/pipx/venvs/neuro-all/lib/python3.8/site-packages/neuro_cli/utils.py", line 332, in invoke
    super().invoke(ctx)
  File "/home/andrii/.local/pipx/venvs/neuro-all/lib/python3.8/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/andrii/.local/pipx/venvs/neuro-all/lib/python3.8/site-packages/neuro_cli/utils.py", line 298, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/andrii/.local/pipx/venvs/neuro-all/lib/python3.8/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/home/andrii/.local/pipx/venvs/neuro-all/lib/python3.8/site-packages/click/decorators.py", line 38, in new_func
    return f(get_current_context().obj, *args, **kwargs)
  File "/home/andrii/.local/pipx/venvs/neuro-all/lib/python3.8/site-packages/neuro_cli/utils.py", line 110, in wrapper
    return root.run(
  File "/home/andrii/.local/pipx/venvs/neuro-all/lib/python3.8/site-packages/neuro_cli/root.py", line 148, in run
    return self._runner.run(main)
  File "/home/andrii/.local/pipx/venvs/neuro-all/lib/python3.8/site-packages/neuro_cli/asyncio_utils.py", line 53, in run
    return self._loop.run_until_complete(main_task)
  File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/home/andrii/.local/pipx/venvs/neuro-all/lib/python3.8/site-packages/neuro_cli/utils.py", line 82, in _run_async_function
    return await func(root, *args, **kwargs)
  File "/home/andrii/.local/pipx/venvs/neuro-all/lib/python3.8/site-packages/neuro_cli/job.py", line 229, in logs
    await process_logs(
  File "/home/andrii/.local/pipx/venvs/neuro-all/lib/python3.8/site-packages/neuro_cli/ael.py", line 102, in process_logs
    async for chunk in it:
  File "/home/andrii/.local/pipx/venvs/neuro-all/lib/python3.8/site-packages/neuro_sdk/_jobs.py", line 577, in monitor
    raise ws.exception()  # type: ignore
  File "/home/andrii/.local/pipx/venvs/neuro-all/lib/python3.8/site-packages/aiohttp/client_ws.py", line 196, in close
    msg = await self._reader.read()
  File "/home/andrii/.local/pipx/venvs/neuro-all/lib/python3.8/site-packages/aiohttp/streams.py", line 657, in read
    return await super().read()
  File "/home/andrii/.local/pipx/venvs/neuro-all/lib/python3.8/site-packages/aiohttp/streams.py", line 627, in read
    raise self._exception
  File "/home/andrii/.local/pipx/venvs/neuro-all/lib/python3.8/site-packages/aiohttp/client_ws.py", line 229, in receive
    msg = await self._reader.read()
  File "/home/andrii/.local/pipx/venvs/neuro-all/lib/python3.8/site-packages/aiohttp/streams.py", line 657, in read
    return await super().read()
  File "/home/andrii/.local/pipx/venvs/neuro-all/lib/python3.8/site-packages/aiohttp/streams.py", line 627, in read
    raise self._exception
  File "/home/andrii/.local/pipx/venvs/neuro-all/lib/python3.8/site-packages/aiohttp/http_websocket.py", line 288, in feed_data
    return self._feed_data(data)
  File "/home/andrii/.local/pipx/venvs/neuro-all/lib/python3.8/site-packages/aiohttp/http_websocket.py", line 295, in _feed_data
    for fin, opcode, payload, compressed in self.parse_frame(data):
  File "/home/andrii/.local/pipx/venvs/neuro-all/lib/python3.8/site-packages/aiohttp/http_websocket.py", line 459, in parse_frame
    raise WebSocketError(
aiohttp.http_websocket.WebSocketError: Received fragmented control frame

After this message appears, no later logs are accessible via neuro cli or webUI either (can be accessed through kubectl)

Example:

See job above.

Expected behavior:

Error is not shown and logs are accessible

Additional context:

$ neuro --version 
Neuro Platform Client 22.7.0
$ uname -a
Linux pop-os 5.17.9-051709-generic #202205230806-Ubuntu SMP PREEMPT Mon May 23 08:22:33 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux