Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make sky logs with piping handle interrupts gracefully #1169

Closed
concretevitamin opened this issue Sep 13, 2022 · 0 comments · Fixed by #1324
Closed

Make sky logs with piping handle interrupts gracefully #1169

concretevitamin opened this issue Sep 13, 2022 · 0 comments · Fixed by #1324

Comments

@concretevitamin
Copy link
Member

If using sky logs .. | grep ... in a pipe, then ctrl-c would print out a long stacktrace:

$ sky logs sky-spot-controller-abb92b6d 34 | grep 'preempted from'
(zh-14-26 pid=97928) I 09-13 04:21:03 recovery_strategy.py:355] Cluster is preempted from zone us-west1-a.
(zh-14-26 pid=97928) I 09-13 05:31:15 recovery_strategy.py:355] Cluster is preempted from zone us-west1-b.
^C--- Logging error ---
Traceback (most recent call last):
  File "/opt/conda/lib/python3.9/logging/__init__.py", line 1087, in emit
    self.flush()
BrokenPipeError: [Errno 32] Broken pipe
Call stack:
  File "/home/ubuntu/.local/bin/sky", line 8, in <module>
    sys.exit(cli())
  File "/home/ubuntu/.local/lib/python3.9/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/home/ubuntu/.local/lib/python3.9/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/home/ubuntu/.local/lib/python3.9/site-packages/sky/utils/common_utils.py", line 147, in _record
    return f(*args, **kwargs)
  File "/home/ubuntu/.local/lib/python3.9/site-packages/sky/cli.py", line 776, in invoke
    return super().invoke(ctx)
  File "/home/ubuntu/.local/lib/python3.9/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/ubuntu/.local/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/ubuntu/.local/lib/python3.9/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/home/ubuntu/.local/lib/python3.9/site-packages/sky/utils/common_utils.py", line 168, in _record
    return f(*args, **kwargs)
  File "/home/ubuntu/.local/lib/python3.9/site-packages/sky/cli.py", line 1210, in logs
    core.tail_logs(cluster, job_id)
  File "/home/ubuntu/.local/lib/python3.9/site-packages/sky/core.py", line 354, in tail_logs
    backend.tail_logs(handle, job_id)
  File "/home/ubuntu/.local/lib/python3.9/site-packages/sky/backends/cloud_vm_ray_backend.py", line 2403, in tail_logs
    returncode = self.run_on_head(
  File "/home/ubuntu/.local/lib/python3.9/site-packages/sky/utils/common_utils.py", line 168, in _record
    return f(*args, **kwargs)
  File "/home/ubuntu/.local/lib/python3.9/site-packages/sky/backends/cloud_vm_ray_backend.py", line 2692, in run_on_head
    return runner.run(
  File "/home/ubuntu/.local/lib/python3.9/site-packages/sky/utils/command_runner.py", line 255, in run
    return log_lib.run_with_log(' '.join(command),
  File "/home/ubuntu/.local/lib/python3.9/site-packages/sky/skylet/log_lib.py", line 233, in run_with_log
    proc.wait()
  File "/opt/conda/lib/python3.9/subprocess.py", line 1189, in wait
    return self._wait(timeout=timeout)
  File "/opt/conda/lib/python3.9/subprocess.py", line 1917, in _wait
    (pid, sts) = self._try_wait(0)
  File "/opt/conda/lib/python3.9/subprocess.py", line 1875, in _try_wait
    (pid, sts) = os.waitpid(self.pid, wait_flags)
  File "/home/ubuntu/.local/lib/python3.9/site-packages/sky/backends/backend_utils.py", line 1909, in interrupt_handler
    logger.warning(f'{colorama.Fore.LIGHTBLACK_EX}The job will keep '
Message: '\x1b[90mThe job will keep running after Ctrl-C.\x1b[0m'
Arguments: ()
Shared connection to 44.210.146.121 closed.

Aborted!
Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>
BrokenPipeError: [Errno 32] Broken pipe
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant