Expanded Job Queue #1636

mraheja · 2023-01-26T18:10:26Z

Implements the expanded job queue as described here:
https://drive.google.com/file/d/1jBjaStavimgh6YWJpuKEgGyL2st65G0A/view?usp=sharing

Tested (run the relevant ones):

All smoke tests: pytest tests/test_smoke.py
Relevant individual smoke tests: pytest tests/test_smoke.py::test_fill_in_the_name
Backward compatibility tests: bash tests/backward_comaptibility_tests.sh

Michaelvll

Thanks for submitting the job queue PR again @mraheja! I haven't finished reading the PR yet, but I have several questions about the design below. : )

sky/backends/cloud_vm_ray_backend.py

Michaelvll

Thank you for the fixes @mraheja! The code looks very clean. Left several comments.

tests/test_smoke.py

sky/backends/cloud_vm_ray_backend.py

Michaelvll · 2023-02-20T22:41:16Z

sky/backends/cloud_vm_ray_backend.py

+        self._code += [
+            textwrap.dedent(f"""\
+                job_lib.set_job_started({self.job_id!r})
+                job_lib.scheduler.schedule_step()


Should this line be moved to after L316, as we may want to schedule the next job as soon as the current job is fulfilled?

Bump @mraheja, I am not sure why this is marked as resolved, as we may want to start scheduling the next job as soon as the placement group is fulfilled for the job, instead of after the setup is done?

Since we schedule the next job immediately after the pg is fulfilled, this line should be able to be removed to reduce the overhead?

sky/skylet/job_lib.py

sky/backends/cloud_vm_ray_backend.py

sky/skylet/job_lib.py

…o expanded-job-queue

Michaelvll

Thanks for the updates @mraheja! The PR looks quite good to me now. I will test it after the comments are fixed.

sky/backends/cloud_vm_ray_backend.py

sky/skylet/job_lib.py

sky/backends/cloud_vm_ray_backend.py

tests/test_smoke.py

Michaelvll

Thanks for the fix @mraheja! Let's try to get this PR in soon. : )
Left several comments. Will test it as well.

sky/backends/cloud_vm_ray_backend.py

sky/execution.py

sky/skylet/job_lib.py

Michaelvll

Some quick comments.

sky/backends/cloud_vm_ray_backend.py

sky/skylet/job_lib.py

Michaelvll · 2023-03-24T16:49:12Z

sky/backends/cloud_vm_ray_backend.py

+# Restart skylet when the version does not match to keep the skylet up-to-date.
+_MAYBE_SKYLET_RESTART_CMD = (
+    f'[[ $(cat {constants.SKYLET_VERSION_FILE}) = "{constants.SKYLET_VERSION}"'
+    ' ]] || (pkill -f "python3 -m sky.skylet.skylet";'
+    f' echo {constants.SKYLET_VERSION} > {constants.SKYLET_VERSION_FILE};'
+    'nohup python3 -m sky.skylet.skylet >> ~/.sky/skylet.log 2>&1 &);')
+


Did we test this manually, just to make sure it does not update the skylet every time?

…o expanded-job-queue

Michaelvll

Thanks for addressing the comments @mraheja! The current one looks quite good to me. I am running the pytest tests/test_smoke.py but got a bunch of failures due to the timeout. Could we try to run those tests and see figure the problem out?

sky/backends/cloud_vm_ray_backend.py

…o expanded-job-queue

Co-authored-by: Zhanghao Wu <[email protected]>

…o expanded-job-queue

…skypilot into expanded-job-queue

…o expanded-job-queue

sky/backends/cloud_vm_ray_backend.py

sky/skylet/attempt_skylet.py

sky/skylet/job_lib.py

sky/backends/cloud_vm_ray_backend.py

sky/skylet/job_lib.py

sky/backends/cloud_vm_ray_backend.py

Michaelvll · 2023-06-06T02:28:26Z

Tested:

pytest tests/test_smoke.py

Submit 300 jobs to a CPU node. (Original master branch fails after 70 jobs due to OOM issue)

sky launch -c test-job-cancellation -y --cpus 4 --cloud gcp
for i in `seq 1 1000`; do sky exec test-job-cancellation -d "echo hi; sleep 1000000000000" done
sky cancel test-job-cancellation -y 94 95 96 979 8 99 100 125 126 127 # Cancelling PENDING job
sky cancel test-job-cancellation -y 2 5 7 # Cancelling running job and testing FIFO

Do not schedule a pending job after manually reboot the cluster on cloud console.
tests/backward_compatibility_test.sh 1 (launch is required after updating to this PR)

Michaelvll

With several additional fixes above, the PR looks quite good to me now! Thanks for the great effort @mraheja! This should help a lot with the usability of the job queue.

concretevitamin · 2023-06-07T02:11:24Z

sky/backends/cloud_vm_ray_backend.py

+        if 'has no attribute' in stdout:
+            # Happens when someone calls `sky exec` but remote is outdated
+            # necessicating calling `sky launch`
+            with ux_utils.print_exception_no_traceback():
+                raise RuntimeError(
+                    f'{colorama.Fore.RED}SkyPilot runtime is stale on the '
+                    'remote cluster. To update, run: sky launch -c '
+                    f'{handle.cluster_name}{colorama.Style.RESET_ALL}')


Ran into this when testing #1890 smoke tests (commit b020963) using a smoke VM launched by sky. I added code to print out stdout:

RuntimeError: SkyPilot runtime is stale on the remote cluster. To update, run: sky launch -c t-env-check-3917-b8 output: Traceback (most recent call last): File "<string>", line 1, in <module> AttributeError: module 'sky.skylet.job_lib' has no attribute 'scheduler'

This test cluster was freshly launched from the smoke VM, so not sure why the initial launch followed by an exec would trigger this staleness error?

cc @Michaelvll

Btw, is this check necessary for the PR to function? The 'has no attribute' in stdout can be brittle.

mraheja added 3 commits January 25, 2023 23:42

working expanded queue

9b9b670

added event

c099057

updated testing and bug fixes

281d798

mraheja mentioned this pull request Jan 26, 2023

Expanded Job Queue #1218

Closed

mraheja added 2 commits January 26, 2023 10:17

fixed missed space

5ad959d

formatting fix

440fc68

Michaelvll reviewed Feb 1, 2023

View reviewed changes

sky/backends/cloud_vm_ray_backend.py Outdated Show resolved Hide resolved

sky/backends/cloud_vm_ray_backend.py Outdated Show resolved Hide resolved

mraheja added 3 commits February 14, 2023 20:07

fixed setting up status + enabled spot

d92da6a

format

0fed010

fixed setting up test

2cb6d19

Michaelvll reviewed Feb 20, 2023

View reviewed changes

mraheja added 7 commits February 23, 2023 11:00

addressed comments

42ce456

change stop lock range

496284e

Merge branch 'master' of https://github.com/skypilot-org/skypilot int…

21c3836

…o expanded-job-queue

error message for needing update

cbf8ab3

removed wandb.py

afedc83

added skylet restart

68dea21

Merge branch 'master' of https://github.com/skypilot-org/skypilot int…

fdc68e2

…o expanded-job-queue

Michaelvll reviewed Mar 9, 2023

View reviewed changes

mraheja added 2 commits March 10, 2023 10:01

updates

5af3146

addressed comments

052a579

Michaelvll reviewed Mar 17, 2023

View reviewed changes

formatting + minor comment addressing

6c29f09

Michaelvll reviewed Mar 24, 2023

View reviewed changes

mraheja and others added 3 commits March 24, 2023 10:28

more comments addressed

be36c35

Merge branch 'master' of https://github.com/skypilot-org/skypilot int…

a12577e

…o expanded-job-queue

Fix rich status

21919c5

Michaelvll reviewed Mar 27, 2023

View reviewed changes

sky/backends/cloud_vm_ray_backend.py Outdated Show resolved Hide resolved

mraheja added 2 commits April 11, 2023 09:42

fixed stalling issue

0b7141a

Merge branch 'master' of https://github.com/skypilot-org/skypilot int…

2e46f14

…o expanded-job-queue

Michaelvll linked an issue May 31, 2023 that may be closed by this pull request

[Job] sky cancel for jobs waiting in placement group #1357

Closed

Michaelvll mentioned this pull request May 31, 2023

[Job] sky cancel for jobs waiting in placement group #1357

Closed

mraheja and others added 6 commits May 31, 2023 18:45

Update sky/backends/cloud_vm_ray_backend.py

4ac4335

Co-authored-by: Zhanghao Wu <[email protected]>

addressed more comments

f1556e6

Merge branch 'master' of https://github.com/skypilot-org/skypilot int…

c4e21d1

…o expanded-job-queue

Merge branch 'expanded-job-queue' of https://github.com/skypilot-org/…

b7d13b1

…skypilot into expanded-job-queue

formatting

d305d26

Merge branch 'master' of https://github.com/skypilot-org/skypilot int…

88dd953

…o expanded-job-queue

Michaelvll reviewed Jun 5, 2023

View reviewed changes

sky/backends/cloud_vm_ray_backend.py Outdated Show resolved Hide resolved

Michaelvll added 3 commits June 5, 2023 18:09

Address comments

cb0d66f

faster job scheduling

8b28637

format

c5881b7

Michaelvll removed a link to an issue Jun 6, 2023

[Job] sky cancel for jobs waiting in placement group #1357

Closed

Michaelvll added 4 commits June 5, 2023 21:47

Fix cancellation logic

cc7b7fd

Don't schedule a job after reboot

4e45030

add comment

6903de7

revert changes in test

a6e82eb

Michaelvll approved these changes Jun 6, 2023

View reviewed changes

Michaelvll added 2 commits June 6, 2023 00:43

Add test for cancelling pending jobs

46b7c11

Make update job status more readable

505f9e3

Michaelvll force-pushed the expanded-job-queue branch from 375bb20 to 505f9e3 Compare June 6, 2023 08:33

schedule more frequently for job cancelling

f4a2b80

Michaelvll merged commit f431fc6 into master Jun 6, 2023

Michaelvll deleted the expanded-job-queue branch June 6, 2023 10:07

This was referenced Jun 6, 2023

[100 jobs] Abnormal failover leading to duplicate instances #656

Closed

[100 jobs/Spot] Submitting 100 long running spot jobs should correctly queue them #2041

Closed

[Spot] Fix spot pending status #2044

Merged

concretevitamin reviewed Jun 7, 2023

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Expanded Job Queue #1636

Expanded Job Queue #1636

mraheja commented Jan 26, 2023 •

edited

Loading

Michaelvll left a comment

Michaelvll left a comment

Michaelvll Feb 20, 2023

Michaelvll May 7, 2023

Michaelvll May 24, 2023

Michaelvll left a comment

Michaelvll left a comment

Michaelvll left a comment

Michaelvll Mar 24, 2023

Michaelvll left a comment

Michaelvll commented Jun 6, 2023 •

edited

Loading

Michaelvll left a comment •

edited

Loading

concretevitamin Jun 7, 2023

concretevitamin Jun 7, 2023

Expanded Job Queue #1636

Expanded Job Queue #1636

Conversation

mraheja commented Jan 26, 2023 • edited Loading

Michaelvll left a comment

Choose a reason for hiding this comment

Michaelvll left a comment

Choose a reason for hiding this comment

Michaelvll Feb 20, 2023

Choose a reason for hiding this comment

Michaelvll May 7, 2023

Choose a reason for hiding this comment

Michaelvll May 24, 2023

Choose a reason for hiding this comment

Michaelvll left a comment

Choose a reason for hiding this comment

Michaelvll left a comment

Choose a reason for hiding this comment

Michaelvll left a comment

Choose a reason for hiding this comment

Michaelvll Mar 24, 2023

Choose a reason for hiding this comment

Michaelvll left a comment

Choose a reason for hiding this comment

Michaelvll commented Jun 6, 2023 • edited Loading

Michaelvll left a comment • edited Loading

Choose a reason for hiding this comment

concretevitamin Jun 7, 2023

Choose a reason for hiding this comment

concretevitamin Jun 7, 2023

Choose a reason for hiding this comment

mraheja commented Jan 26, 2023 •

edited

Loading

Michaelvll commented Jun 6, 2023 •

edited

Loading

Michaelvll left a comment •

edited

Loading