When a task is passed to the global scheduler, if it is not received,… #1106

robertnishihara · 2017-10-11T06:21:38Z

… then try again.

This may solve #1105.

TODO before merging:

Update the comment in

ray/src/global_scheduler/global_scheduler.cc

Lines 419 to 423 in 1e0ab3d

    
             /* Subscribe to notifications about waiting tasks. TODO(rkn): this may need to 
        
              * get tasks that were submitted to the database before the subscribe. */ 
        
             task_table_subscribe(g_state->db, NIL_ID, TASK_STATUS_WAITING, 
        
                                  process_task_waiting, (void *) g_state, NULL, NULL, 
        
                                  NULL);

… then try again.

robertnishihara · 2017-10-11T06:48:43Z

There's still a bug in this PR. On one of the machines, I see the message

[WARN] (/ray/src/common/state/redis.cc:914) No subscribers received the task_table_add message.

and the local scheduler on that machine appears defunct.

AmplabJenkins · 2017-10-11T06:59:33Z

Merged build finished. Test PASSed.

AmplabJenkins · 2017-10-11T06:59:34Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/2086/
Test PASSed.

atumanov · 2017-10-11T21:15:21Z

src/local_scheduler/local_scheduler_algorithm.cc

+
+  TaskSpec *spec = Task_task_spec(task);
+  CHECK(ActorID_equal(TaskSpec_actor_id(spec), NIL_ACTOR_ID));
+  handle_task_submitted(state, state->algorithm_state, spec,


interesting choice. So you want to recirculate the task through the local scheduler's new task handling mechanism, even though the decision has already been made to forward it to the global scheduler? So the assumption is that Redis is busy, so let's try to keep it local again.

The assumption is not that Redis is busy, but rather that the global scheduler has not started up yet.

We could alternatively simply call give_task_to_global_scheduler again.

atumanov · 2017-10-11T21:15:27Z

src/local_scheduler/local_scheduler_algorithm.cc

@@ -948,7 +961,12 @@ void give_task_to_global_scheduler(LocalSchedulerState *state,
  DCHECK(state->config.global_scheduler_exists);
  Task *task = Task_alloc(spec, task_spec_size, TASK_STATUS_WAITING, NIL_ID);
  DCHECK(state->db != NULL);
-  task_table_add_task(state->db, task, NULL, NULL, NULL);
+  auto retryInfo = RetryInfo{
+      .num_retries = 0,  // This value is unused.


if num_retries is unused, what's the behavior on failure? How many retries (and what controls that)?

If task_table_add_task fails (i.e., the global scheduler has not started yet), then give_task_to_global_scheduler_retry will be called. Which will lead to another call to task_table_add_task. This will happen indefinitely until the call to task_table_add_task succeeds (or the local scheduler decides to keep the task locally).

robertnishihara · 2017-10-11T21:17:08Z

src/common/CMakeLists.txt

@@ -6,7 +6,7 @@ include(${CMAKE_CURRENT_LIST_DIR}/cmake/Common.cmake)

 add_subdirectory(redis_module)

-set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -fPIC")
+set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -fPIC -g")


Any reason not to compile with debug info all the time?

size of the executables is the only one that comes to mind, which we don't care about that much

robertnishihara · 2017-10-11T21:29:21Z

cc @stephanie-wang

AmplabJenkins · 2017-10-11T21:34:25Z

Merged build finished. Test PASSed.

AmplabJenkins · 2017-10-11T21:34:25Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/2090/
Test PASSed.

AmplabJenkins · 2017-10-12T03:05:06Z

Merged build finished. Test PASSed.

AmplabJenkins · 2017-10-12T03:05:06Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/2098/
Test PASSed.

I would like to enhance the user experience for installing Helm chart RBAC in KubeRay. Since there is'n an existing document for this, I integrate these PRs including KubeRay #1190, #1162, #1106, and create a comprehensive document to assist users in quickly and easily setting up RBAC resource. --------- Signed-off-by: evalaiyc98 <[email protected]> Signed-off-by: Kai-Hsun Chen <[email protected]> Signed-off-by: Kai-Hsun Chen <[email protected]> Signed-off-by: Yu-Chen Lai <[email protected]> Co-authored-by: Kai-Hsun Chen <[email protected]> Co-authored-by: Kai-Hsun Chen <[email protected]> Co-authored-by: Archit Kulkarni <[email protected]> Co-authored-by: angelinalg <[email protected]>

I would like to enhance the user experience for installing Helm chart RBAC in KubeRay. Since there is'n an existing document for this, I integrate these PRs including KubeRay ray-project#1190, ray-project#1162, ray-project#1106, and create a comprehensive document to assist users in quickly and easily setting up RBAC resource. --------- Signed-off-by: evalaiyc98 <[email protected]> Signed-off-by: Kai-Hsun Chen <[email protected]> Signed-off-by: Kai-Hsun Chen <[email protected]> Signed-off-by: Yu-Chen Lai <[email protected]> Co-authored-by: Kai-Hsun Chen <[email protected]> Co-authored-by: Kai-Hsun Chen <[email protected]> Co-authored-by: Archit Kulkarni <[email protected]> Co-authored-by: angelinalg <[email protected]>

I would like to enhance the user experience for installing Helm chart RBAC in KubeRay. Since there is'n an existing document for this, I integrate these PRs including KubeRay #1190, #1162, #1106, and create a comprehensive document to assist users in quickly and easily setting up RBAC resource. --------- Signed-off-by: evalaiyc98 <[email protected]> Signed-off-by: Kai-Hsun Chen <[email protected]> Signed-off-by: Kai-Hsun Chen <[email protected]> Signed-off-by: Yu-Chen Lai <[email protected]> Co-authored-by: Yu-Chen Lai <[email protected]> Co-authored-by: Archit Kulkarni <[email protected]> Co-authored-by: angelinalg <[email protected]>

When a task is passed to the global scheduler, if it is not received,…

28b564e

… then try again.

robertnishihara added 3 commits October 11, 2017 12:27

Fix spacing.

179092a

Fix bug.

e123ae8

Update comment.

8146b59

atumanov reviewed Oct 11, 2017

View reviewed changes

robertnishihara commented Oct 11, 2017

View reviewed changes

robertnishihara mentioned this pull request Oct 11, 2017

In Jenkins tests, test_0.py hangs occasionally. #1105

Closed

Call give_task_to_global_scheduler directly (same with local).

6768bfa

pcmoritz approved these changes Oct 12, 2017

View reviewed changes

atumanov merged commit b585001 into ray-project:master Oct 12, 2017

atumanov deleted the retrypassingtoglobal branch October 12, 2017 07:05

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

When a task is passed to the global scheduler, if it is not received,… #1106

When a task is passed to the global scheduler, if it is not received,… #1106

robertnishihara commented Oct 11, 2017 •

edited

Loading

robertnishihara commented Oct 11, 2017

AmplabJenkins commented Oct 11, 2017

AmplabJenkins commented Oct 11, 2017

atumanov Oct 11, 2017

robertnishihara Oct 11, 2017 •

edited

Loading

atumanov Oct 11, 2017

robertnishihara Oct 11, 2017

robertnishihara Oct 11, 2017

pcmoritz Oct 12, 2017

robertnishihara commented Oct 11, 2017

AmplabJenkins commented Oct 11, 2017

AmplabJenkins commented Oct 11, 2017

AmplabJenkins commented Oct 12, 2017

AmplabJenkins commented Oct 12, 2017

	/* Subscribe to notifications about waiting tasks. TODO(rkn): this may need to
	* get tasks that were submitted to the database before the subscribe. */
	task_table_subscribe(g_state->db, NIL_ID, TASK_STATUS_WAITING,
	process_task_waiting, (void *) g_state, NULL, NULL,
	NULL);

When a task is passed to the global scheduler, if it is not received,… #1106

When a task is passed to the global scheduler, if it is not received,… #1106

Conversation

robertnishihara commented Oct 11, 2017 • edited Loading

robertnishihara commented Oct 11, 2017

AmplabJenkins commented Oct 11, 2017

AmplabJenkins commented Oct 11, 2017

atumanov Oct 11, 2017

Choose a reason for hiding this comment

robertnishihara Oct 11, 2017 • edited Loading

Choose a reason for hiding this comment

atumanov Oct 11, 2017

Choose a reason for hiding this comment

robertnishihara Oct 11, 2017

Choose a reason for hiding this comment

robertnishihara Oct 11, 2017

Choose a reason for hiding this comment

pcmoritz Oct 12, 2017

Choose a reason for hiding this comment

robertnishihara commented Oct 11, 2017

AmplabJenkins commented Oct 11, 2017

AmplabJenkins commented Oct 11, 2017

AmplabJenkins commented Oct 12, 2017

AmplabJenkins commented Oct 12, 2017

robertnishihara commented Oct 11, 2017 •

edited

Loading

robertnishihara Oct 11, 2017 •

edited

Loading