[Java worker] Migrate task execution and submission on top of core worker #5370

kfstorm · 2019-08-05T11:06:19Z

What do these changes do?

Since #5061 is too large, I submitted #5079 to migrate part of Java worker on top of the core worker. This PR is the continuation of #5079 to migrate task execution and submission of Java worker.

Related issue number

#4850

Linter

I've run scripts/format.sh to lint the changes in this PR.

…e tasks.

…asma-mutext

…peration.

…rker_core

…ker_core

kfstorm · 2019-08-08T15:54:54Z

@raulchen Folder structure updated. Can you take a look?

AmplabJenkins · 2019-08-08T17:43:39Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/16138/
Test PASSed.

AmplabJenkins · 2019-08-08T20:44:52Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/16139/
Test PASSed.

java/api/src/main/java/org/ray/api/id/ActorId.java

java/runtime/src/main/java/org/ray/runtime/RayDevRuntime.java

java/runtime/src/main/java/org/ray/runtime/AbstractRayRuntime.java

java/runtime/src/main/java/org/ray/runtime/RayNativeRuntime.java

java/runtime/src/main/java/org/ray/runtime/actor/NativeRayActor.java

java/runtime/src/main/java/org/ray/runtime/functionmanager/FunctionDescriptor.java

java/runtime/src/main/java/org/ray/runtime/object/NativeObjectStore.java

java/runtime/src/main/java/org/ray/runtime/object/ObjectStoreProxy.java

raulchen

Thanks. I looked through the whole PR again. It looks pretty good. Left some small comments.

raulchen · 2019-08-09T08:45:01Z

java/runtime/src/main/java/org/ray/runtime/task/ArgumentsBuilder.java

-      } else if (arg instanceof RayActor) {
-        data = Serializer.encode(arg);
+      } else if (arg instanceof NativeRayActor) {
+        id = Ray.internal().put(arg).getId();


Is this change intended?

This came from #5061, which refactored serialization and also supports passing actor handles cross languages. I've reverted it in this PR to keep the logic consistent with master branch.

java/runtime/src/main/java/org/ray/runtime/task/TaskExecutor.java

src/ray/core_worker/lib/java/org_ray_runtime_RayNativeRuntime.cc

AmplabJenkins · 2019-08-09T14:55:17Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/16186/
Test PASSed.

AmplabJenkins · 2019-08-09T15:08:49Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/16185/
Test PASSed.

AmplabJenkins · 2019-08-12T12:18:20Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/16234/
Test PASSed.

AmplabJenkins · 2019-08-14T16:45:45Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/16290/
Test PASSed.

AmplabJenkins · 2019-08-14T17:04:06Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/16291/
Test PASSed.

AmplabJenkins · 2019-08-15T09:28:35Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/16309/
Test PASSed.

AmplabJenkins · 2019-08-15T10:35:49Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/16312/
Test PASSed.

… core worker (#5370)" This reverts commit b1aae0e.

AmplabJenkins · 2019-08-16T08:37:59Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/16323/
Test PASSed.

zhijunfu and others added 30 commits June 14, 2019 10:33

test plasma change

050d5d8

Initial version of CoreWorkerProxy

d68e552

add files to plasma BAZEL file

c854792

Merge remote-tracking branch 'origin/master' into java_worker_core

a7218a9

Implement object and task interfce. Pass basic tests for normal remot…

887c575

…e tasks.

update commit

913e38e

Merge remote-tracking branch 'origin/test-plasma-mutext' into test-pl…

3aa2042

…asma-mutext

Support actor creation and call

64e7fc6

Support actor fork

292a776

Support actor serialization

58ace76

Ignore duplicate put

e76928d

Fix typo

a30dfb9

Merge remote-tracking branch 'origin/master' into java_worker_core

fe665e2

Serialize actor handle in core worker

8cfcb9a

Support get exceptions

6416c68

Restore API and recover some tests

500bb2e

Restore java raylet client

784f3cb

Restore some files. add name in TODO.

69db828

Return immediately if any exception found in object interface's Get o…

0789464

…peration.

Fix wrong num_returns if java call python actor

81942a0

Merge remote-tracking branch 'origin/test-plasma-mutext' into java_wo…

4159a01

…rker_core

Fix multi-threading by adding mutex for actor handle.

27d3c0f

Merge commit '7bda5edc16d40880b16ac04a5421dbfe79f4ccb2' into java_wor…

902a39a

…ker_core

Fix set resource API

bc15cf6

Remove testSerializePyActor

bf6f5ac

log a message if the get operation attempted too many times.

52f5d7a

Minor change

1ebed3b

Add metadata support in core worker object interface

e07bd1c

Serialize ActorHandle in core worker. Make ActorHandle thread safe.

6c7fee0

Add name in TODO

f8c2c0c

Add docs. Small changes.

957f3c7

raulchen reviewed Aug 9, 2019

View reviewed changes

Merge ObjectStore and ObjectStoreProxy

a288f53

raulchen reviewed Aug 9, 2019

View reviewed changes

kfstorm added 2 commits August 9, 2019 18:42

Address comments

95bd6f8

Address comments

3c007d8

Address comment

66c9a9f

kfstorm added 2 commits August 14, 2019 21:19

Enable glog in Java worker. Add more logs. Fix bugs.

92622b4

lint

c452df3

kfstorm added 2 commits August 15, 2019 13:54

Merge remote-tracking branch 'origin/master' into java_worker_core

c0bafd5

Fix

a9e5b3e

raulchen approved these changes Aug 15, 2019

View reviewed changes

Address comments

8a44f4b

raulchen merged commit b1aae0e into ray-project:master Aug 16, 2019

raulchen deleted the java_worker_core branch August 16, 2019 05:52

raulchen mentioned this pull request Aug 16, 2019

Revert raylet to worker GRPC communication back to asio #5450

Merged

1 task

pcmoritz added a commit that referenced this pull request Aug 16, 2019

Revert "[Java worker] Migrate task execution and submission on top of…

673f0a3

… core worker (#5370)" This reverts commit b1aae0e.

kfstorm mentioned this pull request Aug 22, 2019

[Java] Fix getCurrentActorId in multi-threading scenario. #5506

Merged

1 task

kfstorm mentioned this pull request Dec 6, 2019

[Java] Fix instanceof RayPyActor #6377

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Java worker] Migrate task execution and submission on top of core worker #5370

[Java worker] Migrate task execution and submission on top of core worker #5370

kfstorm commented Aug 5, 2019

kfstorm commented Aug 8, 2019

AmplabJenkins commented Aug 8, 2019

AmplabJenkins commented Aug 8, 2019

raulchen left a comment

raulchen Aug 9, 2019

kfstorm Aug 9, 2019

AmplabJenkins commented Aug 9, 2019

AmplabJenkins commented Aug 9, 2019

AmplabJenkins commented Aug 12, 2019

AmplabJenkins commented Aug 14, 2019

AmplabJenkins commented Aug 14, 2019

AmplabJenkins commented Aug 15, 2019

AmplabJenkins commented Aug 15, 2019

AmplabJenkins commented Aug 16, 2019

[Java worker] Migrate task execution and submission on top of core worker #5370

[Java worker] Migrate task execution and submission on top of core worker #5370

Conversation

kfstorm commented Aug 5, 2019

What do these changes do?

Related issue number

Linter

kfstorm commented Aug 8, 2019

AmplabJenkins commented Aug 8, 2019

AmplabJenkins commented Aug 8, 2019

raulchen left a comment

Choose a reason for hiding this comment

raulchen Aug 9, 2019

Choose a reason for hiding this comment

kfstorm Aug 9, 2019

Choose a reason for hiding this comment

AmplabJenkins commented Aug 9, 2019

AmplabJenkins commented Aug 9, 2019

AmplabJenkins commented Aug 12, 2019

AmplabJenkins commented Aug 14, 2019

AmplabJenkins commented Aug 14, 2019

AmplabJenkins commented Aug 15, 2019

AmplabJenkins commented Aug 15, 2019

AmplabJenkins commented Aug 16, 2019