Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: send data between agent and worker through files instead of stream-based requests #414

Merged
merged 2 commits into from
Sep 26, 2023

Conversation

aneojgurhem
Copy link
Contributor

@aneojgurhem aneojgurhem commented Sep 15, 2023

This PR changes the way data should be sent between Worker and Scheduling Agent to use files in a shared directory instead of gRPC streams.

Rationale:
Rework communications between scheduling agent and worker so that there is no streams and pass data through file in a shared volume. This will allow some nice optimisations such as:

  • Less gRPC communications between scheduling agent and worker
  • Simpler RPC methods between scheduling agent and worker
  • Tasks that start faster: no need to receive the input payload and dependencies, they will be available in the shared volume at the start of the task due to prefetching
  • Data can be consummed when needed during the task; not forcefully received at the start
  • A node-wide caching system could be implemented through a volume shared between all the scheduling agents and workers that would allow the scheduling agent to know that some data are already there and reuse them to avoid unnecessary data download. This should be implemented in ArmoniK.Core and should not impact APIs and Worker behavior

What changed ?

  • Worker Process RPC is now an Unitary Call (no streams) with a data_location folder in which Payload and Data Dependencies are located as file whose name is the ID of the input
  • Agent Get*Data are now Unitary Calls too : The call returns when expected data are located in the data_location folder as file whose name is the ID of the required data
  • Agent UploadResultData and SendResult became NotifyResultData. Results should be stored as a file whose name is the ID of the results in the data_location folder then the NotifyResultData method should be called to notify Agent that data were submitted.

@aneojgurhem aneojgurhem marked this pull request as draft September 15, 2023 16:56
@aneojgurhem aneojgurhem marked this pull request as ready for review September 25, 2023 06:26
@aneojgurhem aneojgurhem self-assigned this Sep 26, 2023
@aneojgurhem aneojgurhem merged commit 9fd5060 into main Sep 26, 2023
23 of 30 checks passed
@aneojgurhem aneojgurhem deleted the jg/datafromfiles branch September 26, 2023 11:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants