Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

More abstract Worker responsibility #1213

Open
allada opened this issue Jul 29, 2024 · 0 comments
Open

More abstract Worker responsibility #1213

allada opened this issue Jul 29, 2024 · 0 comments

Comments

@allada
Copy link
Member

allada commented Jul 29, 2024 •
edited
Loading

Currently the "worker" component fills a few different roles:

  • Downloads & hardlinks files into scratch space
  • Execute run underlying task
  • Upload results

In reality this needs to be split into a couple layers. Most importantly, we want to support the ability to have a different component that actually executes the work from the others. This will allow us to make custom runner backends that can execute into containerd or another VM, run on host, or maybe even something more exotic, like SSH.

Diagram example of new component layout:
Fully distributed scheduler design
allada added a commit to allada/nativelink-fork that referenced this issue Jul 29, 2024
@allada
Remove ClientOperationId and move all to OperationId
f881479 
This is in prep to support generic layering of workers & schedulers.
We can no longer know if the source is a client or just an
up-stream-component, so we need to fall back to just using OperationId.

towards TraceMachina#1213
@allada allada mentioned this issue Jul 29, 2024
Merged
allada added a commit to allada/nativelink-fork that referenced this issue Aug 2, 2024
@allada
Remove ClientOperationId and move all to OperationId
231676c 
This is in prep to support generic layering of workers & schedulers.
We can no longer know if the source is a client or just an
up-stream-component, so we need to fall back to just using OperationId.

towards TraceMachina#1213
allada added a commit to allada/nativelink-fork that referenced this issue Aug 6, 2024
@allada
Remove ClientOperationId and move all to OperationId
d27f7f4 
This is in prep to support generic layering of workers & schedulers.
We can no longer know if the source is a client or just an
up-stream-component, so we need to fall back to just using OperationId.

towards TraceMachina#1213
allada added a commit to allada/nativelink-fork that referenced this issue Aug 6, 2024
@allada
Move ActionSchedulerListener to ActionStateResult
4f507ea 
These APIs were near identical and with the new scheduler/worker design
we will be eventually removing the ActionScheduler API and moving it to
the ClientStateManager API (eventually renamed).

towards TraceMachina#1213
@allada allada mentioned this issue Aug 6, 2024
Merged
allada added a commit to allada/nativelink-fork that referenced this issue Aug 6, 2024
@allada
Move ActionSchedulerListener to ActionStateResult
1c26c91 
These APIs were near identical and with the new scheduler/worker design
we will be eventually removing the ActionScheduler API and moving it to
the ClientStateManager API (eventually renamed).

towards TraceMachina#1213
allada added a commit to allada/nativelink-fork that referenced this issue Aug 6, 2024
@allada
Move ActionSchedulerListener to ActionStateResult
99f1659 
These APIs were near identical and with the new scheduler/worker design
we will be eventually removing the ActionScheduler API and moving it to
the ClientStateManager API (eventually renamed).

towards TraceMachina#1213
allada added a commit to allada/nativelink-fork that referenced this issue Aug 6, 2024
@allada
Remove ClientOperationId and move all to OperationId
bf2f914 
This is in prep to support generic layering of workers & schedulers.
We can no longer know if the source is a client or just an
up-stream-component, so we need to fall back to just using OperationId.

towards TraceMachina#1213
allada added a commit that referenced this issue Aug 6, 2024
@allada
Remove ClientOperationId and move all to OperationId (#1214)
81db90e 
This is in prep to support generic layering of workers & schedulers.
We can no longer know if the source is a client or just an
up-stream-component, so we need to fall back to just using OperationId.

towards #1213
allada added a commit to allada/nativelink-fork that referenced this issue Aug 6, 2024
@allada
Move ActionSchedulerListener to ActionStateResult
c159e7e 
These APIs were near identical and with the new scheduler/worker design
we will be eventually removing the ActionScheduler API and moving it to
the ClientStateManager API (eventually renamed).

towards TraceMachina#1213
allada added a commit to allada/nativelink-fork that referenced this issue Aug 6, 2024
@allada
Migrate much of the ActionScheduler API to ClientStateManager API
80ab444 
Mostly a cosmetic change to move the compatible parts of
ActionSchedulers to use ClientStateManager API instead and implement all
related requirements to existing schedulers.

towards TraceMachina#1213
@allada allada mentioned this issue Aug 6, 2024
Merged
allada added a commit to allada/nativelink-fork that referenced this issue Aug 6, 2024
@allada
Migrate much of the ActionScheduler API to ClientStateManager API
1b8279e 
Mostly a cosmetic change to move the compatible parts of
ActionSchedulers to use ClientStateManager API instead and implement all
related requirements to existing schedulers.

towards TraceMachina#1213
allada added a commit to allada/nativelink-fork that referenced this issue Aug 6, 2024
@allada
Move ActionSchedulerListener to ActionStateResult
45dc462 
These APIs were near identical and with the new scheduler/worker design
we will be eventually removing the ActionScheduler API and moving it to
the ClientStateManager API (eventually renamed).

towards TraceMachina#1213
allada added a commit that referenced this issue Aug 7, 2024
@allada
Move ActionSchedulerListener to ActionStateResult (#1237)
d57ee8d 
These APIs were near identical and with the new scheduler/worker design
we will be eventually removing the ActionScheduler API and moving it to
the ClientStateManager API (eventually renamed).

towards #1213
allada added a commit to allada/nativelink-fork that referenced this issue Aug 7, 2024
@allada
Migrate much of the ActionScheduler API to ClientStateManager API
aa9e983 
Mostly a cosmetic change to move the compatible parts of
ActionSchedulers to use ClientStateManager API instead and implement all
related requirements to existing schedulers.

towards TraceMachina#1213
allada added a commit that referenced this issue Aug 7, 2024
@allada
Migrate much of the ActionScheduler API to ClientStateManager API (#1241
2b8f1ee 
)

Mostly a cosmetic change to move the compatible parts of
ActionSchedulers to use ClientStateManager API instead and implement all
related requirements to existing schedulers.

towards #1213
allada added a commit to allada/nativelink-fork that referenced this issue Aug 12, 2024
@allada
Remove ActionScheduler and introduce KnownPlatformPropertyProvider
d08f796 
We want to phase out ActionScheduler in favor of ClientStateManager. In
doing so, we need to have a way to let CapabilitiesServer know about
platform properties; if we do this, it'd be a breaking change. Instead
of causing a breaking change, we instead are going to make the provided
scheduler allow the capabilities service query the optionally provided
property provider interface and if it does not exist throw a warning and
have it return an empty set (most RBE clients ignore this field anyway).

towards TraceMachina#1213
@allada allada mentioned this issue Aug 12, 2024
Merged
allada added a commit to allada/nativelink-fork that referenced this issue Aug 12, 2024
@allada
Remove ActionScheduler and introduce KnownPlatformPropertyProvider
3345cc0 
We want to phase out ActionScheduler in favor of ClientStateManager. In
doing so, we need to have a way to let CapabilitiesServer know about
platform properties; if we do this, it'd be a breaking change. Instead
of causing a breaking change, we instead are going to make the provided
scheduler allow the capabilities service query the optionally provided
property provider interface and if it does not exist throw a warning and
have it return an empty set (most RBE clients ignore this field anyway).

towards TraceMachina#1213
allada added a commit to allada/nativelink-fork that referenced this issue Aug 12, 2024
@allada
Remove ActionScheduler and introduce KnownPlatformPropertyProvider
18cc117 
We want to phase out ActionScheduler in favor of ClientStateManager. In
doing so, we need to have a way to let CapabilitiesServer know about
platform properties; if we do this, it'd be a breaking change. Instead
of causing a breaking change, we instead are going to make the provided
scheduler allow the capabilities service query the optionally provided
property provider interface and if it does not exist throw a warning and
have it return an empty set (most RBE clients ignore this field anyway).

towards TraceMachina#1213
allada added a commit to allada/nativelink-fork that referenced this issue Aug 14, 2024
@allada
Remove ActionScheduler and introduce KnownPlatformPropertyProvider
b4fb9e2 
We want to phase out ActionScheduler in favor of ClientStateManager. In
doing so, we need to have a way to let CapabilitiesServer know about
platform properties; if we do this, it'd be a breaking change. Instead
of causing a breaking change, we instead are going to make the provided
scheduler allow the capabilities service query the optionally provided
property provider interface and if it does not exist throw a warning and
have it return an empty set (most RBE clients ignore this field anyway).

towards TraceMachina#1213
allada added a commit that referenced this issue Aug 14, 2024
@allada
Remove ActionScheduler and introduce KnownPlatformPropertyProvider (#…
9c87370 
…1260)

We want to phase out ActionScheduler in favor of ClientStateManager. In
doing so, we need to have a way to let CapabilitiesServer know about
platform properties; if we do this, it'd be a breaking change. Instead
of causing a breaking change, we instead are going to make the provided
scheduler allow the capabilities service query the optionally provided
property provider interface and if it does not exist throw a warning and
have it return an empty set (most RBE clients ignore this field anyway).

towards #1213
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@allada