Skip to content

Commit

Permalink
Remove dynamic node from pending nodes before starting a dataflow (#606)
Browse files Browse the repository at this point in the history
This PR makes a dataflow start independently from dynamic nodes. 

Previously, all nodes wait for each other to be ready before starting
the dataflow, this makes it possible to have synchronised messages, and
avoid feeling up queues before the nodes is even ready to receive
message. I think in the case of dynamic node, we don't want to wait for
users to spawn the nodes to start the dataflow.

This creates the possibility to specify some nodes that can "tap" on the
dataflow as well as remove the risk of having race condition if dynamic
nodes needs to be specified in a specific order.

One recent example is RJ wanting to have multiple nodes within one
python process and this was not possible due to nodes being pending.
  • Loading branch information
haixuanTao authored Aug 8, 2024
2 parents 090df9f + 26a4664 commit d4e629e
Showing 1 changed file with 2 additions and 2 deletions.
4 changes: 2 additions & 2 deletions binaries/daemon/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -647,10 +647,10 @@ impl Daemon {
}
}
if local {
dataflow.pending_nodes.insert(node.id.clone());

if node.kind.dynamic() {
dataflow.dynamic_nodes.insert(node.id.clone());
} else {
dataflow.pending_nodes.insert(node.id.clone());
}

let node_id = node.id.clone();
Expand Down

0 comments on commit d4e629e

Please sign in to comment.