Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Remove dynamic node from pending nodes before starting a dataflow (#606)
This PR makes a dataflow start independently from dynamic nodes. Previously, all nodes wait for each other to be ready before starting the dataflow, this makes it possible to have synchronised messages, and avoid feeling up queues before the nodes is even ready to receive message. I think in the case of dynamic node, we don't want to wait for users to spawn the nodes to start the dataflow. This creates the possibility to specify some nodes that can "tap" on the dataflow as well as remove the risk of having race condition if dynamic nodes needs to be specified in a specific order. One recent example is RJ wanting to have multiple nodes within one python process and this was not possible due to nodes being pending.
- Loading branch information