Refactor flowchart models for modular pipelines change #1937
Conversation
Signed-off-by: ravi-kumar-pilla <[email protected]>
Signed-off-by: ravi-kumar-pilla <[email protected]>
Signed-off-by: ravi-kumar-pilla <[email protected]>
There was a problem hiding this comment.
In the description you mention
Removes namespace property from GraphNode and retain the property only for TaskNode following Kedro Framework
and I see a lot of methods related to namespaces have been removed. Can you explain why that change was needed and how namespaces are now set and used in Viz?
So earlier we used to have For a TaskNode @field_validator("namespace")
@classmethod
def set_namespace(cls, _, info: ValidationInfo):
return info.data["kedro_obj"].namespaceFor other nodes, we do not need namespace. Lot of methods related to namespace like Thank you |
| # https://kedro.readthedocs.io/en/latest/06_nodes_and_pipelines/03_modular_pipelines.html#how-to-connect-existing-pipelines | ||
| internal_inputs: Set[str] = Field( | ||
| set(), description="The dataset inputs within the modular pipeline node" | ||
| inputs: Set[str] = Field( |
| description="""The dataset outputs connecting the modular | ||
| pipeline node with other modular pipelines""", | ||
|
|
||
| outputs: Set[str] = Field( |
| name=dataset_name, | ||
| tags=tags, | ||
| layer=layer, | ||
| kedro_obj=dataset, | ||
| is_free_input=is_free_input, | ||
| stats=stats, | ||
| modular_pipelines=modular_pipelines, | ||
| ) | ||
|
|
||
| @classmethod | ||
| def create_parameters_node( |
There was a problem hiding this comment.
These factory methods look more and more useless to me. They seem to be simply indirections to the constructors and making the creating of the specific nodes more verbose without any benefit from having them. We should revisit them at some point.
| Returns: | ||
| An instance of TaskNode. | ||
| """ | ||
| node_name = node._name or node._func_name | ||
| return TaskNode( | ||
| id=cls._hash(str(node)), | ||
| id=node_id, |
There was a problem hiding this comment.
By not deriving the node_id automatically, we make these create methods even more useless.
Description
Partially resolves - #1899
Development notes
namespaceproperty from GraphNode and retain the property only for TaskNode following Kedro Frameworkmodular pipelinesis changed toOptional[Set(str)]create_nodeclass methods like (create_task_node, create_data_node, create_parameters_node etc) to accept modular_pipelines parameter as we are populating modular pipelines before creating the nodes and associating each node with the associated modular pipeline during node creation.internal_inputs/outputsandexternal_inputs/outputsfrom ModularPipelineNode. We will only have inputs/outputs for a ModularPipelineNode (i.e., the datasets which are free inputs/outputs as per Kedro)QA notes
Checklist
RELEASE.mdfile