%load_node line magic improvements

[AhdraMeraliQB]

Description

In #3510 we introduce a new line magic, aimed at improving the process of debugging Kedro projects in notebooks. This feature is experimental - this issue should be used to add suggestions for extending and improving it. Add a suggestion in the comments, or if already mentioned, bump its priority with a 👍 .

(edited by Nok)

• %load_node support full Kedro Node syntax, with better *args, **kwargs handling #3629
• Add support for other platforms, Support notebook/lab/vscode/ipython now, minimal support for Databricks and others
• Add import statement to import * from node source file - allows nodes with helper functions to be runnable in notebooks
• better handle of MemoryDataset, or guide user to run the necessary node to persist data
• Consider before_node_run, after_node_run or document how to handle edge case if user project has hooks that mutate inputs

(Copy from previous issue)

Two-way sync

Would be handy to debug, but you should do this in the IDE instead of a notebook. If you change the code, you still have to manually sync the changes to VCS
Look awesome
@Nok
. Would there be a path back from notebook into the project’s files (as it was with kedro jupyter convert )?

Recursive definition of function body

Also how would this work if your node function imports other functions which may import other functions? You only show where depth is 1?

[AhdraMeraliQB]

Add support for other platforms, currently only supports jupyter lab/notebook (#3510) and ipython (#3536). Consider including:

• Databricks
• VSCode

[AhdraMeraliQB]

Add import statement to import * from node source file - allows nodes with helper functions to be runnable in notebooks without having to go back to source files and copy paste the code over

Edited by Nok below:
if we can use insppect.getsourcefile, we can directly import the module with importlib, then we can use from <module> import * to make sure everything is loaded.
https://docs.python.org/3/library/importlib.html#importing-a-source-file-directly

[AhdraMeraliQB]

Resolve MemoryDatasets so that users don't have to add them to catalog to access them as node inputs

[noklam]

if we can use insppect.getsourcefile, we can directly import the module with importlib, then we can use from <module> import * to make sure everything is loaded.
https://docs.python.org/3/library/importlib.html#importing-a-source-file-directly
Cc @DimedS

, it works well on my side! This new functionality is fantastic! It would be helpful to include instructions in the documentation for this new command on how to use it in different environments. Additionally, perhaps we should consider adding relative functions when loading specific nodes. Currently, in spaceflights-pandas, a node is loaded but cannot be run because it returns a NameError: name '_is_true' is not defined error, indicating that the function _is_true was not loaded.
#3604 (review)

[noklam]

I find a couple of things to improve when I try to help an user to debug on 0.18.x

• When running %load_node on IPython, if the code block is long enough, the top of the code block will "disappear" unless you hit the arrow. At first I thought the function is broken. This is confusing because most likely the function will not run (assuming there are some error and user want to debug), they will not see the variable declaration in the terminal data_a = catalog.load("xxx")
• It seems broken if the function is from "wheel" or somehow wrapped, cannot reproduce this yet but the error is "FileNotFoundError: [Errno 2] No such file or directory: '<boltons.funcutils.FunctionBuilder-115>'"
• Maybe prepare some standalone script so that we can test 0.18.x
• The function call always have the full function signature, which is problematic if some of them is optional argument.

i.e.
def node_func(a,b, c=None):
return ...

It should be valid to have a node node(node_func, inputs=["data_a","data_b"], ...). Currently the result code block is

node_func(a,b,c)

This will cause error because c is not defined, this will work as long as we delete c from the result code block.

[noklam]

Better handle of *args and **kwargs, currently the %load_node have a simple logic to map node's input to function parameters.

The idea is to use inspect.Signature.bind and inspect.Parameters to identify the special arguments (VAR_POSITIONAL)

For example:

def dummy(a,b,c, *args, **kwargs):
    ...

node(dummy, ["data_1", "data_2", "data_3", "dummy1","dummy2","dummy3"], ...)

should translate to

a = catalog.load("data_1")
b = catalog.load("data_2")
c = catalog.load("data_3")
dummy1 = catalog.load("dummy1")
dummy2 = catalog.load("dummy2")
dummy3 = catalog.load("dummy3")
args = [dummy1, dummy2, dummy3] # Noted here the name of the "dummy_x" variable are arbitrary
dummy(a, b, c, *args)

[noklam]

Consider adding before_node_run and after_node_run. If user mutates the inputs with hooks, the current logic fails to do so. For example, some users have a custom ConfigLoader that only instantiate object with before_node_run, so catalog.load only return a dict object but not the instantiated class.