New feature list for workflows

[JNmpi]

Thanks @liamhuber for implementing the pull mode. I used it to my lammps toy nodes and it worked out of the box. The new syntax feels very natural and the code became much shorter. I also enjoyed very much the automatic generation of the workflow sequence. It makes setting up workflows much simpler and avoids extra code. This is really a great step forward.

Please find below a new list of ideas and wishes:

Next steps and desirable features

In the following some attempts to use the new pyiron workflow platform to provide the same look and feel as our present pyiron implementation. The new node-based platform has many advantages (such as storing the complete workflow) but should provide the features pyiron users are accustomed to. The following are things that nicely work but also issues that we need to address:

%config IPCompleter.evaluation='unsafe'

from pyiron_contrib.workflow.workflow import Workflow

wf = Workflow('simple_lammps_calculation')

structure = wf.create.atomistics.Bulk(cubic=True, name="Al", label='structure')

lammps_job = wf.create.lammps.LammpsStaticNode(label='lammps_job')
lammps_job.inputs.structure = structure
lammps_job.inputs.potential = '1995--Angelo-J-E--Ni-Al-H--LAMMPS--ipr1'

out = wf.run()

potential starts
potential name:  1995--Angelo-J-E--Ni-Al-H--LAMMPS--ipr1
start shell
end shell
parse dump file
parse log file

wf.draw();

What is really nice is that our new workflow object 'knows' all aspects, including how the atomic structure that enters the lammps job has been created.

In the following a wishlist of features:

Raising errors in nodes

wf = Workflow('simple_lammps_calculation')

structure = wf.create.atomistics.Bulk(cubic=True, name="Cu", label='structure')

lammps_job = wf.create.lammps.LammpsStaticNode(label='lammps_job')
lammps_job.inputs.structure = structure
lammps_job.inputs.potential = '1995--Angelo-J-E--Ni-Al-H--LAMMPS--ipr1'  # potential does not contain Cu

# out = wf.run()

potential starts

Issue: Running this script raises correctly an error. However, the error message is long. It would be great if we could directly give the error message thrown by the node without getting all the message from pyiron_workflows.

Running a node within a macro

wf = Workflow('simple_lammps_calculation')

structure = wf.create.atomistics.Bulk(cubic=True, name="Cu", label='structure')

lammps_job = wf.create.lammps.LammpsStaticNode(label='lammps_job')
lammps_job.inputs.structure = structure
lammps_job.inputs.potential = '1995--Angelo-J-E--Ni-Al-H--LAMMPS--ipr1'  # potential does not contain Cu

# lammps_job.potential.run()

potential starts

Issue: It would be great, e.g. for debugging or interactively running/analysing a workflow, to run a node within a macro. The node should then run recursively all its input nodes but not trigger any nodes that use its input. Presently, uncommenting the run line gives rise to errors. It looks like it is triggering follow-up nodes.

Provide ways to show dynamic input options

wf = Workflow('simple_lammps_calculation')

structure = wf.create.atomistics.Bulk(cubic=True, name="Cu", label='structure')

lammps_job = wf.create.lammps.LammpsStaticNode(label='lammps_job')
lammps_job.inputs.structure = structure 

lammps_job.inputs.potential.list   # show all potentials compatible to structure
lammps_job.list_potentials         # alternative syntax (list_potentials would be a node in a workflow that is not called when running the workflow but only when calling it explicitly)

potential starts

Issue: Having nodes that are not part of the main workflow but that can be used by the user to extract/provide information when explicitly called would be very helpful.

Include node containers where the actual node is chosen in the input

wf = Workflow('simple_lammps_calculation')

structure = wf.create.atomistics.Bulk(cubic=True, name="Cu", label='structure')

lammps_job = wf.create.lammps.LammpsStaticNode(label='lammps_job')
lammps_job.inputs.structure = structure
lammps_job.inputs.potential = '1995--Angelo-J-E--Ni-Al-H--LAMMPS--ipr1'  # potential does not contain Cu

lammps_job.inputs.calc.md(temperature=1000)
# or
lammps_job.inputs.calc.static()

potential starts

Issue: In the present implementation I did not find a way to implement a set of calculator nodes such as calc_md, calc_static etc. Having selectable node containers would be very helpful also for other applications.

Make registering of new node repositories easier

Issue: Including a new node repository in the node_library requires to add the following code in the interface.py file:

    @property
    def lammps(self):
        try:
            return self._lammps
        except AttributeError:
            from pyiron_contrib.workflow.node_library.lammps import nodes

            self.register("_lammps", *nodes)
            return self._lammps

It would be nice to have a simpler way to register new node packages. I see two possibilities

• file based (e.g. added to the .pyiron file or in a .libraries file)

• explicit definition in the workflow, e.g.

     Workflow(libraries=['atomistics', 'lammps'], library_path='node_library')
  

Allow substitution of nodes in workflows

from pyiron_contrib.workflow.function import single_value_node

@single_value_node(
    output_labels=['new structure node called']
)
def new_structure_node(structure):
    print ('structure')  # show that the new node has been successfully included
    return structure

wf = Workflow('simple_lammps_calculation')

structure = wf.create.atomistics.Bulk(cubic=True, name="Al", label='structure')

lammps_job = wf.create.lammps.LammpsStaticNode(label='lammps_job')
lammps_job.structure = new_structure_node  # does not work
# lammps_job.structure.replace_with(new_structure_node)  # more explicit syntax, not sure whether needed if above construct works


lammps_job.inputs.structure = structure

wf.run()

potential starts
potential name:  None
start shell
end shell
parse dump file
parse log file





{'lammps_job__engine__output': <pyiron_contrib.workflow.node_library.lammps.ShellOutput at 0x290111050>,
 'lammps_job__collect_output__generic': <pyiron_contrib.workflow.node_library.lammps.GenericOutput at 0x1074adf50>}

Issue: Being able to replace a node in a workflow or macro node would be a powerful extension. It would allow to test and explore new functionality without having to modify the actual code. This would dramatically simplify the way users and developers could implement new ideas and would dramatically lower the barrier to do so (no git, branches etc. needed).

Serialization

In the serialization branch I had implemented an approach to serialize nodes, macros etc. It would be great to have this feature in the main branch, since it allows for an easy connection to Jan's mpipoolexecutor. I also liked very much the pool_iter function, since it provides an easy way to work with tasks without the users noticing it. Do you see any issues in implementing these features in the main branch? They should have no side issues.

[liamhuber]

@JNmpi thanks for all the great feedback and the two new branches!!! I've split each of the topics up into their own issue with a response; this can now be closed and we can close the others off one at a time as they're resolved.

At a very high level, pyiron/pyiron_workflow#12/#859, pyiron/pyiron_workflow#11, pyiron/pyiron_workflow#8, and pyiron/pyiron_workflow#7 are all things we're in broad agreement on and are somewhere in the range of already working/rough idea for solution/specific example for solution; serialization in pyiron/pyiron_workflow#6 is something I'm totally on board with, but I haven't been able to dig into it deep enough to give a meaningful response; and pyiron/pyiron_workflow#9 is something where I am either not understanding fully or disagree.