Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Spec and test #70

Merged
merged 41 commits into from
Nov 11, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
41 commits
Select commit Hold shift + click to select a range
d0f7704
Update channel docs and test to them
liamhuber Nov 6, 2023
4f76dc5
Never allow input data updates while the parent node is running
liamhuber Nov 6, 2023
a9381c8
Update Node docs and make tests for the base class
liamhuber Nov 6, 2023
f138999
Draft module-level overview for API docs/use in readme
liamhuber Nov 6, 2023
70f0d98
Test single point of import
liamhuber Nov 6, 2023
35b5c0e
:bug: fix the expected error to be raised
liamhuber Nov 6, 2023
300ad1c
Update IO docstring
liamhuber Nov 6, 2023
5f567f0
Update Function docstring
liamhuber Nov 6, 2023
7fd250e
Update function tests
liamhuber Nov 6, 2023
37c2262
Test the call aliases
liamhuber Nov 6, 2023
dbddd14
Update Composite docs
liamhuber Nov 10, 2023
7739aeb
Improve working directory test
liamhuber Nov 10, 2023
aa01945
Update composite promises
liamhuber Nov 10, 2023
27ce1d5
:bug: actually iterate over child nodes
liamhuber Nov 10, 2023
fff90cf
Pull up the IO rebuild check after node replacement
liamhuber Nov 10, 2023
a02adee
Test the promises made in Composite directly
liamhuber Nov 10, 2023
bcce14d
Make Macro's graph_creator available as a class method
liamhuber Nov 10, 2023
7d9238e
Use setUp
liamhuber Nov 10, 2023
6e88ef3
Define and test Macro promises
liamhuber Nov 10, 2023
8a31793
Make and test Workflow promises
liamhuber Nov 10, 2023
53557cf
:bug: handle bidict None duplication
liamhuber Nov 11, 2023
2eda950
:bug: check for instance type
liamhuber Nov 11, 2023
811fb79
Apply de-duplication of None values to both I and O
liamhuber Nov 11, 2023
6eab6ae
:bug: make sure the map is persistent
liamhuber Nov 11, 2023
0577034
Pull the bijectivity tests up to Composite
liamhuber Nov 11, 2023
2554682
Reformat Function as promises
liamhuber Nov 11, 2023
29a4b29
Finish parent promises
liamhuber Nov 11, 2023
414d1c5
Update node docs to the "promises" format
liamhuber Nov 11, 2023
8f2b40c
Reorder node tests to better match the order of the promises
liamhuber Nov 11, 2023
cb47ed2
Remove redundant topology tools
liamhuber Nov 11, 2023
d674887
Make method private
liamhuber Nov 11, 2023
ab2435a
Format black
pyiron-runner Nov 11, 2023
75881f7
Add a short sleep
liamhuber Nov 11, 2023
9a47ca9
Update example notebook
liamhuber Nov 11, 2023
724bce7
Rename and re-run the example notebook
liamhuber Nov 11, 2023
290a282
Add quickstart notebook
liamhuber Nov 11, 2023
2f2c79b
Add badges to readme
liamhuber Nov 11, 2023
fc6dcd0
Update overview
liamhuber Nov 11, 2023
3b65a35
Add tailing readme sections
liamhuber Nov 11, 2023
4538443
Update readme code example
liamhuber Nov 11, 2023
4ec4d72
Update the docs badge
liamhuber Nov 11, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
100 changes: 85 additions & 15 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,31 +1,101 @@
# pyiron_workflow

[![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/pyiron/pyiron_workflow/HEAD)
[![License](https://img.shields.io/badge/License-BSD_3--Clause-blue.svg)](https://opensource.org/licenses/BSD-3-Clause)
[![Codacy Badge](https://app.codacy.com/project/badge/Grade/0b4c75adf30744a29de88b5959246882)](https://app.codacy.com/gh/pyiron/pyiron_workflow/dashboard?utm_source=gh&utm_medium=referral&utm_content=&utm_campaign=Badge_grade)
[![Coverage Status](https://coveralls.io/repos/github/pyiron/pyiron_workflow/badge.svg?branch=main)](https://coveralls.io/github/pyiron/pyiron_workflow?branch=main)

[//]: # ([![Documentation Status](https://readthedocs.org/projects/pyiron-workflow/badge/?version=latest)](https://pyiron-workflow.readthedocs.io/en/latest/?badge=latest))

[![Anaconda](https://anaconda.org/conda-forge/pyiron_workflow/badges/version.svg)](https://anaconda.org/conda-forge/pyiron_workflow)
[![Last Updated](https://anaconda.org/conda-forge/pyiron_workflow/badges/latest_release_date.svg
)](https://anaconda.org/conda-forge/pyiron_workflow)
[![Platform](https://anaconda.org/conda-forge/pyiron_workflow/badges/platforms.svg)](https://anaconda.org/conda-forge/pyiron_workflow)
[![Downloads](https://anaconda.org/conda-forge/pyiron_workflow/badges/downloads.svg)](https://anaconda.org/conda-forge/pyiron_workflow)

## Overview

This repository is home to the pyiron code for structuring workflows as graph objects, with different computational elements as nodes and data and execution signals travelling along edges. It is currently in an alpha state, changing quickly, and not yet feature-complete.
`pyiron_workflow` is a framework for constructing workflows as computational graphs from simple python functions. Its objective is to make it as easy as possible to create reliable, reusable, and sharable workflows, with a special focus on research workflows for HPC environments.

Nodes are formed from python functions with simple decorators, and the resulting nodes can have their data inputs and outputs connected.

By allowing (but not demanding, in the case of data DAGs) users to specify the execution flow, both cyclic and acyclic graphs are supported.

By scraping type hints from decorated functions, both new data values and new graph connections are (optionally) required to conform to hints, making workflows strongly typed.

Individual node computations can be shipped off to parallel processes for scalability. (This is an alpha-feature at time of writing and limited to single core parallel python processes; full support of [`pympipool`](https://github.com/pyiron/pympipool) is under active development)

Once you're happy with a workflow, it can be easily turned it into a macro for use in other workflows. This allows the clean construction of increasingly complex computation graphs by composing simpler graphs.

Nodes (including macros) can be stored in plain text, and registered by future workflows for easy access. This encourages and supports an ecosystem of useful nodes, so you don't need to re-invent the wheel. (This is an alpha-feature, with full support of [FAIR](https://en.wikipedia.org/wiki/FAIR_data) principles for node packages planned.)

## Example

`pyiron_workflow` offers a single-point-of-entry in the form of the `Workflow` object, and uses decorators to make it easy to turn regular python functions into "nodes" that can be put in a computation graph.

Nodes can be used by themselves and -- other than being "delayed" in that their computation needs to be requested after they're instantiated -- they feel an awful lot like the regular python functions they wrap:

```python
from pyiron_workflow import Workflow

@Workflow.wrap_as.single_value_node()
def add_one(x):
return x + 1

## The absolute basics
add_one(add_one(add_one(x=0)))()
>>> 3
```

`pyiron_workflow` offers a single-point-of-entry in the form of the `Workflow` object, and uses decorators to make it easy to turn regular python functions into "nodes" that can be put in a computation graph:
But the intent is to collect them together into a workflow and leverage existing nodes:

```python
from pyiron_workflow import Workflow

@Workflow.wrap_as.function_node("sum")
def x_plus_y(x: int = 0, y: int = 0) -> int:
return x + y
@Workflow.wrap_as.single_value_node()
def add_one(x):
return x + 1

@Workflow.wrap_as.macro_node()
def add_three_macro(macro):
macro.start = add_one()
macro.middle = add_one(x=macro.start)
macro.end = add_one(x=macro.middle)
macro.inputs_map = {"start__x": "x"}
macro.outputs_map = {"end__x + 1": "y"}

wf = Workflow("my_workflow")
wf.a1 = x_plus_y()
wf.a2 = x_plus_y()
wf.b = x_plus_y(x=wf.a1.outputs.sum, y=wf.a2.outputs.sum)
Workflow.register(
"plotting",
"pyiron_workflow.node_library.plotting"
)

out = wf(a1__x=0, a1__y=1, a2__x=2, a2__y=3)
out.b__sum
>>> 6
wf = Workflow("add_5_and_plot")
wf.add_one = add_one()
wf.add_three = add_three_macro(x=wf.add_one)
wf.plot = wf.create.plotting.Scatter(
x=wf.add_one,
y=wf.add_three.outputs.y
)

wf.draw()
diagram = wf.draw()

import numpy as np
fig = wf(add_one__x=np.arange(5)).plot__fig
```

![](docs/_static/demo.png)
Which gives the workflow `diagram`

![](docs/_static/readme_diagram.png)

And the resulting `fig`

![](docs/_static/readme_shifted.png)

## Installation

`conda install -c conda-forge pyiron_workflow`

To unlock the associated node packages and ensure that the demo notebooks run, also make sure your conda environment has the packages listed in our [notebooks dependencies](.ci_support/environment-notebooks.yml)

## Learning more

Check out the demo [notebooks](notebooks), read through the docstrings, and don't be scared to raise an issue on this GitHub repo!
Binary file removed docs/_static/demo.png
Binary file not shown.
Binary file added docs/_static/readme_diagram.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/_static/readme_shifted.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Loading