Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FEAT: ExecutionPlan remember compilation results for elaborate plots #29

Open
wants to merge 94 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
94 commits
Select commit Hold shift + click to select a range
617e577
FIX(build): py2 needs pinning networkx-2.2
ankostis Sep 28, 2019
f58d148
FIX(#13): BUG in plot-diagram writtin from PY2 era,
ankostis Sep 29, 2019
c75a2c0
doc(#13): sample code to plot workflow diagram in intro
ankostis Sep 29, 2019
a005bd6
enh(plot): provide help msg on supported file-exts
ankostis Sep 29, 2019
52c0d77
enh(test): + x2 TC breaking UNSATISFIED operations...
ankostis Sep 30, 2019
bc4c221
ENH(net,#18): ignore UN-SATISFIABLE operations with partial inputs
ankostis Sep 29, 2019
b8daa07
refact(net): drop old `dag` nx-package
ankostis Oct 1, 2019
12bdfe4
ENH(core): ORDERED SETs for DETERMINISTIC results
ankostis Oct 1, 2019
b8377ca
merge UNASTIFIABLE + ORDERED_SETs
ankostis Oct 1, 2019
489b32c
refact(net): simpilify del-instruction loop
ankostis Oct 1, 2019
b102d44
REFACT(unsatisfied): doubly-recursive func --> loop on topo-sorted
ankostis Oct 1, 2019
de02885
test(dag,#25): FAILing TC for overriding intermediate data
ankostis Oct 2, 2019
e1454fd
test(dag,#24): FAILing TC for over-pruning inetermediates when outs a…
ankostis Oct 2, 2019
3736738
MERGE (prune_unsatified, ordered_sets) into fix-pruning ...
ankostis Oct 2, 2019
c273068
DOC(net): explain DAG solution & compilation...
ankostis Oct 2, 2019
16d42f1
TEST(prune): +Failing x2 TCs multi-out must run but not...
ankostis Oct 2, 2019
b92f103
refact(dag): call compile() before compute.compute...
ankostis Oct 2, 2019
6d1884e
test(dag): +TC checking DeleteInst vary when inputs change
ankostis Oct 2, 2019
619cae7
ENH(net): move compile() after SOLVE DAG ...
ankostis Oct 2, 2019
eff351d
REFACT(NET) COMPILE+COMPUTE...
ankostis Oct 2, 2019
d959485
doc(net): explain new DAG SOLUTION
ankostis Oct 2, 2019
17eb2fd
FIX(net): new Ops invalidate execution-plan cache...
ankostis Oct 3, 2019
0830b7c
ENH(DAG): NEW SOLVER
ankostis Oct 3, 2019
32409f6
enh(build): replace numpy with pytest...
ankostis Oct 3, 2019
f606ed1
feat(build): add pip-extras [test]
ankostis Oct 3, 2019
0dc1293
WIP/FIX(prune,#26): PIN intermediate inputs if operation before must run
ankostis Oct 3, 2019
06f6554
REFACT(net): part 3 of new dag-solver & pin refactoring
ankostis Oct 4, 2019
1cc733e
enh(CI): +PY3.6 where dicts are stable
ankostis Oct 4, 2019
cd1370b
TEST(plot,ci): test plotting; pip install extras in Travis
ankostis Sep 29, 2019
f676662
fix(plot): don't create file on unsupported formats
ankostis Sep 29, 2019
65d1816
enh(plot.TC): expose supported writers and TC on them
ankostis Sep 29, 2019
d403783
WIP/FIX(PIN): PARALLEL DELs decide on PRUNED-dag (not full)...
ankostis Oct 4, 2019
64e0028
WIP/+ExecPlan class...
ankostis Oct 4, 2019
4e55b30
enh(build,ci): use pytest in travis
ankostis Oct 3, 2019
47b50f6
fix(plot): NetOp did not return pydot instance
ankostis Oct 4, 2019
b1d02a1
refact(plot): extract plot function out of Network class...
ankostis Oct 4, 2019
c11af2a
fix(plot): matplotlib plot was failing in PY3 due IO io misuse
ankostis Oct 4, 2019
344490b
FEAT(plot): overlay Execution STEPS on diagrams
ankostis Oct 5, 2019
23ef81e
ENH(plot): +inputs, +outputs, +solution modify plotting
ankostis Oct 5, 2019
4e8601c
refact(plot.TC): move plot tests to early beggining
ankostis Oct 5, 2019
834a8b0
doc(plot): tell supported formats in doctest, +TC
ankostis Oct 5, 2019
c2e28a4
doc(plot): add legend & example; docstring in netop.plot()
ankostis Oct 5, 2019
e38c8ad
enh(plot): mark optional "needs"
ankostis Oct 5, 2019
d855bf6
ENH(plot): visual enhamcents on nodes & edges
ankostis Oct 5, 2019
ca5d243
test(plot): enhance plot test to try all #13 features;
ankostis Oct 5, 2019
f25f189
test(optional): +x1 TC to check selective optionals with same out...
ankostis Oct 5, 2019
a2de9ef
doc(plot); explain also params in user-facing API
ankostis Oct 5, 2019
dc5a21a
FIX(PLOT.TC): TC was always testing PNG, ...
ankostis Oct 5, 2019
782d9b9
fix(plot): don't require Matplotlib if no Window asked
ankostis Oct 5, 2019
7d389c3
test(plot): check also matplotlib show=True
ankostis Oct 5, 2019
3fe0b40
ENH(plot): return SVG rendered in JUPYTER, ...
ankostis Oct 5, 2019
1471551
refact(plot.TC): avoid writting multiple temp-files
ankostis Oct 5, 2019
af7ae0f
MERGE PYTEST into PLOT to parametrize TCs.
ankostis Oct 5, 2019
b440196
fix(build): reuse dependencies definitions
ankostis Oct 5, 2019
bde9b64
REFACT(plot.TC): PYTESTize and parametrize
ankostis Oct 5, 2019
8e361e6
REFACT(PLOT): MOVE PLOT in own module
ankostis Oct 5, 2019
b08a363
DROP PY3.4 - add PY3.6, PY3.7...
ankostis Oct 5, 2019
3a87959
refact(plot): separate graphviz building from IO
ankostis Oct 5, 2019
17f18c2
MERGE ENHANCED-PLOTS into REFACT-ExecutionPlan to debug issues
ankostis Oct 5, 2019
4d250d3
DROP(net): list/show layers not needed, repr() is ok
ankostis Oct 5, 2019
18191e4
ENH(plot,net): +plot() on ExecPlan; +PlotMixin ...
ankostis Oct 5, 2019
7341529
enh(plan): repr()
ankostis Oct 6, 2019
231ada5
refact(plot): reorder formats-list with io
ankostis Oct 6, 2019
80f110a
FIX(plot): distinguish Del/Pin cmds; +choice utils
ankostis Oct 6, 2019
32eaa80
refact(plot): inline imports, not to cycle with base...
ankostis Oct 6, 2019
4b70cfb
refact(plot): move PlotMixin base-->plot module to group edit
ankostis Oct 6, 2019
b4fa5e0
enh(plot): +`title` arg at the bottom
ankostis Oct 6, 2019
c6f2155
FIX(plot): failing if steps not a list/ is none
ankostis Oct 6, 2019
ec69090
ENH(plan,plot): executed operations drawn as filled
ankostis Oct 6, 2019
24a3d1e
FIX(net): revived last_plan was never set (HEAD~13: 64e0028)
ankostis Oct 6, 2019
77fc887
refact(plot): reodred build-dot utils above use
ankostis Oct 6, 2019
da087df
doc(plot): move centrally all API doc on PlotMixin; Style
ankostis Oct 6, 2019
ac73bbb
FEAT(plot,plan): +yellow broken links on original graph
ankostis Oct 6, 2019
c2829a3
refact(plot): PlotMixin --> Plotter, _plotter() --> _plot()
ankostis Oct 6, 2019
c61947b
TEST(PLOT): Check also ExecPlan
ankostis Oct 6, 2019
434d4e2
REFACT(PLOT): Plotter builds dot & renders...
ankostis Oct 6, 2019
dabc787
ENH(plan.polt): CLUSTER pruned nodes
ankostis Oct 6, 2019
c5b9167
FIX(plot): VISUAL fixes & COLOR-palettes:
ankostis Oct 7, 2019
1d443ce
ENH(netop): mark all its needs as OPTIONAL
ankostis Oct 7, 2019
77bec49
FIX(NET): were FORGETTING PRUNED ASKED-OUTPUTs...
ankostis Oct 7, 2019
89e4edb
ENH(plot,TC): pipeline delegates to last_plan, if exists
ankostis Oct 7, 2019
cef7526
FIX(plot.TC): formats-TC were testing dot-file, not rendered...
ankostis Oct 7, 2019
f9b2415
enh(net): prune also isolate data
ankostis Oct 7, 2019
fce0515
FIX(<PY3.5): ORDERED DiGRAPH for old Python to fix TCs
ankostis Oct 7, 2019
d8ba34f
ENH(plot): switch oval<-->circle ops/netops
ankostis Oct 7, 2019
e8fec22
feat(plot): +legend & TC
ankostis Oct 7, 2019
53c6ce5
DOC(PLOT): +sample GRAPH & LEGEND into README...
ankostis Oct 7, 2019
ff754a1
DROP(plot) jupyter kw, simpler to monkeypath Dot class...
ankostis Oct 7, 2019
f787f26
FIX(plot): LEGEND mistakes, SVGize, egg graphs, ...
ankostis Oct 8, 2019
5cf7189
FIX(DOC): don't use `graph` in sample code, crash DOT, +more ...
ankostis Oct 8, 2019
7f637fe
chore(git): .gitignore plot images in root folder
ankostis Oct 8, 2019
ae01163
chore(TCs): mark SLOW tests, when in hurry, +`setup.cfg`
ankostis Oct 8, 2019
64838a5
FIX(TCs): MERGE TCs were not ASSERTING...
ankostis Oct 8, 2019
8c41066
FIX(MERGE): broken by NEW_DAG_SOLVER (#26 ...
ankostis Oct 8, 2019
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -52,3 +52,8 @@ docs/_build/

# PyBuilder
target/

# Plots genersated when running sample code
/*.png
/*.svg
/*.pdf
15 changes: 12 additions & 3 deletions .travis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,19 +2,28 @@ language: python

python:
- "2.7"
- "3.4"
- "3.5"
- "3.6"
- "3.7"

addons:
apt:
packages:
- graphviz


install:
- pip install Sphinx sphinx_rtd_theme codecov packaging
- "python -c $'import os, packaging.version as version\\nv = version.parse(os.environ.get(\"TRAVIS_TAG\", \"1.0\")).public\\nwith open(\"VERSION\", \"w\") as f: f.write(v)'"
- python setup.py install
- pip install -e .[test]
- cd docs
- make clean html
- cd ..

script:
- python setup.py nosetests --with-coverage --cover-package=graphkit
- pytest -v --cov=graphkit
# In case you adopt -m 'not slow' in setup.cfg.
#- pytest -vm slow --cov=graphkit

deploy:
provider: pypi
Expand Down
32 changes: 26 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@

> It's a DAG all the way down

![Sample graph](docs/source/images/test_pruning_not_overrides_given_intermediate-asked.png "Sample graph")

## Lightweight computation graphs for Python

GraphKit is a lightweight Python module for creating and running ordered graphs of computations, where the nodes of the graph correspond to computational operations, and the edges correspond to output --> input dependencies between those operations. Such graphs are useful in computer vision, machine learning, and many other domains.
Expand All @@ -14,9 +16,12 @@ GraphKit is a lightweight Python module for creating and running ordered graphs

Here's how to install:

```
pip install graphkit
```
pip install graphkit

OR with dependencies for plotting support (and you need to install [`Graphviz`](https://graphviz.org)
program separately with your OS tools)::

pip install graphkit[plot]

Here's a Python script with an example GraphKit computation graph that produces multiple outputs (`a * b`, `a - a * b`, and `abs(a - a * b) ** 3`):

Expand All @@ -30,27 +35,42 @@ def abspow(a, p):
return c

# Compose the mul, sub, and abspow operations into a computation graph.
graph = compose(name="graph")(
graphop = compose(name="graphop")(
operation(name="mul1", needs=["a", "b"], provides=["ab"])(mul),
operation(name="sub1", needs=["a", "ab"], provides=["a_minus_ab"])(sub),
operation(name="abspow1", needs=["a_minus_ab"], provides=["abs_a_minus_ab_cubed"], params={"p": 3})(abspow)
)

# Run the graph and request all of the outputs.
out = graph({'a': 2, 'b': 5})
out = graphop({'a': 2, 'b': 5})

# Prints "{'a': 2, 'a_minus_ab': -8, 'b': 5, 'ab': 10, 'abs_a_minus_ab_cubed': 512}".
print(out)

# Run the graph and request a subset of the outputs.
out = graph({'a': 2, 'b': 5}, outputs=["a_minus_ab"])
out = graphop({'a': 2, 'b': 5}, outputs=["a_minus_ab"])

# Prints "{'a_minus_ab': -8}".
print(out)
```

As you can see, any function can be used as an operation in GraphKit, even ones imported from system modules!


## Plotting

For debugging the above graph-operation you may plot it using these methods:

```python
graphop.plot(show=True, solution=out) # open a matplotlib window with solution values in nodes
graphop.plot("intro.svg") # other supported formats: png, jpg, pdf, ...
```

![Intro graph](docs/source/images/intro.png "Intro graph")
![Graphkit Legend](docs/source/images/GraphkitLegend.svg "Graphkit Legend")

> **TIP:** The `pydot.Dot` instances returned by `plot()` are rendered as SVG in *Jupyter/IPython*.

# License

Code licensed under the Apache License, Version 2.0 license. See LICENSE file for terms.
28 changes: 14 additions & 14 deletions docs/source/graph_composition.rst
Original file line number Diff line number Diff line change
Expand Up @@ -30,15 +30,15 @@ The simplest use case for ``compose`` is assembling a collection of individual o
return c

# Compose the mul, sub, and abspow operations into a computation graph.
graph = compose(name="graph")(
graphop = compose(name="graphop")(
operation(name="mul1", needs=["a", "b"], provides=["ab"])(mul),
operation(name="sub1", needs=["a", "ab"], provides=["a_minus_ab"])(sub),
operation(name="abspow1", needs=["a_minus_ab"], provides=["abs_a_minus_ab_cubed"], params={"p": 3})(abspow)
)

The call here to ``compose()()`` yields a runnable computation graph that looks like this (where the circles are operations, squares are data, and octagons are parameters):
The call here to ``compose()`` yields a runnable computation graph that looks like this (where the circles are operations, squares are data, and octagons are parameters):

.. image:: images/example_graph.svg
.. image:: images/intro.svg


.. _graph-computations:
Expand All @@ -49,18 +49,18 @@ Running a computation graph
The graph composed in the example above in :ref:`simple-graph-composition` can be run by simply calling it with a dictionary argument whose keys correspond to the names of inputs to the graph and whose values are the corresponding input values. For example, if ``graph`` is as defined above, we can run it like this::

# Run the graph and request all of the outputs.
out = graph({'a': 2, 'b': 5})
out = graphop({'a': 2, 'b': 5})

# Prints "{'a': 2, 'a_minus_ab': -8, 'b': 5, 'ab': 10, 'abs_a_minus_ab_cubed': 512}".
print(out)

Producing a subset of outputs
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

By default, calling a graph on a set of inputs will yield all of that graph's outputs. You can use the ``outputs`` parameter to request only a subset. For example, if ``graph`` is as above::
By default, calling a graph-operation on a set of inputs will yield all of that graph's outputs. You can use the ``outputs`` parameter to request only a subset. For example, if ``graphop`` is as above::

# Run the graph and request a subset of the outputs.
out = graph({'a': 2, 'b': 5}, outputs=["a_minus_ab"])
# Run the graph-operation and request a subset of the outputs.
out = graphop({'a': 2, 'b': 5}, outputs=["a_minus_ab"])

# Prints "{'a_minus_ab': -8}".
print(out)
Expand All @@ -70,17 +70,17 @@ When using ``outputs`` to request only a subset of a graph's outputs, GraphKit e
Short-circuiting a graph computation
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

You can short-circuit a graph computation, making certain inputs unnecessary, by providing a value in the graph that is further downstream in the graph than those inputs. For example, in the graph we've been working with, you could provide the value of ``a_minus_ab`` to make the inputs ``a`` and ``b`` unnecessary::
You can short-circuit a graph computation, making certain inputs unnecessary, by providing a value in the graph that is further downstream in the graph than those inputs. For example, in the graph-operation we've been working with, you could provide the value of ``a_minus_ab`` to make the inputs ``a`` and ``b`` unnecessary::

# Run the graph and request a subset of the outputs.
out = graph({'a_minus_ab': -8})
# Run the graph-operation and request a subset of the outputs.
out = graphop({'a_minus_ab': -8})

# Prints "{'a_minus_ab': -8, 'abs_a_minus_ab_cubed': 512}".
print(out)

When you do this, any ``operation`` nodes that are not on a path from the downstream input to the requested outputs (i.e. predecessors of the downstream input) are not computed. For example, the ``mul1`` and ``sub1`` operations are not executed here.

This can be useful if you have a graph that accepts alternative forms of the same input. For example, if your graph requires a ``PIL.Image`` as input, you could allow your graph to be run in an API server by adding an earlier ``operation`` that accepts as input a string of raw image data and converts that data into the needed ``PIL.Image``. Then, you can either provide the raw image data string as input, or you can provide the ``PIL.Image`` if you have it and skip providing the image data string.
This can be useful if you have a graph-operation that accepts alternative forms of the same input. For example, if your graph-operation requires a ``PIL.Image`` as input, you could allow your graph to be run in an API server by adding an earlier ``operation`` that accepts as input a string of raw image data and converts that data into the needed ``PIL.Image``. Then, you can either provide the raw image data string as input, or you can provide the ``PIL.Image`` if you have it and skip providing the image data string.

Adding on to an existing computation graph
------------------------------------------
Expand Down Expand Up @@ -109,17 +109,17 @@ Sometimes you will have two computation graphs---perhaps ones that share operati

combined_graph = compose(name="combined_graph")(graph1, graph2)

However, if you want to combine graphs that share operations and don't want to pay the price of running redundant computations, you can set the ``merge`` parameter of ``compose()`` to ``True``. This will consolidate redundant ``operation`` nodes (based on ``name``) into a single node. For example, let's say we have ``graph``, as in the examples above, along with this graph::
However, if you want to combine graphs that share operations and don't want to pay the price of running redundant computations, you can set the ``merge`` parameter of ``compose()`` to ``True``. This will consolidate redundant ``operation`` nodes (based on ``name``) into a single node. For example, let's say we have ``graphop``, as in the examples above, along with this graph::

# This graph shares the "mul1" operation with graph.
another_graph = compose(name="another_graph")(
operation(name="mul1", needs=["a", "b"], provides=["ab"])(mul),
operation(name="mul2", needs=["c", "ab"], provides=["cab"])(mul)
)

We can merge ``graph`` and ``another_graph`` like so, avoiding a redundant ``mul1`` operation::
We can merge ``graphop`` and ``another_graph`` like so, avoiding a redundant ``mul1`` operation::

merged_graph = compose(name="merged_graph", merge=True)(graph, another_graph)
merged_graph = compose(name="merged_graph", merge=True)(graphop, another_graph)

This ``merged_graph`` will look like this:

Expand Down
150 changes: 150 additions & 0 deletions docs/source/images/GraphkitLegend.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading