Refactor internals, add new features and fix several bugs

See changelog for details.
This commit is a squash merge from a development branch.

.github/requirements-old.txt

@@ -1,7 +1,6 @@
 # Ensure changes to these dependencies are reflected in pyproject.toml
 asyncinotify==4.0.9
 attrs==22.1.0
-msgpack==1.0.7
 parse==1.19.1
 path==16.14.0
 rich==13.0.0

.github/workflows/pytest.yaml

@@ -14,7 +14,7 @@ jobs:
   tests:
     strategy:
       matrix:
-        python-version: ["3.11", "3.12"]
+        python-version: ["3.11", "3.13"]
     runs-on: "ubuntu-latest"
     steps:
       - uses: actions/checkout@v4

.pre-commit-config.yaml

@@ -1,6 +1,6 @@
 repos:
 - repo: https://github.com/pre-commit/pre-commit-hooks
-  rev: v4.6.0
+  rev: v5.0.0
   hooks:
     - id: check-added-large-files
     - id: check-ast
@@ -25,12 +25,12 @@ repos:
   hooks:
     - id: remove-crlf
 - repo: https://github.com/astral-sh/ruff-pre-commit
-  rev: v0.6.6
+  rev: v0.8.4
   hooks:
     - id: ruff-format
     - id: ruff
       args: ["--fix", "--show-fixes"]
 - repo: https://github.com/python-jsonschema/check-jsonschema
-  rev: 0.29.2
+  rev: 0.30.0
   hooks:
     - id: check-github-workflows

.vscode/settings.json

@@ -1,5 +1,9 @@
 {
   "python.terminal.activateEnvironment": false,
-  "ltex.additionalRules.motherTongue": "en-US",
-  "ltex.language": "en-US"
+  "ltex.language": "en-US",
+  "python.testing.pytestArgs": [
+    "tests"
+  ],
+  "python.testing.unittestEnabled": false,
+  "python.testing.pytestEnabled": true
 }

docs/advanced_topics/amending_static_inputs.md

@@ -21,7 +21,7 @@ Make `plan.py` executable and run it as follows:
 
 ```bash
 chmod +x plan.py
-stepup -n -w1
+stepup -n 1
 ```
 
 You should get the following terminal output:

docs/advanced_topics/amending_static_inputs/main.sh

@@ -1,7 +1,8 @@
 #!/usr/bin/env bash
 git clean -qdfX .
+export COLUMNS=80
 unset STEPUP_ROOT
-stepup -n -w1 | sed -f ../../clean_stdout.sed > stdout.txt
+stepup -n 1 | sed -f ../../clean_stdout.sed > stdout.txt
 
 # INP: plan.py
 # INP: config.txt

docs/advanced_topics/amending_static_inputs/plan.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 from stepup.core.api import amend, static
 
 static("config.txt")

docs/advanced_topics/amending_static_inputs/stdout.txt

@@ -1,11 +1,12 @@
-  DIRECTOR │ Listening on /tmp/stepup-########/director
+  DIRECTOR │ Listening on /tmp/stepup-########/director (StepUp version 1.3.1.post6)
+   STARTUP │ (Re)initialized boot script
   DIRECTOR │ Launched worker 0
      PHASE │ run
      START │ ./plan.py
    SUCCESS │ ./plan.py
 ─────────────────────────────── Standard output ────────────────────────────────
 This is a config file.
 ────────────────────────────────────────────────────────────────────────────────
-  WORKFLOW │ Dumped to .stepup/workflow.mpk.xz
+  DIRECTOR │ Trying to delete 0 outdated output(s).
   DIRECTOR │ Stopping workers.
   DIRECTOR │ See you!

docs/advanced_topics/amending_steps.md

@@ -1,42 +1,67 @@
 # Amending steps
 
-Every step in StepUp can inform the director process of additional inputs or environment variables it uses, or of additional (volatile) outputs it creates.
-However, defining an amended input will fail when that file is not yet built nor known as a static file.
-In this case, the step can end early and will be rescheduled by the director process when the amended input becomes available.
+Every step in StepUp can inform the director process of additional inputs
+or environment variables it uses, or of additional (volatile) outputs it creates.
+However, defining an amended input will fail
+when that file is not yet built nor known as a static file.
+In this case, the step wil exit early
+and will be rescheduled by the director process when the amended input becomes available.
 
-The [`amend()`][stepup.core.api.amend] function implements this feature and is convenient in various scenarios:
+The [`amend()`][stepup.core.api.amend] function implements this feature
+and is convenient in various scenarios:
 
 - This is particularly useful when a step uses input files that refer to other input files.
   It may happen that some of these other input files still need to be generated by other steps.
   Once built, the other inputs may again refer to more input files, etc.
-  Such dependencies cannot be discovered in advance, simply because not all inputs are available before running the steps that generate them.
+  Such dependencies cannot be discovered in advance,
+  simply because not all inputs are available before running the steps that generate them.
   (This is a common scenario when writing LaTeX documents of which parts are generated by scripts.)
 
-- Another use case is that some steps may take their default configuration from environment variables if some command-line options are missing.
-  In this case, `amend` can be used to specify the environment variables used.
+- Another use case is that some steps may take their default configuration
+  from environment variables if some command-line options are missing.
+  In this case, `amend()` can be used to specify the environment variables used.
 
 - Some steps may produce a list of volatile outputs, some of which are difficult to know upfront.
-  One may specify such volatile outputs with `amend` after they have been created.
+  One may specify such volatile outputs with `amend()` after they have been created.
 
-To the best of our knowledge, there is no equivalent of `amend` in other build tools.
-Some features in Ninja cover what can be achieved with `amend`.
+To use `amend()` effectively, use it as early as possible,
+before accessing additional input files or creating unforeseen output files.
+Also, try to amend as many possible step arguments in a single call.
+This will avoid the following problems:
+
+- Trying to read from a file that hasn't been created yet.
+
+- Unintentionally overwriting a file, e.g. a STATIC file that you have written by hand.
+
+- Performing unnecessary work
+  (because a call to amend may mean that the step is interrupted and restarted later).
+  This is particularly important when the step is time-consuming
+  or when it uses `stepup.core.api` functions to extend the workflow.
+
+- Rescheduling a step multiple times.
+
+To the best of our knowledge, there is no equivalent of `amend()` in other build tools.
+Some features in Ninja cover what can be achieved with `amend()`.
 
 
 ## Example
 
-Example source files: [advanced_topics/amending_steps/](https://github.com/reproducible-reporting/stepup-core/tree/main/docs/advanced_topics/amending_steps)
+Example source files: [`docs/advanced_topics/amending_steps/`](https://github.com/reproducible-reporting/stepup-core/tree/main/docs/advanced_topics/amending_steps)
 
 This example intentionally creates a simple scenario with an amended input.
 This is a somewhat silly example to illustrate the concept.
-You may achieve the same result without amending, because you have full control over all scripts in the example.
+You may achieve the same result without amending,
+because you have full control over all scripts in the example.
 
-Create the following `plan.py`, where the first step is a script that will discover it needs an additional input.
+Create the following `plan.py`,
+where the first step is a script that will discover it needs an additional input.
 
 ```python
 {% include 'advanced_topics/amending_steps/plan.py' %}
 ```
 
-In addition, create a file `input.txt` with some arbitrary contents and the following `step.py` script:
+In addition, create a file `input.txt`
+with some arbitrary contents and the following `step.py` script:
 
 ```python
 {% include 'advanced_topics/amending_steps/step.py' %}
@@ -46,7 +71,7 @@ Make the scripts executable and fire up StepUp to see how it deals with the amen
 
 ```bash
 chmod +x step.py plan.py
-stepup -n -w1
+stepup -n 1
 ```
 
 You should get the following terminal output:
@@ -55,16 +80,17 @@ You should get the following terminal output:
 {% include 'advanced_topics/amending_steps/stdout.txt' %}
 ```
 
-The output shows that `./step.py` first stopped early due to the missing file `input.txt`.
-Once this became clear, StepUp scheduled the optional step to generate the requested input.
-Later, StepUp ran `./step.py` again.
+The output shows that `./step.py` first stops early due to the missing file `input.txt`.
+Once this required input becomes clear,
+StepUp schedules the optional step to generate the requested input.
+Later, StepUp runs `./step.py` again.
 
 
 ## Try the Following
 
 - Run StepUp again without making any changes.
   As expected, all steps are skipped.
-  The `workflow.mpk.xz` file also stores the amended information,
+  The `.stepup/graph.db` file also stores the amended information,
   so these don't need to be rediscovered later.
 
 - Modify the `plan.py` file to include a second amended input, for example, `other.txt`.

docs/advanced_topics/amending_steps/main.sh

@@ -1,7 +1,8 @@
 #!/usr/bin/env bash
 git clean -qdfX .
+export COLUMNS=80
 unset STEPUP_ROOT
-stepup -n -w1 | sed -f ../../clean_stdout.sed > stdout.txt
+stepup -n 1 | sed -f ../../clean_stdout.sed > stdout.txt
 
 # INP: plan.py
 # INP: step.py

docs/advanced_topics/amending_steps/plan.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 from stepup.core.api import static, step
 
 static("step.py")

docs/advanced_topics/amending_steps/stdout.txt

@@ -1,4 +1,5 @@
-  DIRECTOR │ Listening on /tmp/stepup-########/director
+  DIRECTOR │ Listening on /tmp/stepup-########/director (StepUp version 1.3.1.post6)
+   STARTUP │ (Re)initialized boot script
   DIRECTOR │ Launched worker 0
      PHASE │ run
      START │ ./plan.py
@@ -18,6 +19,6 @@ input.txt
 Contents of input.txt:
 You better read this.
 ────────────────────────────────────────────────────────────────────────────────
-  WORKFLOW │ Dumped to .stepup/workflow.mpk.xz
+  DIRECTOR │ Trying to delete 0 outdated output(s).
   DIRECTOR │ Stopping workers.
   DIRECTOR │ See you!

docs/advanced_topics/amending_steps/step.py

@@ -1,15 +1,16 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 from stepup.core.api import amend
 
 # Parse the sources.txt file
 with open("sources.txt") as fh:
     paths_inp = fh.read().split()
 
+# Request the additional input.
+amend(inp=paths_inp)
+
 # Write all files from source.txt to the standard output.
-keep_going = amend(inp=paths_inp)
-if keep_going:
-    # This branch is only executed if the amended input is present.
-    for path_inp in paths_inp:
-        print(f"Contents of {path_inp}:")
-        with open(path_inp) as fh:
-            print(fh.read())
+# This part is reachable only if the amended input is present.
+for path_inp in paths_inp:
+    print(f"Contents of {path_inp}:")
+    with open(path_inp) as fh:
+        print(fh.read())

docs/advanced_topics/blocked_steps.md

@@ -31,7 +31,7 @@ Blocking steps has some consequences:
 
 ## Example
 
-Example source files: [advanced_topics/blocked_steps/](https://github.com/reproducible-reporting/stepup-core/tree/main/docs/advanced_topics/blocked_steps)
+Example source files: [`docs/advanced_topics/blocked_steps/`](https://github.com/reproducible-reporting/stepup-core/tree/main/docs/advanced_topics/blocked_steps)
 
 The following `plan.py` illustrates the blocking mechanism.
 The copy commands are too simple and cheap to justify blocking,
@@ -45,7 +45,7 @@ Make this plan executable and run it with StepUp:
 
 ```bash
 chmod +x plan.py
-stepup -n -w1
+stepup -n 1
 ```
 
 You should get the following terminal output:

docs/advanced_topics/blocked_steps/main.sh

@@ -1,6 +1,7 @@
 #!/usr/bin/env bash
 git clean -qdfX .
+export COLUMNS=80
 unset STEPUP_ROOT
-stepup -n -w1 | sed -f ../../clean_stdout.sed > stdout.txt
+stepup -n 1 | sed -f ../../clean_stdout.sed > stdout.txt
 
 # INP: plan.py

docs/advanced_topics/blocked_steps/plan.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 from stepup.core.api import copy, step
 
 step("echo hello > ${out}", out="a.txt")

docs/advanced_topics/blocked_steps/stdout.txt

@@ -1,15 +1,16 @@
-  DIRECTOR │ Listening on /tmp/stepup-########/director
+  DIRECTOR │ Listening on /tmp/stepup-########/director (StepUp version 1.3.1.post6)
+   STARTUP │ (Re)initialized boot script
   DIRECTOR │ Launched worker 0
      PHASE │ run
      START │ ./plan.py
    SUCCESS │ ./plan.py
      START │ echo hello > a.txt
    SUCCESS │ echo hello > a.txt
-   WARNING │ 2 steps remain pending due to blocked steps
+   WARNING │ 2 step(s) remained pending due to blocked steps
 ──────────────────────────────── Blocked steps ─────────────────────────────────
 step:cp -aT a.txt b.txt
 ────────────────────────────────────────────────────────────────────────────────
    WARNING │ Skipping cleanup due to incomplete build.
-  WORKFLOW │ Dumped to .stepup/workflow.mpk.xz
+   WARNING │ Check logs: .stepup/warning.log
   DIRECTOR │ Stopping workers.
   DIRECTOR │ See you!

docs/advanced_topics/cyclic_dependencies.md

@@ -1,9 +1,13 @@
 # Cyclic dependencies
 
-Cyclic dependencies are defined in StepUp as closed loops in the "supplier ➜ consumer" graph.
-One may construct these in a `plan.py`, which will result in an error message.
+Cyclic dependencies are defined in StepUp as closed loops in the dependency graph.
+Formally, such a loop is defined as a set of edges that can be followed from supplier to consumer,
+such that one arrives back at the starting point.
+If you construct cyclic dependencies in a `plan.py`, an error message is generated.
 
-In theory, one could also have cycles in the "creator ➜ product" graph, but these are excluded by construction and therefore not relevant to discuss in the tutorial.
+In theory, one could also have cycles in the provenance graph,
+but it is not possible to create such loops with the functions in
+[stepup.core.api](../reference/stepup.core.api.md).
 
 
 ## Example
@@ -18,7 +22,7 @@ Make the plan executable and give it a try as follows:
 
 ```bash
 chmod +x plan.py
-stepup -n -w1
+stepup -n 1
 ```
 
 You will get the following terminal output showing that this plan won't work.

docs/advanced_topics/cyclic_dependencies/main.sh

@@ -1,6 +1,7 @@
 #!/usr/bin/env bash
 git clean -qdfX .
+export COLUMNS=80
 unset STEPUP_ROOT
-stepup -n -w1 | sed -f ../../clean_stdout.sed > stdout.txt
+stepup -n 1 | sed -f ../../clean_stdout.sed | sed -e 's|/home/.*/stepup-core/||' > stdout.txt
 
 # INP: plan.py

docs/advanced_topics/cyclic_dependencies/plan.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 from stepup.core.api import copy
 
 copy("a.txt", "b.txt")