Draft lesson outline

[reid-a]

In light of the decision made at the April co-working meeting, viz. that we should not ever run the Amdahl code on the head node or on learner's laptops, some revisions are appropriate for the lesson content relative to the old lesson.

Initial Amdahl runs will be done on the cluster via batch files, then Snakemake basics will be explored with "Hello, world" executables on the head node, and then the two will be combined. Graduates of HPC Intro will know how to do the first thing, but realistically we should add some time for refreshing this knowledge for actual human learners with imperfect retention.

So a high-level version of the set of tasks now maybe looks something like this:

1. Run the amdahl code on the cluster. Learners should be able to identify what output files the code generates, and know what data is in them.
2. Introduce the Snakemake tool, and construct a "Hello, world" snakefile. Learners should be able to correctly predict whether the rule in the snakefile will fire or not, based on the presence and currency of the output file.
3. Generate a multi-rule snakefile, with a dependency, to introduce the concept of the task graph, and illustrate that the order of operations. We can continue to use "Hello, world" level executables here. Learners should be able to correctly predict which snakemake rules will fire on an invocation, and in what order, based on the presence and currency of the output targets.
4. Generate a single-rule snakefile that runs on the cluster. At first, manually specify all the cluster stuff, like the partition name and so forth, to foreground it. Learners should be able to predict how their snakefile will dispatch to the cluster, and predict the location and character of the resulting outputs.
5. Introduce the cluster config file, and populate it for the local cluster. Repeat the task of the previous lesson, but with the cluster info implicit in the configuration. Same learner capability, I guess?
6. Dispatch multiple jobs to the cluster via snakemake. Observe that the snakemake process itself remains active on the head node until the jobs are finished. (Deal with the thing where a cluster rule exits at dispatch-time, but the target doesn't appear until later?) Once this content is more developed, the goal can probably be clarified, beyond the obvious "learners should be able to correctly predict the sequence of operations that will result from running their snakefile", which is the emerging theme here.

From here, the tasks get a bit more murky in my mind, but the two beats to hit include:

8. Plan and execute the workflow that generates the data needed for the Amdahl plot.
9. Actually generate the Amdahl plot, and observe and appreciate the diminishing returns to increased parallelism.

[ocaisa]

Need to include localrule to ensure simple things are done locally, only "heavy" work on the cluster (ref. jdblischak/smk-simple-slurm#19)

[tkphd]

(docs on localrule)

[ocaisa]

There are actually at least a few things in the docs that we should be aware of:

• https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html#benchmark-rules
• https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html#parameter-space-exploration
• https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html#mpi-support

[reid-a]

I think I have some of this in hand. It looks like in the absence of a profile, rules are run locally by default, at least in my initial experiments -- snakemake won't try to run sbatch unless you tell it to either via a profile or with the --slurm command-line argument.

[ocaisa]

@reid-a I think this can be closed?

[reid-a]

Agreed!