adding notebook tests

[rfl-urbaniak]

Partially addresses an existing issue #283 (which shouldn't be resolved automatically upon merging this PR).
For now with tutorial_i, as smoke screen conditioning needs to be added to other notebooks to speed them up. I can keep going with the other notebooks, but for now, it seemed sensible to touch base and agree on the method.
Sam was fantastic at teaching me to include smoke tests, helping me convert my Python script to a shell script, and commenting on an earlier version of my workflow file. Thanks @SamWitty! After our conversation, the following transpired:

• The notebook dependencies are non-trivial and I assume need to be separated from the build requirements (which should not include things like seaborn or matplotlib ). I added a separate notebook requirements.txt in docs/source, and referred to it and to requirements.txt in test_notebooks.yml. The workflow also needed to use ts-graphviz/setup-graphviz@v1 for the notebooks to be able to produce our precious model graphs.

[SamWitty]

@rfl-urbaniak , now that you've merged in #318 would you like @eb8680 to review this PR or are you planning on adding more notebooks first?

[rfl-urbaniak]

I've added backdoor.ipynb and will keep adding notebooks tomorrow. Will ask for review and change status when done.

[rfl-urbaniak]

Includes notebook tests for backdoor, cevae, dr_learner, mediation, sdid, slc and tutorial.

Two excluded notebooks are deepscm and sciplex. I took a first stab at creating mock datasets by subsetting, but:

1. The Morpho-MNIST link is ambiguous, as the repo contains multiple versions of the datasets. Crucially, in each of those, train-morpho.csv fails to contain a variable named intensity, used to construct raw_intensity in the code.

2. The data used in sciplex when processed, crashes my kernel, and so I cannot generate the .h5 dataset which I would subset to produce a mock dataset.

If no revisions are suggested, this PR can be merged, and we can deal with these two remaining notebooks later.

[rfl-urbaniak]

@eb8680 this can be reviewed now (see a comment above, though).

[SamWitty]

@rfl-urbaniak , it looks like the CEVAE notebook has an error in it. Could you please rerun and confirm that it works end-to-end.

[rfl-urbaniak]

@SamWitty So the notebooks pass the automated tests on a GitHub runner. They pass locally on my laptop. They fail locally on my desktop. cevae.ipynb fails with the first cell, which succeeds as soon as I comment out import pytorch_lightning as pl. So I investigate the error and there seem to be compatibility issues.

torchaudio 0.13.1+cpu has requirement torch==1.13.1, but you have torch 2.1.0.
torchvision 0.14.1 has requirement torch==1.13.1, but you have torch 2.1.0.

The problem is, as soon as I downgrade torch to meet this requirement, pyro ppl compains about requiring torch > 2.0.
Are these problems similar to the ones you're running into?
How would you like me to proceed?

[SamWitty]

@rfl-urbaniak , does this conflict persist with a fresh start from a clean virtual environment?

If so, I'd recommend removing the CEVAE notebook and submitting this PR for review without it. We can add back more notebooks in subsequent PRs.

[SamWitty]

@rfl-urbaniak , is this ready for review? You added the tag status:awaiting review and then made future changes, so I just want to check in.

[rfl-urbaniak]

After changing the status, I re-run the notebooks locally outside of the CI=1 context to recover the original figures and pushed. Should be ready for review.

[SamWitty]

Great. @eb8680 , I was fairly involved in setting this up, so I think it makes sense for you to review and merge (if it passes your review). I'll remove myself from the reviewers on GH.

[eb8680]

Thanks for tackling this, seems like a reasonable starting point. If it ends up adding too much time to CI builds, we can try to parallelize over the notebooks at the level of the GitHub workflow, rather than within a single CI instance via pytest-xdist.

It would also be nice to typecheck and lint notebooks in CI, if possible. I will create a separate issue for that.