Run notebooks once a week in a separate workflow

[ismael-mendoza]

This page was useful for how to setup the cron job: https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#schedule.

I set the cron job to run every Monday at 7a EST but I'm open to suggestions on this

[ismael-mendoza]

@thuiop - I've been wandering about this for a while. Should we bother checking the output of the cells of the notebook and checking if they are the same with sanitize? I think I would be satisfied with just checking if any of the cells failed to run.

[codecov]

Codecov Report

Merging #263 (6e0f745) into main (ce447d9) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main     #263   +/-   ##
=======================================
  Coverage   88.26%   88.26%           
=======================================
  Files          13       13           
  Lines        1526     1526           
=======================================
  Hits         1347     1347           
  Misses        179      179           

Flag	Coverage Δ
unittests	88.26% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ce447d9...6e0f745. Read the comment docs.

[ismael-mendoza]

I would suggest this flag

-nbval-lax, which runs notebooks and checks for errors, but only compares the outputs of cells with a #NBVAL_CHECK_OUTPUT marker comment.

if there is a output that is really important is the same then we can add that #NBVAL_CHECK_OUTPUT to that cell.

[ismael-mendoza]

I realized that if we want to check outputs are exactly the same that should be a test anyways so I removed the sanitize.cfg and switched to pytest --nbval-lax notebooks.

[thuiop]

@thuiop - I've been wandering about this for a while. Should we bother checking the output of the cells of the notebook and checking if they are the same with sanitize? I think I would be satisfied with just checking if any of the cells failed to run.

I'm not sure. I agree that it may be sufficient ; checking the output value ensures that it is "exactly the same" as it should be, but I am not entirely sure this is useful.

[ismael-mendoza]

Thanks @thuiop I think if we really want to check a function outputs exactly the same output this is better off as a test. The only thing about notebooks is that there are images but those we need to look at them by eye anyways

[thuiop]

It looks good to me but actually I wonder whether this is not a bit too far. I think we should check that the notebooks are actually runnable before merging to main ; maybe there is a way so that it does not do it automatically but there is a button we can click on before merging.

[ismael-mendoza]

In my terminal I ran it w/ both nbval and nbval-lax and both of them pass so I think we should be ok ?

Could you remind me why are the last 3 cells of 00-intro.ipynb skipped ? (and how you set them up so they would be skipped?)

[ismael-mendoza]

It looks good to me but actually I wonder whether this is not a bit too far.

I understand the concern, if you think it is useful to check output is exactly the same (maybe you have some notebook/cell in mind?) we can revert this last change.

In my experience a notebook not having exatly the same output is usually not a sign of an error (hopefully the tests would catch that!) like when we added the progress bar in your other PR.

Incidentally, how would you fix this last error? Where it is ok that the notebook cell changed with the progress bar? Would you need to sanitize it away?

[ismael-mendoza]

I can also see having an alert once a week notifying us that a notebook cell changed (even if small) might be good to review so I'm open to keeping it

[ismael-mendoza]

After more thought and learning about nbval a bit more, I decided to keep the sanitize as is

[ismael-mendoza]

nbval passing in my terminal so I'm merging this now