Static (preview) rendering (EBI-Metagenomics#19)

* adds Quarto static-rendering of notebooks (e.g. for github pages)

* revert some temporary content changes

* run GHA Preview action on feature branch

* adds test retries for more resilience to random failures

* fix docker run command on GHA Preview job

.github/workflows/preview.yaml

@@ -0,0 +1,51 @@
+name: Preview
+
+on:
+  workflow_dispatch:
+
+  push:
+    branches:
+      - main
+      - 'feature/static-preview'
+
+permissions:
+  contents: read
+  pages: write
+  id-token: write
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    environment:
+      name: github-pages
+      url: ${{ steps.deployment.outputs.page_url }}
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v3
+
+      - name: Setup Pages
+        uses: actions/configure-pages@v2
+
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@v2
+
+      - name: Build Dockerfile
+        uses: docker/build-push-action@v3
+        with:
+          context: .
+          file: ./docker/docs.Dockerfile
+          load: true
+          tags: notebooks-static
+
+      - name: Quarto render
+        run: |
+          docker run -v $PWD:/opt/repo -w /opt/repo notebooks-static render --execute
+
+      - name: Upload artifact
+        uses: actions/upload-pages-artifact@v1
+        with:
+          path: '_site'
+
+      - name: Deploy to GitHub Pages
+        id: deployment
+        uses: actions/deploy-pages@v1

.gitignore

@@ -5,4 +5,8 @@ shinyproxy.log*
 .DS_Store
 node_modules
 tests/*.png
-.idea
+.idea
+_site
+_freeze
+/.quarto/
+*.pyc

README.md

@@ -10,32 +10,52 @@ This repository contains example notebooks, written in Python and R, for using t
 
 There are various ways to use these Notebooks, including opening them locally in an existing Jupyter environment or via cloud services.
 
-The Notebooks themselves are in the `notebooks-src` dir.
+The Notebooks themselves are in the `src/notebooks` dir.
 
 
+## Development prerequisites
+You need [Docker](https://www.docker.com/products/docker-desktop/) installed.
+(Podman will also work for basics like editing the notebooks.)
+
+You need [Task](https://taskfile.dev/) installed for handy shortcut commands. If you don't want to install that, check `Taskfile.yml` for the long commands.
+
 ## Opening the notebooks (for use or development): use Docker
 Between the [base image](https://jupyter-docker-stacks.readthedocs.io/en/latest/using/selecting.html#jupyter-datascience-notebook) 
 and the extra requirements (`dependecies/*`), the Docker contains all the libraries we need.
 
+### To add a new notebook
 ```bash
-docker build -f docker/Dockerfile -t mgnify-nb-dev .
-docker run -it -v $PWD/notebooks-src/notebooks:/home/jovyan/mgnify-examples -p 8888:8888 mgnify-nb-dev
+TITLE='My New Notebook' AUTHOR='My Name' task add-py-notebook
 ```
-This binds the `notebooks-src/notebook` directory of this repo to `/home/jovyan/mgnify-examples` inside the Docker (overwriting the notebooks built into the image),
-so you can edit the notebooks and have changes relected in the repository.
-("jovyan" is always the user for these Jupyter Docker images.)
+This copies and fills in a template notebook stub file with a standard header, into `src/notebooks/Python Examples`.
+
+(TODO: `R` version).
+
+### To open the notebooks server in edit mode
+```bash
+task edit-notebooks
+```
+
+This runs the Docker image `quay.io/microbiome-informatics/emg-notebooks.dev:latest` which will either by pulled from Quay, 
+or run from your local image if you have built it (see below).
 
-Your (host) browser will not automatically open Jupyter Lab. 
-Copy one of the URLs from the console into your browser to open it.
+The folder `src/notebooks` will be mounted into the Docker container, so that changes you make are reflected in the repository.
 
-### Guidance for authoring notebooks
+Open a web browser to one of the URLs printed in your console to see Jupyter Lab.
+It should be localhost port 8888, with a random token.
+
+When you're finished editing, use normal `git add` and `git commit` to contribute your changes.
+
+For info, ("jovyan" is always the user for these Jupyter Docker images.)
+
+#### Guidance for authoring notebooks
 - Notebooks should be complete examples, that can be run with zero code changes needed
 - Notebooks should showcase good practice and use of popular libraries
 - Use the Jupyter menu option to "Clear All Outputs" before checking changes into git
 - Datasets should run reasonably quickly (i.e. no step should take more than a few minutes)
   - If large (slow) data fetches are needed, these should be cached in the Docker image.
 
-#### Caching data in the image
+##### Caching data in the image
 MGnifyR uses a cache of pulled MGnify data.
 This is populated during the Docker build, into `/home/jovyan/.mgnify_cache`, by the script in `dependencies/populate-mgnify-cache.R`.
 Add commands to this to include other datasets in the cache.
@@ -51,6 +71,37 @@ exit
 git add depdencies/mgnify-cache.tgz
 ```
 
+### Changing dependencies and Docker build
+The add dependencies, edit the `dependencies/environment.yml` file.
+
+You can temporarily try things by opening a Terminal inside Jupyter Lab and `mambda install`ing the package(s).
+But make sure you reflect everything in the conda environment file.
+
+Then check the environment builds by (re)building the Docker:
+```bash
+docker build -f docker/Dockerfile -t quay.io/microbiome-informatics/emg-notebooks.dev:latest .
+```
+
+## Generating static previews
+There is a setup to use [Quarto](https://quarto.org/) to render the notebooks – including inputs and outputs – as a static website (amongst other mediums).
+
+This is useful as a kind of documentation resource.
+
+There is a Dockerfile to add Quarto on top of the regular Docker stack: `docker/docs.Dockerfile`.
+
+To preview the statically rendered notebooks, use:
+```bash
+task render-static
+```
+This builds a docker image tagged as `notebooks-static`, runs Quarto inside it, executes all cells of the notebooks, 
+and renders the completed notebooks to the `_site` folder (which is mounted from your host machine into Docker).
+
+You can then open the generated HTML, or use
+```bash
+task preview-static
+```
+to render the notebook in watch-mode and serve them to [serve a preview of them](http://localhost:4444).
+
 
 ## Shiny-Proxy application
 The notebooks can also be built into a Docker container suitable for running as an Application on ShinyProxy.
@@ -79,11 +130,16 @@ This extenion is needed because Shiny Proxy does not pass the URL path beyond an
 
 The extension does two things:
 1. **Allows deeplinking to notebooks.** It watches for a URL querystring parameter, `?jlpath=`, and uses this to forward Jupyter Lab to an internal URI which activates that path. This works because Shiny Proxy **does** include the query params in the iframe. E.g. browsing to `localhost:8080/app/mgnify-notebook-lab?jlpath=notebooks/home.ipynb` will trigger Jupyter Lab to open the `home.ipynb` notebooks once the application initialises. This is a frontend extension in `shiny_proxy_jlab_query_params/src/index.ts`.
-2. **Sets ENV VARs based on query params.** On Jupyter Lab launch, it sends the querystring parameters to a Jupyter Lab "server-side" extension handler. The handler takes any querystring params beginning `?jlvar_` and sets corresponding ENV VARs. E.g., `?jlvar_MGYS=MGYS007` results in an ENV VAR of `MGYS=MGYS007` being available to any kernels launched after this. These ENV VARs can then, of course, be read in Notebooks (e.g. `os.getenv('MGYS')` in Python or `Sys.getenv('MGYS')` in R). There are helper utilities for both R and Python, that try to read such an ENV VAR otherwise ask for user input. These are in `notebooks-src/notebooks/{Python Examples | R Examples}/lib/variable_utils.{R|Python}`.
+2. **Sets ENV VARs based on query params.** On Jupyter Lab launch, it sends the querystring parameters to a Jupyter Lab "server-side" extension handler. The handler takes any querystring params beginning `?jlvar_` and sets corresponding ENV VARs. E.g., `?jlvar_MGYS=MGYS007` results in an ENV VAR of `MGYS=MGYS007` being available to any kernels launched after this. These ENV VARs can then, of course, be read in Notebooks (e.g. `os.getenv('MGYS')` in Python or `Sys.getenv('MGYS')` in R). There are helper utilities for both R and Python, that try to read such an ENV VAR otherwise ask for user input. These are in `src/notebooks/{Python Examples | R Examples}/lib/variable_utils.{R|Python}`.
 
 Together, this means a URL like: `localhost:8080/app/mgnify-notebook-lab?jlpath=notebooks/home.ipynb&jlvar_MGYS=MGYS00005116` will trigger Shiny Proxy to start the container, then open the `home` notebook, and have an MGYS environment variable ready to use in code.
 
 
+## Jupyter Lab Extension, for MGnify-specific help
+There is also an extension to render a MGnify-specific help pane and menu inside Jupyter Lab.
+This is in the `mgnify_jupyter_lab_ui` folder.
+
+
 ## Testing
 A small integration test suite is written using Jest-Puppetteer.
 You need to have built or pulled the docker/Dockerfile (tagegd as `quay.io/microbiome-informatics/emg-notebooks.dev`), and have Shiny Proxy downloaded first.
@@ -103,6 +159,13 @@ Just push to the repository (all branches are built and tagged). If you push to
 
 This built image can be (and is) deployed to multiple servers, e.g. a ShinyProxy instance or the Galaxy project.
 
+### Releases
+Tag releases with semver tags, like `v1.0.0`.
+
+- Increment by `0.0.1` for bugfixes, new notebooks, new packages etc.
+- Increment by `0.1` for changes in the base docker image, or refactors of the docker/environments etc.
+- Increment by `1` for changes in how the entire stack is run.
+
 
 ## Contributors ✨
 

Taskfile.yml

@@ -0,0 +1,55 @@
+version: '3'
+
+tasks:
+  add-py-notebook:
+    summary: |
+      Creates a new Python-language notebook stub from a template.
+
+      Set the TITLE and AUTHOR variables, which are used to name
+      the notebook file and set them in the notebook front matter.
+
+      Usage: TITLE='Notebook Name' AUTHOR='Your Name' task add-py-notebook
+    cmds:
+      - cp src/templates/python.ipynb "src/notebooks/Python Examples/{{.TITLE}}.ipynb"
+      - sed -i '' 's/::NBTITLE::/{{.TITLE}}/g' "src/notebooks/Python Examples/{{.TITLE}}.ipynb"
+      - sed -i '' 's/::NBAUTHOR::/{{.AUTHOR}}/g' "src/notebooks/Python Examples/{{.TITLE}}.ipynb"
+    preconditions:
+      - sh: test -n "{{.TITLE}}"
+        msg: "TITLE parameter missing. Set: TITLE='Notebook Name' AUTHOR='Your Name' task add-py-notebook"
+      - sh: test -n "{{.AUTHOR}}"
+        msg: "AUTHOR parameter missing. Set: TITLE='Notebook Name' AUTHOR='Your Name' task add-py-notebook"
+
+  edit-notebooks:
+    summary: |
+      Opens Jupyter Lab (via Docker) in edit mode – with the notebooks source bound to this repository
+
+      The files in src/notebooks are mounted as editable, and served on port 8888.
+    cmds:
+      - docker run -it -v $PWD/src/notebooks:/home/jovyan/mgnify-examples -p 8888:8888 quay.io/microbiome-informatics/emg-notebooks.dev:latest
+
+  build-static-docker:
+    summary: |
+      Builds a docker image with Quarto included, for statically rendering the notebook outputs.
+
+      The built image is tagged as `notebooks-static`.
+    cmds:
+      - docker build -f docker/docs.Dockerfile -t notebooks-static .
+    status:
+      - docker image inspect notebooks-static
+
+  render-static:
+    summary: |
+      Runs and renders the notebooks as a static website
+
+      The site is built to ./_site
+    cmds:
+      - docker run -it -v $PWD:/opt/repo -w /opt/repo notebooks-static render --execute
+    deps: [build-static-docker]
+
+  preview-static:
+    summary: |
+      Runs, renders, and serveces the notebooks as a static website, watching for changes
+    cmds:
+      - echo 'When the rendering is finished, the static preview of notebooks will be at http://127.0.0.1:4444 ...'
+      - docker run -it -v $PWD:/opt/repo -w /opt/repo -p 4444:4444 notebooks-static preview --no-browser --port 4444 --host 0.0.0.0
+    deps: [build-static-docker]

_environment

@@ -0,0 +1,2 @@
+API_ENDPOINT="super-studies"
+MGYS="MGYS00005292"

_quarto.yml

@@ -0,0 +1,53 @@
+project:
+    type: website
+    render:
+        - src/notebooks_list.qmd
+        - src/notebooks/Python Examples/*.ipynb
+        # - src/notebooks/R Examples/*.ipynb
+
+execute:
+    freeze: auto
+
+website:
+    title: MGnify Notebooks
+    favicon: src/static-resources/mgnify_logo.png
+    page-navigation: true
+
+    body-header: |
+        :::{.callout-note}
+        ## This is a static preview
+        You can run and edit these examples interactively on [Galaxy](https://usegalaxy.eu/root?tool_id=interactive_tool_mgnify_notebook)
+        :::
+
+    repo-url: https://github.com/ebi-metagenomics/notebooks
+    repo-subdir: src
+    repo-actions: [edit, issue]
+
+    navbar:
+        right:
+            - href: https://usegalaxy.eu/root?tool_id=interactive_tool_mgnify_notebook
+              text: Run on Galaxy
+
+    sidebar:
+        style: docked
+        search: true
+
+        contents:
+            - href: src/notebooks_list.qmd
+
+        tools:
+            - icon: github
+              menu:
+                  - text: Notebooks source
+                    url: https://github.com/ebi-metagenomics/notebooks
+                  - text: MGnify repositories
+                    url: https://github.com/ebi-metagenomics
+            - icon: question-circle-fill
+              text: Help
+              href: mailto:[email protected]
+format:
+    html:
+        theme:
+            - cosmo
+            - src/static-resources/styles.scss
+        toc: true

docker/Dockerfile

@@ -27,5 +27,8 @@ RUN pip install /tmp/mgnify_jupyter_lab_ui
 # Clean yarn cache else chown'ing home is very slow on container start
 RUN jlpm cache clean 
 
+# Clean tmp
+RUN rm -rf /tmp/*
+
 COPY shiny-proxy/custom.js /home/jovyan/.jupyter/custom/custom.js
-COPY notebooks-src/notebooks /home/jovyan/mgnify-examples
+COPY src/notebooks /home/jovyan/mgnify-examples

docker/docs.Dockerfile

@@ -0,0 +1,7 @@
+FROM quay.io/microbiome-informatics/emg-notebooks.dev
+
+ARG QUARTO_VERSION="1.2.269"
+WORKDIR /tmp
+RUN wget https://github.com/quarto-dev/quarto-cli/releases/download/v${QUARTO_VERSION}/quarto-${QUARTO_VERSION}-linux-amd64.deb
+RUN dpkg -i quarto-${QUARTO_VERSION}-linux-amd64.deb
+ENTRYPOINT ["quarto"]

notebooks-src/notebooks/Python Examples/Download paginated API data to a CSV.ipynb → src/notebooks/Python Examples/Download paginated API data to a CSV.ipynb

@@ -1,5 +1,24 @@
 {
  "cells": [
+  {
+   "cell_type": "raw",
+   "id": "b989dfc0-ed07-4741-8bfd-18363bb1a930",
+   "metadata": {
+    "jupyter": {
+     "source_hidden": true
+    },
+    "tags": []
+   },
+   "source": [
+    "---\n",
+    "title: \"Download paginated API data to a CSV\"\n",
+    "author: \"Sandy R (MGnify team)\"\n",
+    "categories: [Python]\n",
+    "execute: \n",
+    "  enabled: true\n",
+    "---"
+   ]
+  },
   {
    "cell_type": "markdown",
    "id": "ce2d1dc9-1a56-4103-8353-5a5b3fdb24bc",
@@ -107,7 +126,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.8.12"
+   "version": "3.10.6"
   }
  },
  "nbformat": 4,