Experience Report: Steps to “Emacs Hyper Notebooks”

TL;DR — enough to get it working!

This is a way to set up connections between Emacs and long running processes, with an Org Mode interface. To use it, you first need to patch ob-core.el in the Org Mode distribution.

cd ~/org-mode/
git apply ~/ob-servant/ob-servant.diff

You’ll then need to reload the file you just patched:

(load "~/org-mode/lisp/ob-core.el")

Next, you need to load the code defining the package.

(load "~/ob-servant/ob-servant.el")

The same for our example implementation:

(load "~/ob-servant/ob-servant-example.el")

With all of this in place, blocks such as the following can be evaluated:

display2d:false

expand((x+1)^9)

Development plan — 8 December 2020

Reread things in the Elisp manual about processes, and make the system robust to all of these

• Robustify the connections to the underlying processes (e.g., deal with packets being sent separately)
  • Write a shell script for testing the edge cases of Accepting Output.
  • Make sure we fully understand e.g. the standard error stuff
  • Pipe processes

Next demos

• Hook Maxima and Julia together, e.g., with raindrops or whatever other demo
  • Note that we might want multiple versions of Maxima running with different settings also (e.g., Fermin did this)
• Make an Emacs demo? - What are the different ways of doing this?
• Write to Eric Schulte and Bastien figuring out how to get the changes into Org Mode

Future Scenarios

• Consider cases that run autonomously of human and need to e.g., re-update output or act on the basis of some other long running process
• Think about relationships with Arxana
• Think about relationships with MUD architecture (e.g., aborted work on MMTN)
• Think about relationships with agents in Clojure

Scenario

Ray and Cameron have been collaborating for years on theoretical biology research. In a typical project, they might use Maxima to make symbolic calculations and Julia for simulations.

Their work combines biology, physics, and computer science.

• E.g., Latest work-in-progress is on branching processes for cancer modelling.

The problem

• Moving code and data between different programs by hand.
• Separate workflows for writing notes and preparing publications.
• This is time consuming and error prone!

Possible solutions

Cameron is experienced with Jupyter and…

• we found Script of Scripts (https://vatlab.github.io/sos-docs/) but it only solves some of the problems.

Joe and Ray were happy explore Emacs-based solutions. (And Cameron got enthusiastic about Doom Emacs!)

Emacs+Org Mode…

TOOLS	EMACS/ORG	———————————————————————————————————————
Maxima	maxima-mode	Solving DE symbolically
Julia	julia-mode	Run the numerical solver
—	org-poly	Run multiple modes inside Org mode
Beamer	org-tree-slide	Make slides for a presentation
Wikis	Org Roam	Prepare the paper in a nonlinear wiki
—	Logseq	… with coauthors who don’t use Emacs
Pairing	lockstep.el	Collaborate in real time
Etherpad	crdt.el	… with multiple people typing at once
Zotero	ORB	Reference related work
TeX	Org Mode!	Typeset the results
Jekyll	Firn	Publish work in progress on a blog
KaTeX	fragtog	… including LaTeX math symbols

Let’s try a calculation using Maxima mode

Maxima doesn’t have a long-running process by default! Here’s a workaround.

⌗+NAME: solve-for-u
⌗+BEGIN_SRC maxima :results raw
tex(usol : part(solve ([u/(D*u + s) = %e^(-s*v) * u0/(D*u0 + s)], [u]), 1 , 2));
⌗+END_SRC

Problem 1: passing variables and outputs! — maybe not so bad?

Something like this should be enough to stitch Maxima and Julia together.

⌗+name: block-1
⌗+BEGIN_SRC emacs-lisp :cache yes :results raw
(current-time-string)
⌗+END_SRC

⌗+RESULTS[8dfa404531eb9e20742feb333c9031cd519de5ce]: block-1
Fri Nov 27 19:17:23 2020

⌗+BEGIN_SRC emacs-lisp :results raw :var input=block-1 :noweb yes
(format "Then %S and now: %S" <<block-1>> input)
⌗+END_SRC

⌗+RESULTS:
Then "Fri Nov 27 19:18:26 2020" and now: "Fri Nov 27 19:17:23 2020
"

— Thanks to the Kitchin Group for their docs!

Problem 2: Making the process long running

Solution: access maxima via a new “ob-servant” facility

display2d:false

expand((x+1)^9)

(Bonus feature: notice that we made it do tex output by default.)

Minor change to ob-core.el

We added a :servant parameter to override org-babel-execute:<lang> inside org-babel-execute-src-block. (We still want to keep the language around to control fontification etc.)

715c715,719
< 		 (cmd (intern (concat "org-babel-execute:" lang)))
---
> 		 ;; OVERRIDE WITH :servant TO ACCESS THE PROCESS WE CALL
> 		 ;; e.g. (:servant . "calculator")
> 		 (cmd (if (assoc :servant params)
> 			#'org-babel-servant
> 			(intern (concat "org-babel-execute:" lang))))

Quick overview of ob-servant.el

(defvar org-babel-servant-info nil)     ;; A hash table storing :preproc, :postproc etc., per service
(defun org-babel-servant-setup () ... )
(defun org-babel-servant-callback (prc str) ... )
(defun org-babel-servant-error-callback (prc str) ... )
(defun org-babel-servant (body params)
      ;; ① Extract the process from the param list.
      ;; ② Preprocess the body if possible.
      ;; ③ Set the timeout.
      ;; ④ Should make sure we really have a process before proceeding further!
      ;; ⑤ Clear the last output.  Later on, we might want to allow the
      ;;    option of archiving the old output for safe keeping. (Like McCarthy Elephant 2000?)
      ;; ⑥ Record the time
      ;; ⑦ Send out a request to the program we’re calling
      ;; ⑧ Wait for the replies... 
      ;; ⑨ Coda for post-processing (when required)
)

Example set up

(get-buffer-create "maxima-error")
(setq maxima-proc
      (make-process
       :name "maxima-proc"
       :command '("maxima" "--very-quiet") 
       :stderr "maxima-error"
       :filter #'org-babel-servant-callback))

(puthash '("maxima-proc" :preproc) (lambda (x params) (concat "tex(" x ");\n"))  org-babel-servant-info)
(puthash '("maxima-proc" :postproc) (lambda (x params err) (substring x nil -6)) org-babel-servant-info)
(puthash '("maxima-proc" :timeout) 2.0                                           org-babel-servant-info)

BTW, the substring is b/c Maxima prints out false as a second return here:

tex(expand((x+1)^9));
$$x^9+9\,x^8+36\,x^7+84\,x^6+126\,x^5+126\,x^4+84\,x^3+36\,x^2+9\,x+1$$
false

Reflections: pleasure and pain!

1. Getting emacs-juypter working via TRAMP not resolved (#191)! :-(
2. Working on ob-servant was pretty fun! :-)
3. We got more experience with co-editing code :-)
4. The stack is work in progress (e.g. crdt.el bugfixes thanks to Qiantan). :-)
5. A few small compatibility issues came up with Firn and Logseq (these were addressed by the maintainers quickly!) :-)
6. We tried getting Emacs running in the browser to make these tools widely accessible — but command keys didn’t pass through properly on the most popular browsers. :-(
7. We got benefit from presenting early prototypes at Emacs NYC :-)
8. Dialogues continue around EmacsConf 2020 w/ Fermin and others :-)

Future work 1

• How to integrate this workflow with Arxana (EmacsConf 2013, FARM 2017)?
  • One idea: Arxana deals with transclusions, and could potentially help with the combined notes+writeup workflow.
  • Also relates to the general idea of “network programming”
• How do we think about “remote control” for long-running processes?

Conclusions: we have taken steps to address:

Technical experiments are about accessing any longrunning process with a simple interface. We’re not the only people to look into “notebooks” but we think that Emacs has some advantages.

Reproducible research

• Something is “reproducible” if it is teachable to someone new!
• Org Mode (and literate programming in general) is useful for this.

Interdisciplinary collaboration

• Collaboration across different skill sets is challenging.
• Our collaboration was already interdisciplinary…

• … but what about collaborations between a scenario planner, simulation scientist, and local farmers, etc.?

Future work 2

Science should be:

• Widely teachable
• Sharable
• Semi-automated
• Transdisciplinary
• Real-time, like EmacsConf!

EMAIL        [email protected]
             [email protected]
             [email protected]

IRC          jcorneli

CODE (WIP)   github.com/exp2exp/ob-servant