Expose a cljs prepl via socket

[mynomoto]

Currently shadow exposes a socket repl but some tooling require a socket prepl to work. The most useful one for shadow would be a cljs prepl so that it could share the shadow-cljs context. There is a description of how it works with figwheel on https://oli.me.uk/2019-03-22-clojure-socket-prepl-cookbook/

[thheller]

prepl suffers from the same problems that nrepl has.

Implementing this I feel like I repeat all the nREPL problems (nREPL, not shadow-cljs) all over again. nREPL and pREPL are built for Clojure and not ClojureScript. Sure, we can make it look like Clojure but then we are omitting certain important aspects of CLJS all over again.

Can anyone point me to an implementation of a CLIENT that actually uses prepl?

I will probably write a more detailed post about this at some point to explain my frustrations with all of this in more detail. Sorry pREPL support is delayed by this.

[daveyarwood]

Can anyone point me to an implementation of a CLIENT that actually uses prepl?

https://github.com/Olical/conjure

Conjure is a plugin for Neovim that provides prepl tooling. You configure it declaratively by describing a number of prepl servers, and Neovim connects to them and lets you evaluate code over the relevant connection. Return values of various types (ret, out, err, tap, etc.) are collected in a log buffer.

[thheller]

2.8.59 has initial prepl support but it probably still needs some tuning.

You configure it via :prepl at the top level in shadow-cljs.edn.

{:source-paths [...]
 :dependencies [...]
 :prepl {:app 12345}
 :builds {...}}

:prepl takes a map of {build-id port-number-or-map} so it either takes a simple number to use as port or a map with more options. Although currently thats only {:port 12345 :host "localhost"}. :host defaults to "localhost".

Some open questions about the operation of the prepl servers.

• When to start the servers?

Currently they are started with the shadow-cljs server instance. So they might be running although the build isn't actually running. Not sure this makes much of a difference overall.

• What to do about missing or mulitple runtimes?

As I tried outlining here there are scenarios (common, not unlikely) where there is either no connected JS runtime at all or many at once. So it might be a browser build where the user hasn't loaded the page, or the page could be open in multiple browsers (eg, chrome + firefox).

Currently if there is no runtime connected at all a new prepl connection is immediately terminated after sending one :err message. It could remain open but you can't actually eval anything. This is the most common problem people run into with the CLJS REPL in shadow-cljs I still haven't found a proper way to explain this.

I don't know what to do about multiple runtimes. There should be some way to "select" a given runtime but given prepls design that isn't easy.

• What to do when the primary runtime disappears?

When a prepl connection is made one runtime is selected. That runtime may disappear and re-appear or not (eg. browser reload). Currently I decided to disconnect the prepl connection when the runtime disappears (after sending an :err message). The connection could remain open but all REPL state will be lost so it seemed like the best option to disconnect so the client can reset accordingly?

[Olical]

This is fantastic news! I know it'll be a key use case for Conjure, having more and more easy was to hook a tool up to an environment via a prepl will help get newer users up and running really quickly.

I'm not sure if it helps, but my approach with prepls has been to have one per environment, so if I was starting multiple ClojureScript environments I'd start a prepl on a port for each one, that may not be possible with your constraints but I thought I'd mention it.

I'll try it out with Conjure soon 😄 thank you so much for this!

[thheller]

@Olical that is already supported. You can run one prepl per build each on its own port. That however does not address the "runtime" issue I described. Pretty sure the other CLJS REPLs don't address that problem at all and just pretend it doesn't exist. Figwheel has some support but I'm not sure how it works with prepl.

One simple example:

Using shadow-cljs watch app with :app config being a :browser build. Using :prepl {:app 12345}. Open the resulting :app code in the Browser. Connect the prepl and everything is fine. Now open the app in another Browser (just a tab or actual doesn't matter). prepl still works fine but evals only happen in the first. How do you address the second one? What if you open the prepl socket when 2 runtimes are already connected. Which one does it use?

The problem isn't isolated to the Browser. Same can happen for react-native builds with the App open in iOS and Android side-by-side.

shadow-cljs on the CLJ side technically exposes all the information you would ever need. So you could ask it which runtimes are available and you could start a "session" for any of these and you'd be informed when a runtime leaves or joins. Not all REPLs will have this issue and a node-repl for example will only every have one managed runtime, others however may have zero or more.

I have an internal REPL protocol that I'll might expose publicly and document a bit which addresses all these issues and is currently in use by the UI. I tried making a quick video showing how the UI currently displays the runtime information and how you can pick one particular runtime directly from the UI. Runtimes are added the moment they connect and removed if they are closed. The UI isn't quite finished and I don't actually want to run a REPL in the UI itself. Just using it as a testbed for how I'd like things to work.

The problem is that currently no editor even attempts to show this information, probably because that information isn't available from any other REPL impl.

[Olical]

Ah I understand what you mean now, thanks for the explanation! I would argue you go for the first environment and maybe disconnect the prepl if there's nothing there, that way the client will know it's reconnecting to a new environment.

Conjure, for example, requires and injects things it needs on connection. If the env is replaced but Conjure doesn't know it might die when you try to execute certain things, which isn't the end of the world. If it gets kicked off the prepl though that'll help a lot.

[theophilusx]

@thheller Nice explanation. I think this is the sort of content which would be great in a FAQ section of the documentation. (or a link to this issue to draw attention to the content). It is one reason I really like shadow-cljs - you put in the time to discuss and explain, which is very appreciated.

[Olical]

Noting here for future reference, here's my guide to prepls for tool authors: https://oli.me.uk/clojure-prepl-for-tool-authors/

There's still a few things in the shadow prepl that aren't quite right but I hope I (or anyone else!) can help Thomas get them inline soon. There's so many people coming to Conjure because they want to use it with shadow. They'll all be happy to jump on it as soon as it's working 🎉

[martinklepsch]

I'm currently trying to improve my understanding of Shadow's prepl support and was trying to get a prepl running for a :node-script build, by doing the following within the shadow-cljs repo:

1. Enable prepl for the :self-contained-script build

diff --git a/shadow-cljs.edn b/shadow-cljs.edn
index 0e10555f..711e0c3f 100644
--- a/shadow-cljs.edn
+++ b/shadow-cljs.edn
@@ -14,7 +14,8 @@
  {8600 "out/demo-browser/public"
   8606 "out/demo-test-browser"}

- :prepl {:browser 12345}
+ :prepl {:browser 12345
+         :self-contained-script 44444}

2. Start a regular REPL via Leiningen and kick off the build

lein with-profiles +cljs repl

(do (require 'repl)
    (repl/go)
    (shadow.cljs.devtools.api/watch :self-contained-script))

3. Connect via nectat

$ nc localhost 44444
{:tag :err, :val "No available JS runtimes!"}

On these connection attempts shadow-cljs logs the following:

:watching[:self-contained-script] Build completed. (260 files, 259 compiled, 0 warnings, 21,66s)
...
[2019-12-12 11:14:12.367 - FINE] :shadow.cljs.devtools.server.repl-system.clojure/process-clj-msg - #:shadow.cljs.model{:op :shadow.cljs.model/tool-disconnect, :tool-id "prepl:0"}

It seems that this is what @thheller refers to here: #508 (comment)

Currently I decided to disconnect the prepl connection when the runtime disappears (after sending an :err message).

... but maybe more like a runtime was never there?

Some questions resulting from this for @thheller:

• Did I miss any steps or is this initial prepl support mostly intended for browser builds for now?
• For the :self-contained-script build I needed to npm I request which, should this be captured in a package.json somewhere? I tried looking for one but couldn't find any.
• Similarly for the :browser build I needed to npm i process.

Some other observations

I used the :browser build for these experiments. Mostly based off of this article and the description of the prepl protocol therein: https://oli.me.uk/clojure-prepl-for-tool-authors/

(println "Hello World!")
EXPECTED {:tag :out, :val "Hello World!\n"}
ACTUAL {:tag :out, :val "Hello World!"}

(the :ret message is returned properly)

(throw (js/Error. "test"))
EXPECTED {:tag :ret, :val "{:via [...ELIDED FOR READABILITY ...] :cause \"test\", :phase :execution}", :ns "user", :form "(throw (Error. \"test\"))", :exception true}
ACTUAL [nothing]

(tap> :foo)
EXPECTED {:tag :ret, :val "true", :ns "user", :ms 2, :form "(tap> :foo)"}
EXPECTED {:tag :tap, :val ":foo"}
ACTUAL {:tag :ret, :ns "cljs.user", :form "(tap> :foo)", :ms 0, :val "true"}
ACTUAL [no :tag :tap message]

[thheller]

@martinklepsch the "No available JS runtimes!" errors means exactly that. I don't know how to make that error any clearer. I tried here:

• https://shadow-cljs.github.io/docs/UsersGuide.html#_repl_2
• https://shadow-cljs.github.io/docs/UsersGuide.html#repl-troubleshooting

Suggestions on how to make it clearer would be very very very welcome. I keep having to explain this and I don't understand why its so confusing for others.

In short if you don't run node yourself (or open the Browser) then there is no available runtime to connect to ...

The dummy builds like :browser and :self-contained-script kinda don't declare their dependencies correctly since I have the root-level package.json in .gitignore. I don't want that to be in git because I constantly install packages for testing purposes only (at one point I had 70 dependencies in package.json). Haven't cleaned that up yet since those builds aren't really "test" builds anyways. For the :browser build you need npm i node-libs-browser (which will bring process and a few others).

The other prepl issues are already listed here: #610

[martinklepsch]

Suggestions on how to make it clearer would be very very very welcome. I keep having to explain this and I don't understand why its so confusing for others.

Thanks and sorry for asking a question that I seemingly could have answered myself. One suggestion I may have is to add links to the docs to the error messages. At least that way people are more pushed to read those properly first.

I guess I didn't consider that I have to run the node script myself because:

• I've previously used (shadow/node-repl) to interact with this :node-script build (via nREPL though)
• The script is running only a few seconds so I didn't consider that Shadow would inject something that keeps it running.

Is there a way to start the bare bones node-repl with a prepl socket? As far as I understood this then wouldn't be connected to my build but sources would be available which would be sufficient for my usecase.

I've also tried to run node out/demo-script-bundle/script.js but am getting an $jscomp is not defined error but maybe this is easier to hash out on Slack.

[Olical]

This is an aside to the current conversation, but I've started working on a tool to check compliance https://github.com/Olical/prepl-compliance-test

Contributions of tests will be much appreciated! I've documented it a little, hopefully it's easy enough to get started and running. I only had an hour over lunch while eating a sandwich 😬

[thheller]

@martinklepsch you can get a prepl for node-prepl by setting :prepl {:node-repl 12345} and then just running shadow-cljs node-repl separately.

[thheller]

prepl sort of works but I'm not really interested in maintaining it any further.

I'll document shadow.remote a bit more soon so anyone that wants to write a less capable prepl layer can do so as a separate lib. I will likely remove remove all prepl related bits from shadow-cljs completely and maybe provide the current code as an example impl.

[Olical]

Totally understandable! I think any capturing of the work you put in will be valuable for others in the future ☺️ I've had success with the nREPL server in shadow, for what it's worth. Thank you for all of your effort!

…

On Thu, 4 Jun 2020, 22:31 Thomas Heller, ***@***.***> wrote: prepl sort of works but I'm not really interested in maintaining it any further. I'll document shadow.remote a bit more soon so anyone that wants to write a less capable prepl layer can do so as a separate lib. I will likely remove remove all prepl related bits from shadow-cljs completely and maybe provide the current code as an example impl. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#508 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACM6XMYAMANNQYRYYOUWK3RVAHCXANCNFSM4HWUUIFA> .