Help Browser code evaluation

[elgiano]

https://github.com/elgiano/rfcs/blob/helpbrowser/rfcs/0009-helpbrowser-code-evaluation.md

[dyfer]

Hey @elgiano

Thanks so much for the RFC! And thank you for the fixes in the PR!

Just a little technical note: I think it would be great if you could provide direct link to RFC text in the description (and possibly in the related PR).

I was thinking about looking into the help system, too.
In my opinion, in the long term it would be great to move entirely to WebSockets, so the help can be accessed from the system web browser (with code evaluation). But I haven't looked into it enough to really evaluate whether it's feasible.
For what it's worth, I've worked with WebSockets before a little bit.
Just to be clear: I'm not suggesting to move to WebSockets right away... but I wonder if

1. WS is the long-term direction we'd be going
2. and if so, whether this informs anything about the fixes/changes in the RFC and related PR

[elgiano]

Thanks @dyfer, I added links at both places!
I'm also interested in websockets (to make javascript guis, bridges to browser applications, collaborative editors...), and I guess it will be an important step, to be handled with care, so, let's see what people think!
In the meanwhile, there are some fixes for help browser's present state :)

[nhthn]

Hi @elgiano, thanks for making this, I think making an RFC is the wisest next step. Reading the room, I don't know if I'll have much success pushing my opinion on this, but I'll try. (Also, I know it wasn't my fault, but sorry you got caught in the crossfire in that other thread. I hope the air is clear now.)

My stance is: I believe that help browser code evaluation should be removed and replaced with a button that allows copying the contents of the code to the clipboard.

Motivation:

• Development burden. We will end up maintaining the IDE code editor, the IDE help browser code editor, and the sclang HelpBrowser code editor. The IDE codebase has mostly nonexistent automated test coverage, which means that the addition and indefinite maintenance of new IDE features automatically adds technical debt.
• Copy-paste is only a few clicks different from help browser code eval, and consistent with how most other programming language documentation works.
• No work is needed to unify customized keyboard shortcuts between the different editors.
• The UX will be unified between different viewing contexts:
  • hosted copies of the SC docs like doc.sccode.org
  • viewing local files in a regular browser, which may eventually be integrated with SC for WebEngine-free installations
  • HelpBrowser, and the IDE help browser

Rebuttals to counter-arguments:

• Users will complain: fair point. It's a balance between user complaints vs. complexity of the project and developer burden.
• Max/MSP and Pure Data have executable documentation: moot point because they are graphical programming languages, and copy-paste isn't practical.

Maybe a user survey will help us quantify who would miss this feature if it were removed. I'm envisioning two questions: "Do you execute code from the help browser?" "Would you miss it if it were gone and there was a 'Copy to Clipboard' button instead?"

[dyfer]

Who'd be the one to make a survey? That's certainly more burden for the OP (I believe it is a good idea though).
I personally would like the code evaluation to stay and I'm happy to help to make that happen.

[elgiano]

@snappizz
I see your points. It was indeed a lot of work to figure out what's going on with WebViews.
Anyway, code execution (and WebView) is there since a long time: we won't remove it quickly (if ever), I think it would be bad style. So it's worth fixing it while we decide and eventually discontinue it gently. I think my PR would put it already in a better state, but again, I would need another pair of eyes on it.

On to your points:

• Development burden: isn't it actually only two code-editors? IDE HelpBrowser and HelpBrowser class both use the same javascript code editor, which is in turn the same one we have on hosted documents: it is defined in javascript, within Help files. I would personally support keeping these two.
• Copy/paste: I would support this too, and add a copy/paste button to help files anyway.
• User poll: good idea, let's first make up our mind about a plan, and then phrase it gently not to scare anyone. I would also be scared to see something like "Would you like Help Browser code execution to stay or to go? :)", but of course that would be easier to understand with a good and well-discussed plan (or hypotesis).

It would make sense for me to first (or anyway) discontinue code execution in WebViews. The message would be: if you want your WebView to execute code, write a javascript function that returns the code you want to execute. Then hook the js function to a keypress, button or whatever you want, on sclang side, so that you can get that code back and have the interpreter evaluating it. Without WebSockets, there's no other way for a WebView to send a "signal" to sc.

About the HelpBrowser SC class, I think it could implement this exact behavior (in fact it is doing that now, and it's fixed by my PR). I believe HelpBrowser is used mainly outside of scide, I don't know how keyboard shortcuts are handled there, but everyone who uses HelpBrowser could define their own.

About WebEngine-free installations, what I understand is that to integrate a browser with sclang we need WebSockets. It's not uncommon to write bridge applications, I've seen (and wrote) a few in node/electron, for web pages to send OSC to sclang, or to communicate with the interpreter, and I believe we could start thinking about a native implementation. Then we should talk about implementing WebSockets with or without Qt... I can see that Qt has a WebSocket module, not part of WebEngine, which we could look into for the sake of consistency.

--- not so important stuff ---
Another alternative (no-go for me) would be to provide something like TextView has: a callback for selected text. But TextViews can't run javascript, so I guess they really need it, while WebViews could be left without it, since they are capable of more complex and custom implementations, and they are a "DIY" widget anyway.

Furthermore, if sc starts using a lot of js, there could be a js library providing some standard functions, like selectCodeBlock() to do parens matching, or others to deal with and eventual sc's native websocket implementation.
--- / not so important stuff ---

tl;dr: let's fix it anyway while we decide. Delegate webview code selection to javascript, and webview code execution to users' implementations. A copy button is anyway good to have.

[mossheim]

Hey @elgiano -- thank you for this RFC! i unfortunately haven't had time yet to look at this PR and proposal in detail, but i just wanted to say that i'm glad to see it and glad that the discussion is moving forward. this is a pain point for many people on this project, definitely for myself and @snappizz , and i think you're doing a good job of balancing concerns. i share in Nathan's opinion that at some point, we ought to find some way to reduce the maintenance cost of the help browser, but i also agree that in the short term we can make worthwhile steps toward improving this feature. looking forward to discussing it more!

[elgiano]

Hey, happy to be part of this!
Could you explain where the pain is? I think I understand, but I would like to connect more precisely to previous efforts and problems you encountered. Let's share the pain a bit :)
For me most of the frustration was understanding that keypresses in webview are eaten by web inputs. And I feel the way I'm dealing with that in the PR is kind of hacky... but I couldn't find any other solution...
Then there is one thing I did to CMakeFiles, to include QcWebView in ScIDE without pulling the whole sclang, adding a switch to disable widget registration, just to reuse QcWebView. I think a better smelling implementation would have a separate class encapsulating our WebView workarounds, and being then wrapped by both QcWebView and HelpBrowser.

But then, that's it (for me, of course). I can imagine that maintaining the javascript part (or even coming up with it in the first place) might have been another huge thing... but I think there is actually some future for it, to live a more independent life, if it will provide easy sc syntax and websocket utilities

[mossheim]

Could you explain where the pain is? I think I understand, but I would like to connect more precisely to previous efforts and problems you encountered. Let's share the pain a bit :)

in a nutshell, this area of code has had a lot of bugs in the last 2 years, and this project suffers from a lack of people who know how to solve them well.

historically, this is because:

1. the code was written mostly by people who are not around anymore (someone can correct me if i'm wrong)
2. the code was then ported from webkit to webengine by @scztt, who is not very active on this project
3. many of the bugs spanned Qt and Javascript together, and the only person on this project with knowledge of both enough to seriously address bugs was @snappizz at the point when they arose

now, IMO it is quite predictable that (2) caused a lot of these bugs we're seeing now. this is not to criticize Scott; on the contrary, i am very grateful, because it was desperately needed and nobody else was able to handle it. however, it was a lot of work for one person to do. while i was reviewing it Scott and i tried to do our best to test everything, but old C++ / JS code written on top of aging APIs is just naturally going to be a bit unstable when you start prodding it that way.

[elgiano]

Summing up a talk during last dev meeting:
Removing all web dependencies from sc would be beneficial, obviously to get a smaller size (and thus favoring bundles/standalones/embeddings), and also to reduce technical debt and burden.
A way to go would require identifying use cases and providing better alternatives to the current implementation of web technologies. Here are the main points discussed:

• academic/learning resources: we need a format where documentation and code can be freely mixed. Jupyter could be one, also because it is already widely used for the same purpose in the academic world.
• interoperability with web technologies: opens up extendability on the web, gives back what we would take away by discontinuing WebViews. Main candidate is a WebSockets server.
  AFAICT, the current HelpBrowser functionality could be recreated by both, but preferably by WebSocket because it wouldn't require any change of Help files format.

Since this process requires careful consideration and a smooth transition ( = a long time if ever :) ), I'm marking my PR as ready for review to fix the current implementation

[telephon]

One thing to consider is if in this transition, we could get back some of the beauty of the system before it became the one we have now (don't get me wrong, the current system does have great advantages, too!). What was very good at the time when we just had plain rtf documents for everything, that is for documentation and code, was that there was practically no hurdle to overcome in changing, appropriating, and correcting documentation/code. At that time the main hurdle was reintegration of the changes, the changes themselves were so simple that anyone who can use a text editor can make them. This unity is really something interesting, very much at the core of hypertext.

I don't know if my comment is productive, but it might be worth considering, since hill down is always the stuff that is well established practice. If there was a wysiwyg-markdown-wiki with a bit of back end to organise the specifics of help files as well, that would be the best I can imagine.

[muellmusik]

• interoperability with web technologies: opens up extendability on the web, gives back what we would take away by discontinuing WebViews. Main candidate is a WebSockets server.
  AFAICT, the current HelpBrowser functionality could be recreated by both, but preferably by WebSocket because it wouldn't require any change of Help files format.

One small observation, which I offer without taking a strong view on the long term direction of this: Websockets would be a wonderful addition in any case, and could indeed replace a lot of current functionality (doc code execution; bridging to other contexts). One thing that would be lost if WebViews were removed however would be the possibility of integrating browser content directly within sclang GUIs. I'm not sure how big a deal that is, but I do think it's worth being clear about every implication of removal, as that does go beyond just help issues. It might be worth exploring the range of current uses in any survey.