Investigate using WebAssembly as a compiler target

[Ajedi32]

WebAssembly is a sort of bytecode intended to be used as a compiler target for languages that run in web browsers. It's an open standard being developed by Google, Microsoft, Mozilla, and others. (See Google, Microsoft, Mozilla And Others Team Up To Launch WebAssembly, A New Binary Format For The Web)

Okay, maybe it's a bit early to be thinking about this given that WebAssembly hasn't even been standardized yet, but it definitely seems like something to keep an eye on. See https://github.com/WebAssembly/design for details.

[adambeynon]

This would be a fun experiment. I did work on a ruby -> swift compiler before, using the opal lexer and parser. It didn't get very far, but it was surprisingly easy to reuse most of the compiler and just having optional code generation parts.

[ylluminate]

That's really interesting to hear @adambeynon. I'm kind of curious, even having not gotten very far, would you mind throwing what you did onto github? I've personally contemplated using opal as the base for a couple of other transpiler ideas, including swift, and I think just seeing your commits and the thought process would be useful.

[Ajedi32]

Yeah, it seems WebAssembly is a pretty low-level language, so maybe there wouldn't be much benefit to compiling Ruby down to it, but I'm not really sure. Definitely worth looking into though.

[bbatsov]

Seems to me this will be a worth-while endeavour down the road, when WebAssembly is better established. Right now we should probably be focused on compiling to JS only.

[zetachang]

WebAssembly has reached the Browser Preview milestone

Some works has been done by @xxuejie, webruby compiles mruby vm to asm.js. And the good news is the toolchain which compile C/C++ to asm.js (emscripten) also got WebAssembly as compiling target support now.

[Ajedi32]

FYI, WASM is now considered stable.

Still no garbage collection though. I imagine that'd be a rather important feature for Opal, since you probably don't want to have to implement your own GC.

[elia]

Not familiar enough but would be great if we could give alternative more performat implementations of subsets of the code, that would also allow for a gradual transition I think.

[ylluminate]

Was probing a bit here, might give a touch of insight: oracle/truffleruby#82

@eregon, you have any insights on what may need to be done from your perspective of working in the truffleruby arena? Realize it's not the same, but thinking with your exposure to LLVM you might have some thoughts...

[elia]

I'll close this issue for now. We'll surely reconsider wasm if something viable comes down the pike.

See also #1715

[erlend-sh]

Maybe it's time to revisit WASM?

http://www.blacktm.com/blog/ruby-on-webassembly

[elia]

@erlend-sh yes, the best route would be to start with replacing some performance-critical small parts with WASM implementations, and then gradually migrate the whole codebase. If anything like this comes up (a PR would be great but an example is good too) then we can start considering WASM in a more practical way.

Last time I checked anything that wasn't already going through LLVM for the compilation was basically unsupported, also I think mixing WASM with regular JS wasn't possible (but I might be wrong on this last one).

[ylluminate]

I think a few points are relevant:

1. WASM is going to be critical to keep Opal relevant in the coming years. It would be both silly and a sad shame if something popped up to "compete" with Opal on the WASM front that was accepted ONLY because of its ability to run WASM ("hot new kid on block") or because of the previously absurd drama that we probably don't want to really rehash here.

2. JS WASM interaction is possible and becoming more possible. It will be critical for DOM interaction as far as I understand it until direct DOM is integrated. Perhaps this is relevant to review before starting to fiddle here. Yew (Rust Framework) implements a "Virtual DOM, independent loops, fine updates" here. There's also some discussion re: "The DOM, GC integration, and more" here in rust-wasm. Also relevant: wasm-bindgen - Interoperating JS and Rust code

3. A more sane approach for WASM support would probably be by using Emscripten since you get a LOT of support that will be needed until WASM is more mature. Ruby -> Emscripten -> JS or WASM will then be nice and readily possible without a big performance hit and will also support better DOM interop as far as I also understand. I don't think pure WASM is entirely desirable yet until a good deal of maturation happens over the next couple years. Just a word of warning, I have a pretty strong belief that we'd be stupid to ignore Emscripten and roll it all ourselves...

It may also be worth reviewing the thread here since the maker of Emscripten / Binaryem has some solid and important input surrounding Crystal in WASM. Crystal's main hangup is twofold: a) garbage collection (which seems doable now since this was resolved), and b) DOM access. I've always thought tying Crystal into Opal vs having it a separate project would make a lot of sense from a marketing standpoint, but ... I know others seem to perhaps have some different feelings on the matter. There's a lot of interest, for example, in making a Crystal Hyperloop implementation.

I believe the idea of getting started in WASM is still pretty nebulous to most of us. The DOM issue feels like a real dealbreaker if we don't get our hands around it fully (it does appear to be entirely possible if you can grok the big picture) since it's relevant to probably about 95% of all projects that would really want to use Opal, etc.

Maybe we should start off with a Google Doc that discusses the DOM issue / situation based on above (and / or other) information and then following that we can have a discussion and further writing to document an implementation before digging in to anything substantial?

[iliabylich]

From what I understand LLVM IR is statically typed, which means that we need to have a VM to handle complex stuff. We can't do a simple source-to-source compilation.

MRuby can handle some simple things, but it's not a full replacement of MRI (for example, MRuby doesn't support kwargs, it handles differently post-args, and that's only what I've got from the quick 1 min check). It was initially designed as an "embedded ruby", so I guess it's wrong to require MRuby to do such things.

I think it's easier to partially compile MRI as it has VM, GC and other cool stuff that is required. Please, correct me if I'm wrong, I'm open for this discussion, but I don't see what we can do here.

[ylluminate]

Hmm, I think the MRuby approach may be the most sane entry point as per @blacktm's article from above regarding MRuby. His repo is interesting. As per his article, the size he notes for just shipping MRI is indeed size prohibitive (I don't think we want to ship a 20-40 MB runtime over the web)...

We might be better off trying to figure out and push efforts forward in MRuby that are missing from MRI that we want / need such as kwargs.

Is there another vector of approach? I suppose so, but we might end up with something more akin to an even more Ruby-like Crystal in the end vs true MRI Ruby. Is that a bad thing? Maybe not... maybe so...

Definitely needing some more input all the way around here so we don't shoot ourselves in the foot with a lot of wasted effort and time in a poor or less optimal solution.

Another thought, could the work from Truffle Ruby be used in some fashion? I guess I'm not familiar enough with how they use LLVM as per @chrisseaton's remarks there.

[blacktm]

Hey all. I wrote that post @erlend-sh linked to. It's worth noting that WebAssembly is, like, really simple. They refer to it as a "virtual machine," but it's not a VM as we think of them. Rather, it's just a slight hardware abstraction — instructions, types, linear memory, that's about it (details here). The reason MRuby works on WebAssembly, as described in my post, is because they've implemented a VM, GC, and everything else you'd expect a stand-alone Ruby implementation to have. To my knowledge, Opal relies on JavaScript for all that. This is one of the reasons JavaScript is most likely never going away — browsers already ship with a mature and performant (and large) VM, scriptable with JavaScript. Therefore, there's really no reason to "compile" JS to WebAssembly, since you would have to ship an additional complete VM with your app code. (See this robust discussion in the WebAssembly/design repo about whether there will be a JS to WASM compiler.) For clarity, we could describe Opal as a "Ruby implementation built on the JavaScript VM." With all that said — and sorry to be blunt — I don't think WebAssembly will be relevant to Opal.

[ylluminate]

Hmmmm. That's a very interesting remark with wide reaching ramifications. I guess this, by extension, begs the question: Is WASM relevant to Ruby generally?

I appreciate the frankness and clarity of the thought process @blacktm.

I suppose it makes sense for a language like Crystal where it's based on LLVM and there seems to be a clear path, but given that there is already a very robust VM in place (JavaScript) and WASM doesn't facilitate that which is needed (yet), then the idea of Ruby atop WASM may be relegated to simply an interesting topic and something to think about until WASM is more mature and essentially facilitates what JavaScript VMs already provide... and even when that happens, how much longer past that point will it finally become feasible given adoption / uptake of web browsers capable of running it. In all reality we may be looking at 5+ years unless things drastically change.

Perhaps it would be worthwhile "playing" in this area and seeing where things go, especially with the MRuby path, but Opal does seem to perhaps be ideally situated for at least the time being.

So a big "hmmmmmm" and I think we're all definitely interested in seeing some other perspectives percolate up here...

[pannous]

FYI: I uploaded and hosted the mruby to wasm experiment by blacktm[0] :
https://github.com/pannous/wruby

Thanks to emscripten mruby really works out of the box in wasm, but can be optimized (loading mrbc bytecode on the fly, replacing window.prompt in irb etc)

[0] mruby/mruby#3709

Maybe wasm will become relevant for opal again once the JS VM can be accessed directly from wasm (planned in the next release of webassembly).

[elia]

Maybe wasm will become relevant for opal again once the JS VM can be accessed directly from wasm (planned in the next release of webassembly).

yes, that would open a bunch of possibilities 🤩

[go2null]

There is also run.rb as per the Awesome WebAssembly Languages list.

It would be great if there was more cooperation to reduce effort duplication in this space.

[ylluminate]

@go2null it is interesting, but I don't think they're entirely compatible. It appears that all that run.rb is doing is to compile the MRI to run in browser. Right now they don't have the stdlib and who knows how much size that's going to add to the payload... I mean, I don't want to deploy Ruby web apps that are enormous download sizes...

Kind of a crazy thought, but I wonder if it would be possible to rework the stdlib so that it's highly modular and only what's needed would be pulled in as it's needed... I'd think that would be an ENORMOUS job.

[eregon]

Kind of a crazy thought, but I wonder if it would be possible to rework the stdlib so that it's highly modular and only what's needed would be pulled in as it's needed... I'd think that would be an ENORMOUS job.

Looks like you are describing MRuby to me 😉

[ylluminate]

LOL, yes, this is true @eregon - and this is something I've seen thrown around in terms of MRuby being the next Ruby to run via WASM. I guess I've not spent enough time in it yet to understand just how compliant it is to MRI. From what I'm hearing with the last drop, it seems as though there is some serious effort involved to give it feature parity with MRI... and if that's the case, then MRuby could simply become the next mainstream Ruby (aside from Truffle) to really take the language forward to all platforms (including WASM). Are you seeing anything in the MRuby space that flows along these lines?

[eregon]

Final size and modularity of the stdlib is what MRuby shines at. See https://github.com/mruby/mruby/blob/master/mrbgems/default.gembox for instance about how each part (including IO, the parser & compiler, etc) can be included or excluded.
MRuby is "embedded Ruby" and given the current constraints of WebAssembly (notably, no direct access to the DOM/JS) I think they make a lot of sense together.

In terms of compatibility, I think it's good for most core classes which do not depend on low-level OS facilities (Process, Socket, etc), but it's a bit different to traditional Ruby, e.g., require/require_relative are not there by default, but then again in an embedded scenario you might not want to have a filesystem, parse and compile code at runtime, but instead compile everything to MRuby bytecode on the host system, and only ship the MRuby bytecode interpreter + the bytecode.

[go2null]

Two interesting links:

• https://github.com/artichoke/artichoke
• https://github.com/ruby-llvm/ruby-llvm

Artichoke already compiles to WebAssembly - https://artichoke.run/
(Last I checked, it also used MRuby, but plans to switch to CRuby/MRI.)

Another cool Artichoke TODO: artichoke/playground#6