Use QuickJS instead of Rust: Rust requires at least 1GB disk space to install

[guest271314]

Is your feature request related to a problem? Please describe.

The last time I checked Rust requires at least 1GB of disk space to install. When running a live OS using RAM starting out with 1GB of disk space installing Rust is prohibitive, essentially not possible.

Describe the solution you'd like

Develop the server application relying on QuickJS or txiki.js. QuickJS after strip is less than 1MB. tjs (txiki.js) executable after building is 7.6MB without strip and supports WASM.

Describe alternatives you've considered

I proposed WinterCG take up specifying a server that can be used by Node.js, Deno, Bun, QuickJS, txiki.js. The reply was that is not in the scope of their goals.

Options are limited when Rust is a requirement.

Additional context

We only have to build the network binding once. Then we don't have to rely on 1GB+ Rust installation anymore just to compile a local server.

[Angelmmiguel]

Hello @guest271314 !

Thank you for submitting the issue 😄

The last time I checked Rust requires at least 1GB of disk space to install. When running a live OS using RAM starting out with 1GB of disk space installing Rust is prohibitive, essentially not possible.

Rust is only requirement to develop and build wws. However, we always release a set of pre-compiled binaries, so you don't need to compile the project. You can download the latest binaries from the release page.

Develop the server application relying on QuickJS or txiki.js. QuickJS after strip is less than 1MB. tjs (txiki.js) executable after building is 7.6MB without strip and supports WASM.

Currently, QuickJS is the engine that allows you to run JavaScript files as workers. This specific engine is embed in the wws binary, so you can develop and run JavaScript workers directly. You have more information about it on [our documentation].

Regarding txiki.js, we don't have plans to add new JavaScript engines soon, but we may consider it as an alternative in the future.

I proposed WinterCG take up specifying a server that can be used by Node.js, Deno, Bun, QuickJS, txiki.js. The reply was that is not in the scope of their goals.

I may miss some context here. Wasm Workers Server is not a full WinterCG compatible server. We are following the Request / Response model, although we several APIs defined in the Minimum Commom Web Platform API. However, Wasm Workers Server is not limited to JavaScript. We also support other languages like Rust (via wasm compilation), Ruby, Python, and more to come in the future.

We discussed in the past the possibility of supporting multiple engines for the same language, but for now we're selecting one per language.

We only have to build the network binding once. Then we don't have to rely on 1GB+ Rust installation anymore just to compile a local server.

Talking about workers, the only language that requires a compilation step before running the worker is Rust (Zig, Go and others will come in the future). Interpreted languages like JavaScript, Ruby and Python run directly inside the WebAssembly engine.

[guest271314]

I installed the binary with this

curl -fsSL https://workers.wasmlabs.dev/install | bash -s -- --local && \
  ./wws --help

index.js in the directory. When I run ./wws .:

$ ./wws .
⚙️  Loading routes from: .
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Is a directory (os error 21)', src/router/route.rs:57:64
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

So I think there is an issue in your Rust code or I am missing something. After readng your documentation the server should run out of the box.

[Angelmmiguel]

Ohh I see @guest271314! That's a trace from the program, so you spotted a bug there :). The wws binary doesn't require Rust, but errors like this shows a Rust backtrace as they were not properly handled.

This error indicates wws tried to read a resource in the filesystem as a worker, but it's a directory instead.

It would be great if you can provide more context about your environment:

• What's the operating system you're using?
• What folders / files are in the folder you run ./wws .?

[guest271314]

Linux xubuntu 5.15.0-43-generic #46-Ubuntu SMP Tue Jul 12 10:30:17 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

I just downloaded the index.js file to a directory where wws is located, and other directories are, including a shell script with .sh extension.

[guest271314]

When I create a dedicated directory and place the executable and index.js no errors are thrown. Now I can stress-test. I am streaming raw PCM from the local server to the browser.

[guest271314]

After setting Access-Control-Allow-Origin to '*' the header does not appear to be set

 if (request.method === "OPTIONS") {
    // Don't allow other methods.
    // Here you can see how to return a custom status
     let response = new Response(null);

  // Add a new header
    response.headers.set("x-generated-by", "wasm-workers-server");
    response.headers.set("Access-Control-Allow-Origin", "*");
    response.headers.set('Access-Control-Allow-Private-Network', 'true');
    response.headers.set('Access-Control-Allow-Headers', 'Access-Control-Request-Private-Network,Cache-Control,Last-Event-Id');
    response.headers.set('Cache-Control','no-store');
    return response;
 }

fetch('http://127.0.0.1:8080/', {headers: {'Access-Control-Request-Private-Network':true}})

at 'http://127.0.0.1:8080/' from origin 'https://github.com' has been blocked by CORS policy: Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.

[guest271314]

Alright I got it working. We can't return a response to the OPTIONS request alone, and passing Uint8Array to Response doesn't appear to work.

[guest271314]

Currently, QuickJS is the engine that allows you to run JavaScript files as workers.

You have more information about it on [our documentation].

I didn't locate any information in your documentation about how to use qjs runtime in the JavaScript.

./wws runtimes check does not list QuickJS.

I think you can use QuickJS to achieve the requirement without Rust at all. qjsc compiles C to an executable. Rust is rather expensive to install and run on a temp filesystem in RAM.

[Angelmmiguel]

When I create a dedicated directory and place the executable and index.js no errors are thrown. Now I can stress-test. I am streaming raw PCM from the local server to the browser.

Regarding this error, did you have other folders in the directory you used the first time? I would like to check why this is happening as wws should navigate through subfolders directly.

Alright I got it working. We can't return a response to the OPTIONS request alone, and passing Uint8Array to Response doesn't appear to work.

Thank you for reporting this! Definitively, Uint8Array is a missing part of the JavaScript support. We currently support binary content in Rust workers, so this should be available for JavaScript and other languages too.

I didn't locate any information in your documentation about how to use qjs runtime in the JavaScript.

./wws runtimes check does not list QuickJS.

The QuickJS engine is embedded into the wws binary. Since it's a small engine (3MB), we decided to make it part of wws instead of having to install via wws runtimes. You can find the QuickJS build in the kits/javascript folder.

To build it, we're using the QuickJS bindings that the Shopify team created for the Javy project. Then, we use a custom entrypoint to load the script and encode / decode output. We may revisit this in the future and compile QuickJS directly as we're doing with other languages in WebAssembly Language Runtimes.

I think you can use QuickJS to achieve the requirement without Rust at all. qjsc compiles C to an executable. Rust is rather expensive to install and run on a temp filesystem in RAM.

Once wws is built, the QuickJS is embed as a Wasm binary file. We don't require to have Rust installed for supporting JavaScript workers. You only need Rust in your system to build Rust workers.

[guest271314]

Regarding this error, did you have other folders in the directory you used the first time?

Yes.

The QuickJS engine is embedded into the wws binary. Since it's a small engine (3MB), we decided to make it part of wws instead of having to install via wws runtimes. You can find the QuickJS build in the kits/javascript folder.

Right, however the functionality of qjs is not exposed as far as I can tell. Can't call std.in.read() and so forth.

The last time I tested a few days ago the server couldn't keep up with raw PCM stream.

It is challenging to even install Rust on system running with a temp file system and RAM. The Rust tool chain is well over 1GB. It is far simpler to fetch QuickJS source code from GitHub and run make.

There are several QuickJS developers interested in a server for QuickJS. Thus this feature request to just develop QuickJS from the ground up making it work - because of that less than 1MB after strip rather than spend at least 1GB for Rust tool chain to use the 3MB executable.

[Angelmmiguel]

Right, however the functionality of qjs is not exposed as far as I can tell. Can't call std.in.read() and so forth.

The last time I tested a few days ago the server couldn't keep up with raw PCM stream.

These are good points. Since qjs is wrapped into a Rust project + bindings, the common qjs features are not exposed. For this reason, we are looking to add QuickJS to Wasm Language Runtimes based on the C source code directly. This will allow you to use it with wws or as a standalone wasm module.

It is challenging to even install Rust on system running with a temp file system and RAM. The Rust tool chain is well over 1GB. It is far simpler to fetch QuickJS source code from GitHub and run make.

Yes, I understand this concern about resource-restricted environments. However, for wws you shouldn't need to compile your own QuickJS Wasm module. We will work on having pre-compiled quickjs binaries that are ready to use.

Said that, we always leave wws open so you can build your own repository of binaries.

There are several QuickJS developers interested in a server for QuickJS. Thus this feature request to just develop QuickJS from the ground up making it work - because of that less than 1MB after strip rather than spend at least 1GB for Rust tool chain to use the 3MB executable.

This is an interesting use case. Do you have any reference, article or conversation I can look at? I would like to learn more about this and how wws can cover it.

Thank you!

[guest271314]

Sure https://www.freelists.org/post/quickjs-devel/tcpudp-sockets.

[Angelmmiguel]

Sure https://www.freelists.org/post/quickjs-devel/tcpudp-sockets.

Thank you! I will take a look at this. Most likely, I will split this issue into a bug fix for wws and a feature request for WebAssembly Language Runtimes :)

[guest271314]

The QuickJS runtime needs to be exposed. I don't think we need Rust at all but you are are dependent on Rust. The server needs to support sending streams, and at least handle streaming.

[guest271314]

Basically we can use Chrome or Chromium browsers as a local server for testing or as a remote server, so the functionality of the code in this repository needs to be able to emulate or implement a ServiceWorker onfetch event, including responding with a ReadableStream.

With the ability to use qjs runtime, else you are not really using QuickJS completely. We don't need to compile QuickJS to WASM.

[Angelmmiguel]

The QuickJS runtime needs to be exposed. I don't think we need Rust at all but you are are dependent on Rust. The server needs to support sending streams, and at least handle streaming.

This is a great solution for a lightweight JS server. However, it's out of the scope of the Wasm Workers Server project. This framework allows you to develop and run serverless apps in a WebAssembly sandbox. We decided to use quickJS to support JavaScript workers, although it's not the only language wws supports.

For this reason, I believe that having a full-fledged quickjs server (more like txiki.js) could be a different and interesting project.

[guest271314]

What I am saying is you claim to be using QuickJS, but you really don't.

You just happen to be using JavaScript that uses the QuickJS JavaScript engine for minimal JavaScript in your code. The observer can't tell what the JavaScript engine is, because the JavaScript used is so restricted.

Exposing the complete engine used should be user-defined, an option for the user, not restricted by default.

[Angelmmiguel]

What I am saying is you claim to be using QuickJS, but you really don't.

Currently, wws uses QuickJS to run JavaScript workers, although the engine is not directly exposed to the users.

You just happen to be using JavaScript that uses the QuickJS JavaScript engine for minimal JavaScript in your code. The observer can't tell what the JavaScript engine is, because the JavaScript used is so restricted.

This is a fair point, although it's the end goal of this project. wws isolates the users from the underlying engines by creating a common interface to develop workers. We started with QuickJS, but in the future we plan add support for Spidermonkey and others. Users won't require to change anything, but selecting the engine they want to use.

Exposing the complete engine used should be user-defined, an option for the user, not restricted by default.

In wws, we expose the API and hide the language runtime that runs it. In WebAssembly Language Runtimes, we do the opposite. We build artifacts with the language runtimes as they are, ready for consumption. We plan to add QuickJS there too, although the scope of that project is to make it available as a Wasm module, so the engine will be limited by the Wasm capabilities (ex. socket support, etc).

[guest271314]

I get it. It's your playground. What I find remarkable is all the folks who are using QuickJS as a JavaScript engine for a server, or otherwise implementing a worker type server are reluctant to act on user feedback and provide the user with the option to do stuff outside of what the maintainer arbitrarily restricts. Just ran into similar philosophy over here cloudflare/workerd#423, cloudflare/workerd#419, cloudflare/workerd#420, then the maintainers banned me.

It's the same story.

Anyway, at least you can handle constructive feedback withut going nuclear because I question your narrative or functionality of the code.

I think I'll do better persuading Chromium browser, whic correctly implements Streams and ServiceWorker, rather than maintainers of one of these repositories exposing user options which go against their restrictive philosophies - even when the application is out of their control.

[guest271314]

@Angelmmiguel FWIW see https://github.com/guest271314/webserver-c/tree/quickjs-webserver.