Working with Audio Worklet

[jsroga]

Hey guys,
I have a hard time integrating WASM + Emscripten with Audio Worklet.
He's how we tried to do it:

1. We are downloading and compiling .wasm file from the main thread.
2. We are instantiating Emscripten's WASM module in the main thread

Module.instantiateWasm = (imports, successCallback) => {
       wasm.then((wasmBinary) => {
          WebAssembly.instantiate(new Uint8Array(wasmBinary), imports)
            .then((output) => {
              $window.Module.testWasmInstantiationSucceeded = 1;
              successCallback(output.instance, output.module);
            })
        });
        return {};
      };

3. We are sending wasmModule and wasmMemory to AudioWorklet, thanks to the patch you did on message port
4. From AudioWorklet thread we wanted to instantiate it by passing its' memory and module, like that:

AudioWorkletGlobalScope.Module['instantiateWasm'] = function(imports, receiveInstance) {
  AudioWorkletGlobalScope.wasmInstance = new WebAssembly.Instance(module, imports);
  receiveInstance(AudioWorkletGlobalScope.wasmInstance, AudioWorkletGlobalScope.Module.wasmModule);
  return AudioWorkletGlobalScope.wasmInstance.exports;
}

but this is problematic. To do that we need to import Emscripten glue-js output file in AudioWorklet context. So I've tried to do something like that: audioCtx.audioWorklet.addModule('wasm/sndt.js') but it didn't work since Emscripten is not prepared to work in the AudioWorklet context.

Best,
Jacek.

[niekvlessert]

Hi,

I'm following (kind of) the same path and I'm running into these issues as well. At a certain point I just tried to inline the Emscripten code in the worklet, that didn't work obviously. The console didn't give me a lot (yet), but I discovered it's not allowed to set variables outside of a function in the worklet. I stopped there because I figured it would be difficult to get my 1MB Emscripten 'blob' in 1 or more functions.

Since a lot of audio applications use transpiled C code I guess it would be a nice addition.

Regards,

Niek

[FalkorX]

I'm not sure this helps, but I link my C/C++ object files for use in the AudioWorkletProcessor with

emcc -s SINGLE_FILE=1 -s WASM=1 -s BINARYEN_ASYNC_COMPILATION=0 -s ASSERTIONS=0 -O0 -o <OUT>.js

and use Module['ENIVIRONMENT']='WORKER' in the pre.js. I register the Processor class in the post.js. Through this I can directly import the result with audioCtx.addModule(<OUT>.js). This way I don't even have to send the wasm from the main thread to the render thread, but load it directly there.

[niekvlessert]

@FalkorX Sure sounds like a solution! So you generate the whole worklet? Is there an example online?

The code that comes out overhere starts with

var Module;

if (typeof Module === 'undefined') Module = {};
...

Not a great start for a worklet I'd say. What would I add to the pre.js to get this to work?

I tried to add the class ... extends AudioWorkletProcessor { process { ... } } to the pre.js, but it ends up half way of the file, and the register to post.js, removing the --embed-file thing I have changed that, but the module is not loadable at all, since var Module; etc. is in there. Probably something wrong with the Module['ENVIRONMENT']='WORKER'

[FalkorX]

@niekvlessert Sorry, I don't have it online, and yes, I generate the whole worklet, no extra steps needed.

My pre.js is really only the line Module['ENVIRONMENT'] = 'WORKER'; and in the post.js, I define the processor and just use the Module there:

// post.js
registerProcessor('ProcessorName', class extends AudioWorkletProcessor {
  ...
  process(input, output) {
    ...
    Module.process();
    ...
  }
}

If you're worried about polluting your AudioWorkletGlobalScope's namespace, you can also just use the -s MODULARIZE_INSTANCE=1 linker option to keep the Module in a closure and -s EXPORT_NAME='"<MODULE_NAME>"' to give it a unique name.

[niekvlessert]

@FalkorX Amazing, I never thought I would be possible to create some javascript code, put a class behind it and then just use the class. I have one issue left; I want to access the Module in the process function in the class as you said, but the Module variable does not exist in the class, only outside of it?

[FalkorX]

That is not a problem, the Module will keep existing in the class's (and thus the process function's) closure.

var Module = ...; // *
...
class Proc {
  process() {
    Module.func(); // refers to *
  }
}

[niekvlessert]

@FalkorX My mistake, it works fine. I'm pretty sure I got undefined for Module, but it probably had something to do with the fact that I was using an old version of Emscripten and had some browser issues.

This is a nice method, for me at least, thanks!

[flatmax]

Hey there, this approach seems to give the following error :

/usr/local/emsdk-portable/emscripten/1.37.35/tools/eliminator/node_modules/uglify-js/lib/parse-js.js:272
        throw new JS_Parse_Error(message, line, col, pos);
        ^
TypeError: {} is not a function

Where my post.js contains the following :

class AudioProcessor extends AudioWorkletProcessor {
  process(inputs, outputs, parameters) {
    console.log('processed once and exiting')
    return false;
  }
}
registerProcessor('audio-processor', AudioProcessor);

Any suggestions ?

[niekvlessert]

Well, not right now, more information is required. Is the whole source online somewhere? Maybe, If it's easy to fetch and is ready to build on Linux, I can try to build it for you.

[flatmax]

Yes, here it is :
https://github.com/madChopsCoderAu/WASMAudio/tree/AudioWorklet

It is the AudioWorklet branch of that code base.

[FalkorX]

I had the same (or a similar) problem: the class and extends keywords are ES6 features, just as let, const and some others. The Emscripten uglifier currently only understands ES5. This is why you need to compile with -O0 or -O1, because it doesn't invoke the uglifier. -O2 and -O3 won't work.
However, the people here are alrady working on this issue, see #6041.

[flatmax]

I made this change - now post is correctly added. When run in the browser this is executed in test-element.html :

      runAudioWorklet(){
        if (this.context==null)
          this.context = new AudioContext();
        this.context.audioWorklet.addModule('libwasmaudio.js').then(() => {
          let oscillator = new OscillatorNode(this.context);
          let audioWorkletNode = new AudioWorkletNode(this.context, 'audio-processor');
          oscillator.connect(audioWorkletNode).connect(this.context.destination);
          oscillator.start();
        });
      }

There is now a bug where the code doesn't seem to be executed by the addModule command :

test-element.html:47 Uncaught (in promise) DOMException: Failed to construct 'AudioWorkletNode': AudioWorkletNode cannot be created: The node name 'audio-processor' is not defined in AudioWorkletGlobalScope.
    at context.audioWorklet.addModule.then (http://127.0.0.1:8081/components/test-element/test-element.html:47:34)

I have updated the repo branch AudioWorklet :
https://github.com/madChopsCoderAu/WASMAudio/tree/AudioWorklet

[flatmax]

OK - got it.
The problem is when you modularize : -s "MODULARIZE=1" -s EXPORT_NAME="'libwasmaudio'"
The code doesn't get called, once removed, then the WASM code gets compiled.

[ArnaudBienner]

I wanted to do the same thing and I managed to get this done by using es6 modules.
They seem to be widely supported now (Edge, Chrome, Safari and current Firefox beta according to 1).

As suggested in #6284 I added export default Module; using --post-js option then use import Module from 'myLib.js'; in my other JavaScript module. This other JavaScript module could be the one defining defining the AudioWorklet class and being imported in the AudioWorkletGlobaleScope using addModule.

[stale]

This issue has been automatically marked as stale because there has been no activity in the past year. It will be closed automatically if no further activity occurs in the next 7 days. Feel free to re-open at any time if this issue is still relevant.

[TimDaub]

Hi,

I'm running into the same problem now again. Since your conversation, defining ENVIRONMENT as worker in pre.js has been deprecated. Now it's recommended to use the argument -s ENVIRONMENT=worker.

However, an AudioWorklet is not a worker. If we still use the above used approach, then we'll run into the problem that in a Worklet WorkerGlobalScope.self is not defined.

If I simply compile without an environment parameter, add my registerProcessor call as --post-js, I run into the following error:

Cannot assign to read only property '__wasm_call_ctors' of object '[object Object]'

Essentially a -s ENVIRONMENT=worklet option would be nice.

Edit:

The solution by @ArnaudBienner seems to work, requires though a browser that can handle es modules. So as long as every browser supports audioworklets and es modules, it's possible. IMO however there should be a way to do it without modules.

[Archie3d]

Hi,

What is the latest state of this issue?
I am compiling with -s ENVIRONMENT=worker but the audio worklet still fails to load:

// in generated glue code js
if (ENVIRONMENT_IS_WORKER) {
 _scriptDir = self.location.href;
}

with the error Uncaught ReferenceError: self is not defined.

Also further down it is looking for window or importScripts which are not available from within an audio worklet:

if (!(typeof window === "object" || typeof importScripts === "function")) throw new Error ("...");

Now if I don't compile it as worker environment, I get another exception because my WebAssembly uses Pthreads and Fetch API (don't even know whether it is possible to use threads from within audio worklet).

[flatmax]

The simple AudioWorklet C++ project has moved to here : https://github.com/flatmax/WASMAudio/tree/AudioWorklet-litelement

It was working before, but unfortunately something has changed. It now returns a new error :
DOMException: Failed to construct 'AudioWorkletNode': AudioWorkletNode cannot be created: The node name 'audio-processor' is not defined in AudioWorkletGlobalScope.

You can check some of the old tricks to getting it working on that branch ... if you get past the problem I mentioned link back.

[Archie3d]

@flatmax thanks, I guess it's because how post-js (which contains the worklet class) gets appended with MODULARIZE=1 - it goes inside the generated module now, and not in a global scope.

[kripken]

--extern-post-js is an easy way to append code outside the modularize scope.

[flatmax]

Thanks for the tips! When I use the --extern-post-js the browser can't find the libwasmaudio module - probably because it hasn't been compiled in the browser yet. The binded js file starts with this : var libwasmaudio = (function() { and this function ends with this : })(); I would expect the browser to have executed that function which compiles the WASM in the browser after executing the following in the webapp :     this.context = new AudioContext();     this.context.audioWorklet.addModule('libwasmaudio.js').then(() => { But for some reason it doesn't seem to run the libwasmaudio function from the binded libwasmaudio.js file.

…

On 15/7/20 3:56 am, Alon Zakai wrote: |--extern-post-js| is an easy way to append code outside the modularize scope. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#6230 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFLUBYFKGPXPACTNTEEW6TR3SL47ANCNFSM4EQI6PGQ>.

[flatmax]

Got it working again.

I added an external post.js file which runs the module like so : libwasmaudio();
Now the module runs in the browser as expected.

If anyone needs an AudioWorklet and Emscripten reference in the future, checkout WASMAudio (the AudioWorklet litelement branch) :
https://github.com/flatmax/WASMAudio/tree/AudioWorklet-litelement

[Archie3d]

Thanks, it works, unfortunately I still cannot use Pthreads since Worker in not defined in audio worklet's scope :(

[flatmax]

I also get problems when I add "-s ENVIRONMENT=worker" to [the bind command].(https://github.com/flatmax/WASMAudio/blob/AudioWorklet-litelement/src/Makefile.am#L39) :

@emcc --bind --llvm-opts 1 --memory-init-file 0 -s ENVIRONMENT=worker -s MODULARIZE=1 -s EXPORT_NAME="'libwasmaudio'" -s SINGLE_FILE=1 -s "BINARYEN_METHOD='native-wasm'" -s WASM=1 -s ALLOW_MEMORY_GROWTH=1 -s BINARYEN_ASYNC_COMPILATION=0 -s ASSERTIONS=0 $(AM_CXXFLAGS) -O1 --pre-js ../js/pre.js --post-js ../js/AudioProcessor.js --extern-post-js ../js/post.js .libs/libwasmaudio.so -o .libs/libwasmaudio.js

The browser reports :
libwasmaudio.js:106 Uncaught ReferenceError: self is not defined

[Archie3d]

Right, I had to add something like this to pre.js

var self = {
    location: {
        href: "https://localhost:4443" // URL where the module was loaded from
    }
}

[boourns]

I followed this tutorial, and it worked for me to load an emscripten-compiled C++ class into an AudioWorklet:

https://timdaub.github.io/2021/02/25/emscripten-wasm/

[lmiguelgato]

@boourns the SINGLE_FILE option suggested by that tutorial creates a JS file with the WASM embedded in it (no separate WASM binary). It is equivalent to use the -s WASM=0 option. The problem with this approach is that the resulting module is bigger in size (i.e. the JS + WASM files separately are smaller in size that a single JS file with WASM embedded). In addition, it runs slower than with the WASM binary as a standalone file.

In my personal experience with WASM in audio worklets, the SINGLE_FILE option increases the size in around 30 %.

Quoting the Emscripten FAQs:
"Compile with -s WASM=0 to disable WebAssembly (and emit equivalent JS instead) [...]
-s WASM=0 output should run exactly the same as a WebAssembly build, but may be larger, start up slower, and run slower, so it’s better to ship WebAssembly whenever you can."

Quoted from:
https://emscripten.org/docs/getting_started/FAQ.html?highlight=single_file#what-is-no-webassembly-support-found-build-with-s-wasm-0-to-target-javascript-instead-or-no-native-wasm-support-detected

[sbc100]

IIUC SINGLE_FILE is not the same as WASM=0 in that it does not convert the wasm binary to JS, but instead just embeds the wasm binary in the JS file (as a base64 string or similar).

[lmiguelgato]

IIUC SINGLE_FILE is not the same as WASM=0 in that it does not convert the wasm binary to JS, but instead just embeds the wasm binary in the JS file (as a base64 string or similar).

Thanks @sbc100 for the correction. Does it mean that SINGLE_FILE produces only a higher module size, and not a lower speed?

[sbc100]

The speed of SINGLE_FILE should be the same because the WebAssembly module that gets run will be identical.

It is probably more bytes over the wire because the way the binary gets encodes in the JS text. Its also fewer requests over the wire.. so maybe slower download due to less download parallelism?

[lmiguelgato]

The speed of SINGLE_FILE should be the same because the WebAssembly module that gets run will be identical.

It is probably more bytes over the wire because the way the binary gets encodes in the JS text. Its also fewer requests over the wire.. so maybe slower download due to less download parallelism?

@sbc100 is there a way to reduce the JS size when using the SINGLE_FILE flag? I mean, is there a way to change the encoding so that the JS size is smaller? Either by using some Emscripten flag or some other hacking?

For example, I have a Wasm binary of size 5.84 MB + ~200 KB of the JS glue code (already using -Oz during compile and link). However, using SINGLE_FILE for its use inside an audio worklet, I get 8.02 MB of the single JS.

It looks like a huge loss.

[sbc100]

Today the wasm file gets encoded as base64 here:

emscripten/emcc.py

Lines 3250 to 3253 in 9d63033

	if settings.MINIMAL_RUNTIME:
	js = do_replace(js, '<<< WASM_BINARY_DATA >>>', base64_encode(read_binary(wasm_target)))
	else:
	js = do_replace(js, '<<< WASM_BINARY_FILE >>>', shared.JS.get_subresource_location(wasm_target))

Its possible that you could use a more compact encoding perhaps.

Is this related to the original issue? Are you trying to use a single file to solve some worklet related issue? If not, perhaps we should move this discussion to a separate issue?

[the-drunk-coder]

It seems like I'm still missing something, I followed the advice above (setting self in pre.js) and use the following flags (with emcmake):

set(CMAKE_EXECUTABLE_SUFFIX ".wasm.js")
set_target_properties(mymod PROPERTIES LINK_FLAGS "--embed-file ${CMAKE_CURRENT_SOURCE_DIR}/build/assets/somefile -L${CMAKE_CURRENT_SOURCE_DIR}/libsamplerate-js/lib -lsamplerate --pre-js ${CMAKE_CURRENT_SOURCE_DIR}/glue/pre.js --post-js ${CMAKE_CURRENT_SOURCE_DIR}/glue/glue.js -s ALLOW_MEMORY_GROWTH=1 -s WASM=1 -s EXPORT_ES6=1 -s MODULARIZE=1 -s USE_ES6_IMPORT_META=1 -s ASSERTIONS=1 -s SINGLE_FILE=1 -s BINARYEN_ASYNC_COMPILATION=0 -s ENVIRONMENT=\"worker\"")

The post-js is occupied by the glue file generated by WebIDL, which I used to create the bindings.

Now, when I try to load the Module in the Worklet, I get the following error message:

mymod.wasm.js:9 Uncaught Error: not compiled for this environment (did you build to HTML and try to run it not on the web, or set ENVIRONMENT to something - like node - and run it someplace else - like on the web?)

What am I missing here ?

[lmiguelgato]

You are using the flag: ENVIRONMENT="worker", but it seems that you are not running the wasm code inside a web worker.
The js wrapper will check the actual environment at run time.
And a worklet is not strictly the same as a worker.

[the-drunk-coder]

Hmm I'm running the code in an AudioWorklet, yes ...

I just tried ENVIROMENT=shell, now I can at least load the module, haven't tried the real-time part yet ...

EDIT: using ENVIRONMENT=shell seems to do the trick, this way it works !