Replace ScriptProcessorNode in p5.SoundRecorder with AudioWorkletNode #369

oshoham · 2019-06-25T07:54:35Z

Polyfill AudioWorklet for non-Chrome browsers using audioworklet-polyfill (falls back to ScriptProcessorNode)
Rewrite p5.SoundRecorder to use an AudioWorkletNode instead of a ScriptProcessorNode
Use raw-loader to:
1. transpile an ES6 AudioWorkletProcessor class to ES5
2. import the transpiled ES5 source as a string
3. create a Blob from the source string
4. create an ObjectURL for the Blob
5. load the AudioWorkletProcessor via audioContext.audioWorklet.addModule using the ObjectURL
Use babel-plugin-preval to share the AudioWorkletProcessor name (passed to both registerProcessor and the AudioWorkletNode constructor) between the transpiled processor and p5.SoundRecorder
Take advantage of p5's init hook and preload system to synchronously load all AudioWorkletProcessor Blobs before the p5 setup() function runs

…WorkletNode; polyfill AudioWorklet

oshoham · 2019-06-27T03:37:34Z

@therewasaguy One issue that I haven't addressed in this PR is that the buffer size for the ScriptProcessorNode in p5.SoundRecorder was 1024, while the buffer size for an AudioWorkletNode is locked at 128. I haven't noticed any audible glitches resulting from this change, but presumably the decreased buffer size means an increase in overall CPU usage.

The recommended way of dealing with this AudioWorklet problem seems to be a ring buffer. That article from Google even suggests audio recording as an ideal use case.

Do you think I should look into adding a ring buffer as part of this PR, or should I work on that separately?

therewasaguy

This is fantastic, @oshoham ! Many great finds in this PR, and while I'd love to simplify sharing the AudioWorkletProcessor name, I think you've come up with a great solution given the constraints of the transpiled processor.

I think it's ok to look into the RingBuffer in a separate PR—great suggestion, I haven't noticed any glitches but buffer size mismatch seems like something we should handle.

therewasaguy · 2019-06-27T05:24:19Z

src/audioWorklet/.eslintrc

@@ -0,0 +1,9 @@
+{
+    "globals": {
+        "sampleRate": true,


ah, so this is global to the AudioWorklet global scope? Maybe worth adding the other attributes listed here?
https://www.w3.org/TR/webaudio/#AudioWorkletGlobalScope-attributes

yeah that's a good suggestion, I'll go ahead and add the other two global variables listed there

therewasaguy · 2019-06-27T05:34:17Z

src/audioWorklet/recorderProcessor.js

+    const buffers = this.getBuffers();
+    const leftBuffer = buffers[0].buffer;
+    const rightBuffer = buffers[1].buffer;
+    this.port.postMessage({ name: 'buffers', leftBuffer: leftBuffer, rightBuffer: rightBuffer }, [leftBuffer, rightBuffer]);


do we need the array?

we do, actually - this syntax transfers ownership of the buffers to the main thread rather than copying them, which (at least as I understand it) is faster

see https://developer.mozilla.org/en-US/docs/Web/API/Worker/postMessage#Transfer_Example

therewasaguy · 2019-06-27T05:54:56Z

src/audioWorklet/recorderProcessor.js

+      return true;
+    } else if (this.sampleLimit && this.recordedSamples >= this.sampleLimit) {
+      this.stop();
+      return true;


interesting, reading this:

Nodes that act as sources of output, typically with a lifetime. Such nodes SHOULD return true from process() until the point at which they are no longer producing an output

so this should always return true

yeah, my understanding is that you should only return false from a processor when you're ready for that node to be garbage-collected, although I could be wrong there

@oshoham

### New - AudioWorklet replaces the deprecated ScriptProcessorNode (when available) in p5.SoundRecorder, p5.Amplitude, and p5.SoundFile, as part of @oshoham's GSoC project https://github.com/processing/p5.js/blob/master/developer_docs/project_wrapups/orenshoham_gsoc_2019.md #369 #373 - p5 library compiled with webpack so it looks a little different and we had to remove comments that were tripping up YUIDoc here in the p5.js repo ### Fixes - p5.SoundFile: stop() won't stop a soundfile loop https://github.com/processing/p5.js-sound/issues/401 - p5.SoundFile: addCue() not triggering function calls https://github.com/processing/p5.js-sound/issues/371 - p5.SoundFile: jump() only called if soundfile is playing #404 - p5.SoundFile: save() bugfix #406 - remove bad console call (#378)

@oshoham

- improved inline documentation - all examples start audio on user gesture processing/p5.js-sound#403 - AudioWorklet replaces the deprecated ScriptProcessorNode (when available) in p5.SoundRecorder, p5.Amplitude, and p5.SoundFile, as part of @oshoham's GSoC project https://github.com/processing/p5.js/blob/master/developer_docs/project_wrapups/orenshoham_gsoc_2019.md processing/p5.js-sound#369 processing/p5.js-sound#373 - p5 library compiled with webpack so it looks a little different and we had to remove comments that were tripping up YUIDoc here in the p5.js repo - p5.SoundFile: stop() won't stop a soundfile loop https://github.com/processing/p5.js-sound/issues/401 - p5.SoundFile: addCue() not triggering function calls https://github.com/processing/p5.js-sound/issues/371 - p5.SoundFile: jump() only called if soundfile is playing processing/p5.js-sound#404 - p5.SoundFile: save() bugfix processing/p5.js-sound#406 - remove bad console call (processing/p5.js-sound#378)

Replace scriptProcessorNode in p5.SoundRecorder with transpiled Audio…

d90c3fc

…WorkletNode; polyfill AudioWorklet

oshoham requested a review from therewasaguy June 25, 2019 07:54

therewasaguy approved these changes Jun 27, 2019

View reviewed changes

Add remaining AudioWorkletGlobalScope attributes to .eslintrc

50a06b5

oshoham force-pushed the audioworklet-soundrecorder branch from e978673 to 50a06b5 Compare June 27, 2019 08:22

oshoham merged commit e03ab3d into processing:master Jun 27, 2019

oshoham deleted the audioworklet-soundrecorder branch June 27, 2019 08:25

therewasaguy mentioned this pull request Sep 4, 2019

p5.sound 0.3.12 processing/p5.js#3988

Merged

therewasaguy mentioned this pull request Jan 6, 2020

v0.3.12 #408

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Replace ScriptProcessorNode in p5.SoundRecorder with AudioWorkletNode #369

Replace ScriptProcessorNode in p5.SoundRecorder with AudioWorkletNode #369

oshoham commented Jun 25, 2019

oshoham commented Jun 27, 2019

therewasaguy left a comment

therewasaguy Jun 27, 2019

oshoham Jun 27, 2019

therewasaguy Jun 27, 2019

oshoham Jun 27, 2019 •

edited

Loading

therewasaguy Jun 27, 2019

oshoham Jun 27, 2019 •

edited

Loading

Replace ScriptProcessorNode in p5.SoundRecorder with AudioWorkletNode #369

Replace ScriptProcessorNode in p5.SoundRecorder with AudioWorkletNode #369

Conversation

oshoham commented Jun 25, 2019

oshoham commented Jun 27, 2019

therewasaguy left a comment

Choose a reason for hiding this comment

therewasaguy Jun 27, 2019

Choose a reason for hiding this comment

oshoham Jun 27, 2019

Choose a reason for hiding this comment

therewasaguy Jun 27, 2019

Choose a reason for hiding this comment

oshoham Jun 27, 2019 • edited Loading

Choose a reason for hiding this comment

therewasaguy Jun 27, 2019

Choose a reason for hiding this comment

oshoham Jun 27, 2019 • edited Loading

Choose a reason for hiding this comment

oshoham Jun 27, 2019 •

edited

Loading

oshoham Jun 27, 2019 •

edited

Loading