Poor offline rendering performance

[salomvary]

Rendering several minutes worth of audio is not only slow but also blocks the UI thread causing locked up user interface and some browsers prompting the user to "kill the script".

The rendering speed itself might be something we can live with (the price we pay for the abstractions) but since it is not possible to use OfflineAudioContext in a web worker (see also #435) it would be essential to optimize offline rendering to block the UI thread for as little time as possible.

The current performance makes offline rendering of more than a couple of minutes not feasible (and it's already a bad user experience).

I created a small benchmark that renders nothing but silence using Tone.js: https://jsfiddle.net/salomvary/4nq8mhsw/

Results by length and browsers:

Firefox:

• 1 hour: 20 seconds
• 30 minutes: 10 seconds
• 10 minutes: 3 seconds
• 3 minutes: 1 second

Chrome:

• 1 hour: 8 seconds
• 30 minutes: 4 seconds
• 10 minutes: 1.5 seconds
• 3 minutes: 0.5 second

Did the same test with plain web audio too: https://jsfiddle.net/salomvary/29bpxmf4/

Firefox:

• 1 hour: 240ms
• 30 minutes: 120ms
• 10 minutes: 50ms
• 3 minutes: 20ms

Chrome:

• 1 hour: 1700ms
• 30 minutes: 700ms
• 10 minutes: 280ms
• 3 minutes: 70ms

Hardware:

• 2,9 GHz Intel Core i5
• 16 GB 1867 MHz DDR3

[tambien]

Was able to remove some of the overhead by increasing the processing chunk size.

https://jsfiddle.net/yotammann/d2ero8z7/

[tchakabam]

Was able to remove some of the overhead by increasing the processing chunk size.

Interesting. Thanks for the hint!

We can also help with some better solution to reduce overhead given some advice from you maybe.

I was wondering, what is Tone really doing on top of the native OfflineContext? Checked the docs but it just says: Wrapper around the OfflineAudioContext.

Looking at:

Tone.js/Tone/core/OfflineContext.js

Line 71 in 7818531

this._currentTime += 0.005;

I understand that Tone does it's own scheduling on top of the offline context, which is of course necessary to pre-schedule all the events at some point.

Just using 5ms as an incremental value doesn't seem it's a general solution for any number of events needing to be scheduled. First of all this should be configurable I would say, if it has an impact.

But still this solution isn't really likely to scale with any length of tracks or number of events, no?

How about everything would be pre-scheduled in a more batched way that could indicate progress, with an async callback - just by emitting doing the actual tick events distributed in batches over several timeout callbacks? That would systematically take load of the main thread instead of trying to run it all on one (UI event loop not audio) tick, no?

It would just make the render method async / take a callback argument in some way.

Let me know what you think or if I am just completely off with my understanding of it :D

[tchakabam]

@salomvary Unfortunately it is not possible to use the native OfflineAudioContext in a Worker, and Tone just has a very thin wrapper around it to map the events scheduled on the Transport to the native scheduling calls.

The rendering speed itself might be something we can live with (the price we pay for the abstractions)

Just in defense of Tone :) This isn't something we would/could do much differently without using Tone's event scheduling engine. In the end these calls have to happen on the native API, and thus they need to be called in main thread. The only solution (with or without Tone) can be to distribute calling these scheduling methods in a more distributed manner afaui (in terms of not running through that whole loop in one tick, but via several timeouts reasonably spaced, and invoking a completion callback when done), which will avoid the UI event queue to get backlog.

[tambien]

@tchakabam that value that i changed used to be 1 block size (128 samples) or roughly 0.3ms, i changed to 5ms. This value is the time between successive "tick" events which would then schedule a callback in Tone.Clock to process any events since the last "tick". So increasing the processing size decreases the number of tick calls, but all events since the last tick are still evaluated, so it does not decrease any kind of timing resolution.

Calling these tick events is what allows things scheduled to the Transport to be rendered in the offline context. One simple optimization is to only invoke "ticks" when there are scheduled events on the clock. This would optimize out the situation when Offline is used but the Transport is not, and would help in this specific benchmark since nothing is scheduled to the Transport.

I like your suggestion of making those callbacks async so they don't block the main thread as much. It could defer the next call between successive loops using a setTimeout, but i assume the rendering would take much longer. and it would be especially wasteful if there was nothing scheduled on the Transport. Since the OfflineAudioContext is decoupled from the Transport by only passing "tick" events, the Offline rendering doesn't know if it needs to do this optimization or not.

Another option could be to pass in these as parameters. to the "render" method and put it on the user to decide how it should render.

[salomvary]

@tambien Thanks for the quick response! Just to clarify I did not propose optimizing for "nothing scheduled" aka. silence (although it's nice to be fast when doing "nothing") but it was the simplest way to report the performance overhead I found in Tone.js while optimizing our own code.

Will report back how much this helped making our use case faster and maybe post a more elaborate offline rendering benchmark (something that does real scheduling for a long offline rendering).

[Smona]

fwiw, i've dealt with this problem by displaying ui for rendering in a popup window and showing rendering progress and allowing cancelling from the popup, passing progress data through localStorage. i don't like having to use a popup for this, but i need to be able to display a progress bar for such a long-running operation.

The ideal solution would be if we could render audio from the web audio api in a separate thread (or better yet, on a server -- ouch). But i'm wondering if it would be feasible to use an interval of 0-1ms instead of a while loop to schedule ticks. While this would certainly slow down rendering, it would allow the main thread to pick up draw calls off the call stack and not completely freeze up the dom. From profiling it seems like the majority of rendering time is spent in this loop, with the offline rendering itself going pretty quickly. I think most developers would take a slightly longer render time over complete blocking of the main thread.

[salomvary]

@Smona interesting trick, I wonder how it actually solves the problem of not blocking the user interface. As far as I know a popup window opened to the same domain runs on the same thread. But with Web Audio + Web Workers this might be more complicated.

Btw if popups are not an acceptable experience the same might be doable with an invisible iframe.

[tboie]

@Smona, offline rendering in a separate environment appears to be possible using puppeteer:

https://github.com/GoogleChrome/puppeteer

Just rendered a .wav file to disk using Tone.Offline

[tambien]

Been looking into this a little in the past couple of days. Here's my findings and updates:

I've made it so that the clock loop no longer blocks the UI thread by adding a setTimeout for every 1 second of audio which is rendered. Other than that, there doesn't seem to be a lot more that i could do at the moment other than small incremental optimizations which don't seem to shave a significant amount of time off the rendering loop.

In real world use-cases the bulk of the time is spend on the Web Audio side not in the clock rendering side. The timing examples that you show initially only seem to be to illustrate the amount of time it takes these platforms to allocate a large silent buffer, and not really how long it takes to do any rendering since the rendering time is proportional to the amount of audio nodes and scheduled events, not just the duration.

Here's the timing of rendering about 750 notes with Tone.Offline in a 6 minute buffer:

clockRendering: 2911.427978515625ms
startRendering: 50019.18701171875ms

The clock doesn't seem to add that much time compared to the native audio rendering in this case.

Thanks again for taking the time to make these benchmarks and for this helpful discussion. For now, I'm going to close this, but i'll continue to tag questions and future optimizations to this thread.

[salomvary]

@tambien thanks for the update!

[tchakabam]

@tambien Great! 🎉

We had no time to work on this either unfortunately, as it wasn't high enough in our priorities to optimize this further. Please let me know in case there would be any other work to do in this area or another! Cheers