RFC: p5.initSound method

[therewasaguy]

add initSound method

• add a method p5.initSound on the global p5 object. It starts the AudioContext on a user gesture. The user can specify which element should listen for a user gesture, otherwise it falls back to creating a button on the first p5 sketch like this:

^ that's the easiest usage, to call it before preload.

But it can also be used with arguments specifying which element should be used to initialize audio, and it returns a promise, or calls a callback.

function setup() {
  background(255, 0, 0);

  var myButton = createButton('click to start audio');
  myButton.position(0, 0);

  p5.initSound(myButton).then(() => {
    var mySynth = new p5.MonoSynth();
    mySynth.play('A6');
    background(0, 255, 0);
    myButton.remove();
  });
}

or

var myButton;

function setup() {
  background(255, 0, 0);

  myButton = createButton('click to start audio');
  myButton.position(0, 0);
  myButton.mousePressed(startAudio)
}

function startAudio() {
  p5.initSound();
  var mySynth = new p5.MonoSynth();
  mySynth.play('A6');
  background(0, 255, 0);
  myButton.remove();
}

or, similar to a custom preload, we can create a custom HTML element and stylize it however we want.

<button id="p5_init_sound">Click here to start</button>

p5.initSound('#p5_init_sound')

deprecate userStartAudio method

• deprecate p5.prototype.userStartAudio, which historically the same thing as initSound, but is on the prototype so cannot be called before preload. There is no need to have this method on the p5 instance, because the audio context is shared across instances (there should only be one audio context on the page)

Testing

To test in Chrome, go to People --> Add Person before loading a site so that (this resets your Media Engagement Index, which is used to determine whether autoplay is allowed on a given domain)

[stalgiag]

Hi @therewasaguy thanks for jumping on this! Sorry for the delay, I was taking a brief break from the computer. This is really exciting!

I wonder if the automatic adding of a button will feel obtrusive at times. I like it as an option but I wonder if there is a way that initSound could know if it is already in a gesture. For example, I am curious how this would work with something like:

function setup() {
  background(255, 0, 0);
}

function mousePressed() {
  p5.initSound();
  var mySynth = new p5.MonoSynth();
  mySynth.play('A6');
  background(0, 255, 0);
}

would I see a button push the canvas down for a brief second?

My only other concern is the member method dot syntax for p5.initSound(). I understand that this enables a ton of functionality (like calling outside of p5 functions) and that I proposed dot syntax in a previous comment 😛 , but it can be opaque for new learners. This is just a note for background consideration though.

[therewasaguy]

Thanks for taking a look @stalgiag ! Really appreciate your perspective here.

I wonder if the automatic adding of a button will feel obtrusive at times.

Yes, probably!

Some options come to mind, option 1 is my favorite, but I would love some input.

1. Only create a button if the user supplies an argument for a CSS selector that does not already exist. (There is already an arg to supply an existing element, or a p5 / DOM Element)

2. Never create a button - depend on the user supplying (and styling) the html element

3. [bad idea] Create a separate method p5.initSoundAndCreateButton. But this adds to the API.

Another path...

4. Rethink this approach, continue to use a method that is on the p5 context, like userStartAudio, and update examples to have something like

var btn;

function setup() {
  btn = createButton("Tap to start audio");
}

function mousePressed() {
  userStartAudio(btn, removeBtn);
}

function removeBtn() {
  btn.remove();
}

I like the clarity of 4, the main downside is that this would really clutter the examples.

dot syntax

I hadn't considered this, thanks for bringing it up. I guess this could lead to some confusion because the way we instantiate p5 sound object (i.e. new p5.MonoSynth()) refers to the p5 instance, and so can't be used outside of p5 functions.

This could be a reason to lean towards 4...

[stalgiag]

I like the flexibility of 4. It might crowd some examples, but perhaps it would be possible to simplify some examples by making the arguments to userStartAudio() optional and just requiring that it is placed inside a gesture callback?

Which I guess brings up some questions about how someone would write a simple sketch that played a sound upon mouse click. Would it look like this?

let sound;

function setup() {
  sound = loadSound('mySound.mp3');
}

function mousePressed() {
  userStartAudio();
  sound.play();
}

The meaning of userStartAudio() becomes a bit obtuse in this example. I don't know if there is any way around this problem without requiring a dom element for starting sketches.

Definitely a tough design decision. Perhaps @lmccart @outofambit @limzykenneth or any other contributors have thoughts?

[limzykenneth]

From what I can come up with, there's obtrusive and unobtrusive methods. Obtrusive ones will be once the coder include userStartAudio or p5.initSound (whichever we go for) in their sketch, the user will either see an element or some kind of message to instruct them to interact somehow. Unobtrusive will be just adding those functions in mousePressed() or otherwise and let the coder handle informing the users to interact with the sketch. Each have their own details as well as pros and cons.

I wonder can we combine the two so that if it is called in setup (or globally in the case of p5.initSound) we go for the obtrusive method and maybe grey out the sketch with a semi-opaque background with a message to click; then if it is called anywhere else the coder will need to take care of informing their users themselves.

[lmccart]

To me, having at minimum an unobtrusive version, similar to @stalgiag's snippet above makes sense.

Maybe the name userStartAudio is confusing, because you're not actually restarting it with each mouse press. I'd assume you would check behind the scenes and start the audio context only if it didn't already exist. Something like enableAudio, while still not quite accurate, feels a little less confusing. Or ensureAudio which is more accurate but maybe less intuitively named.

Additionally having an obtrusive method, that is, putting up default element to prompt user interaction, is an interesting idea. I do like the simplicity of being able to put userStartAudio in setup.

[therewasaguy]

Thank you everyone for your thoughtful comments, and happy 2020!

For now, I'm going to continue down the unobtrusive route and just revise all the examples to demonstrate the best practice of waiting for a user gesture to make any noise as described in https://github.com/processing/p5.js-sound/issues/388

I'm pretty sure this can work without any explicit call to userStartAudio, so maybe we can punt on (re-)naming this method. Though I do like ensureAudio.

For the inline documentation examples, I think we also should include some feedback to the user about how to start audio, and that it should be tied to canvas mousePressed events rather than anywhere on the page (otherwise sounds toggle while editing the example).

So that example would look something like

let sound;

function preload() {
  sound = loadSound('mySound.mp3');
}

function setup() {
  var cnv = createCanvas(100, 100);
  cnv.mousePressed(playSound);

  textAlign(CENTER);
  text('tap to play', width/2, 20);
}

function playSound() {
  sound.play();
}

For the long run, there are a few reasons I think it would be useful to have an initSound method that hooks into preload and ensures the audio context has started before setup is run:

• we can start the audio context outside of p5 functions (maybe this is not necessary)
• we can ensure that the audio context is started before setup and draw begin (but users can also handle this themselves)
• it would be nice to get rid of this warning in Chrome: The AudioContext was not allowed to start. It must be resumed (or created) after a user gesture on the page. It would take substantial refactoring to actually initialize the audio context on that user gesture, rather than simply resume/start an existing one. (https://github.com/processing/p5.js-sound/issues/389)
• If we could actually initialize the audio context on a user gesture, we could also load audioworklet modules at that point, rather than hooking into preload and p5's init hook.
• starting the audio context is asynchronous, because AudioContext.resume() returns a Promise. So far this doesn't seem to be an issue.

I wonder can we combine the two so that if it is called in setup (or globally in the case of p5.initSound) we go for the obtrusive method and maybe grey out the sketch with a semi-opaque background with a message to click; then if it is called anywhere else the coder will need to take care of informing their users themselves.

I like exploring this hybrid direction!

[therewasaguy]

I'm going to close this for now and keep working on revising the examples to use best practices with the api we already have, PR in progress here #403