Audio glitches when screen is not being touched

[rpattabi]

When I try to play a steady frequency, after few seconds there are noticeable breaks in the audio, even while screen is still on.

Interestingly, when I keep tapping on the screen, there are no breaks in the audio.

I tested this with couple of Nexus 4 devices with Lollipop (stock android and CyanogenMod). On the other hand, I don't get this problem in emulator.

I was assuming pd-for-android would retain audio thread priority for the app.

By the way, there is no problem with simpler patches. My patch involved some reasonably heavy calculation, getting rid of this part from the patch avoided this problem. As I mentioned, even if that calculation part is kept and the screen kept tapped, no problem of audio breaks.

I am using v1.0.0, native sampling rate, 2 output channels, 1 input channel, and PdService.

[joebowbeer]

Apparently, glitchy audio playback has been a long-standing issue on Android, due to a variety of causes, probably, and with many different solutions offered:

https://code.google.com/p/android/issues/detail?id=22763

Assuming, on the other hand, that your issue is something we can fix, I wonder if requesting audio focus would address this:

http://android-developers.blogspot.com/2013/08/respecting-audio-focus.html
http://developer.android.com/training/managing-audio/audio-focus.html

The AudioWrapper class does attempt to raise the audio thread priority, but apparently that does not keep the audio device focused and alert. Eyeballing the code, it looks fairly straightforward to request audio focus on start and abandon focus on stop:

https://github.com/libpd/pd-for-android/blob/master/PdCore/src/org/puredata/android/io/AudioWrapper.java

@nettoyeurny Comments?

[nettoyeurny]

We should definitely do something about audio focus. A few thoughts off the top of my head:

• AudioWrapper is only there for backwards compatibility with ancient devices (Android 2.2 or earlier); as of Android 2.3, it's never instantiated.
• Potential places to add support for audio focus are PdAudio and PdService. My gut reaction is that PdAudio should only be responsible for managing audio playback, and that PdService is the place to put high-level functionality such as this.
• Do we want to bake this into a core class like PdService, or make a subclass like PdServiceWithFocus? Or maybe this can be implemented as a decorator.
• What does it mean if we fail to get the audio focus? Does that mean we can't play audio? Do we want PdAudio.startAudio to fail in that case, or log a warning and carry on, or something else? Are we willing to break backward compatibility in order to do this correctly?
• How do we handle audio focus callbacks? One of the goals of pd-for-android is to protect developers from the complexities of Android audio. Do we want to implement some default behavior, or create a simplified interface, or expose OnAudioFocusChangeListener in all its complexity?

[okramis]

I'm happy to find this thread, was searching for days now for this issue. I ported my "Gamelan Gender" from iOS to android using libPd, as there is no AUSampler like framework in android. Everything is fine so far, except on a nexus 6 on marshmallow, when the sequencer is running and there's no user input, the sound starts to crackle and gets distorted, that stops as soon I tap the screen or move a finger on the screen. Tried to implement the focus-thing to no avail, despite get granted focus, the crackling resists.

[nettoyeurny]

@okramis Thanks, this is a crucial bit of information.

Now that I've thought about this some more, and given okramis's post, I believe that handling audio focus is not unlike handling phone state (e.g., for the purpose of pausing playback when a phone call comes in). It's useful to have, and a complete audio app should probably pay attention to it, but it's not a core responsibility of pd-for-android and doesn't require access to the internals of core classes such as PdAudio or PdService.

My current take is to file this under "nice to have", and to add a decorator for PdService to the utils package if we want to provide support for this. We might even combine this with the handling of phone state (or would this be redundant --- maybe we lose audio focus when the phone rings?), and have developers interact with it through a simplified listener interface, say AudioInterruptionListener, with methods like OnAudioInterrupted(boolean canDuck) and OnAudioContinued().

[tkirshboim]

Since it seems that for both @rpattabi as well as @okramis touching the screen resulted in better audio performance, I thought it's worth mentioning that faking touch events and disabling the screen saver is apparently a hack that JUCE for Android uses to achieve better audio performance:
https://www.youtube.com/watch?v=vPc7zvfpUos&feature=youtu.be&t=1h21m6s

[rpattabi]

I was hoping that audio focus would solve this issue. I agree that would be 'nice to have' now since that is not solving this serious issue.

Other than the work around of artificial tapping on the screen (not sure how to do, need to look at how JUCE did it), is there anything pd-for-android do to address this? This is clearly pd-for-android's responsibility since it should never let the app to lose audio thread priority.

[okramis]

@rpattabi I agree, it should be fixed in pd-for-android. It might not so easy, as I think the issue lies somewhere in the aggressive power management of the nexus devices. I tried the artificial tapping - it is tapping - but unfortunately it does not help. It must have to do with the sensors becoming active, as I also can just rotate the device (portrait - landscape) to stop the crackling.

[rpattabi]

@okramis thanks for the information about artificial tapping not helping.

In my case, the problem is addressed by updating the pd patch, by drastically reducing the amount of processing during the playback.

[okramis]

I did some more tests on more devices. The only device with the issue is a Motorola Nexus 6 on android 6.x. I put the PdService in a thread raised process priority to THREAD_PRIORITY_URGENT_AUDIO, tried to simplify the pd patch as much as possible - no change. No activity on ui-sensors, the distortion starts after ~3 secs, also when the player runs in background. Touch on screen or change orientation not important in what app, the distortion stops... So I'm giving up, hope there'll be not to many devices out, showing this issue.

[rpattabi]

Android Audio team recognized this problem in last week's google I/O. You know what they recommend? Fake touches! Check this out: https://youtu.be/F2ZDp-eNrh4?t=9m0s

[rpattabi]

I found ripples in some UI elements using the API they suggested. So I used different API to do the same which doesn't impact the UI, provided you have a view that is just kept for this purpose (typically a text view that doesn't have to react to touch events).

Here is my code: https://gist.github.com/rpattabi/5252d38eb470e11bd4d0e8bb29f239bc

Please let me know how it works for you.

[okramis]

@rpattabi thanks a lot for sharing! I did some tests with the API from the video and indeed it solves the crackles. The only problem, this can't run in background (my app can play in background), so you have to watch out for visibility changes. I'll have a look at your class when there's some more spare time.

[okramis]

@rpattabi had a look at your class, unfortunately it won't work in my app, looks similar to my own attempt. I provided a TextView just for this purpose and the timer is firing, but crackling persists. Do I have to set some special properties on the TextView to have it working?

[rpattabi]

@okramis thanks for taking a look at my code. I was not sure it is working since I don't have audio glitches on my devices. But I thought this implementation is an improvement than what they recommended (which caused ripples in my app). Apparently I was incorrect. Android system seems to understand that my implementation's fake touches are not from real user. So instrumentation is the way to go.

I wonder how you could avoid ripples due to fake touches with their suggestion.

[rpattabi]

Android audio team talked about ideas to avoid glitches at Google I/O 17. See #66 for details.

[funkyfourier]

One advice from the Google I/O video was to use nop (no operation) assembly calls in the callback to keep the CPU busy, so it would not introduce scaling. Someone knows how to do that?

[rpattabi]

No idea about no-op assembly calls. Hope someone can answer. I would just like to add that it is now possible to request for sustaining performance from android N and above.

    public static boolean sustainPerformance(Activity activity) {
        boolean success = false;

        if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.N
                && powerManager(activity).isSustainedPerformanceModeSupported()
                && activity.getWindow() != null) { // NOTE: Window is available only after activity is visible

            activity.getWindow().setSustainedPerformanceMode(true);
            success = true;
        }

        return success;
    }

It shall be requested after activity is visible to the user, possibly like

    @Override
    public void onWindowFocusChanged(boolean hasFocus) {
        super.onWindowFocusChanged(hasFocus);

        if (hasFocus)
          sustainPerformance(this);

More info: Sustained Performance Mode

Update:

• Not all devices support sustained performance mode. So even if a device has android N, it doesn't mean that this mode is supported by the device.
• Even if sustained performance mode is supported by the device, it is enabled only if native sample rate is used. This was noticed through logcat warning/error message.
• Of course, this mode works only when the activity's window is in focus.

[funkyfourier]

@rpattabi Thanks. That will only help for N+, of course, but good to know.

There seems to be a workaround in the opensl callback function by keeping the CPU busy there, but how to do it is beyond me at this point.

I have hacked PdBase and PdAudio to force it to use AudioTrack. That actually worked quite well, and none of these issues arised. It would be nice to be able to choose between opensl and audiotrack, but it seems that requires changes to PdBase, and the native libraries would have to be loaded from some init method instead of from a static initializer.

PS: Yeah, I changed my username.

[rpattabi]

@funkyfourier I found that superpoweredSdk provided SuperpoweredCPU class that exposes a way to enable sustained performance mode. It used the no-op assembly calls which you mentioned. Unlike android's inbuilt sustainPerformanceMode, this allows sustaining performance irrespective of whether activity window is in foreground or not.

Unfortunately SuperpoweredCPU source is not public. So not sure what else they implemented in addition to no-ops.

I haven't tried it myself, it is just what I learned.

[funkyfourier]

@rpattabi Hi. That sounds interesting. However, as you point out, we do not have access to the source code.

I have been thinking that the way to go here is to use Google's oboe library rather than opensl_stream. It provides a simple API which is implemented on OpenSL ES or AAudio depending on what is available. It would be reasonable to assume that they have solved the CPU migration issues in oboe.

Once oboe matures a little bit more I might take a stab at it.

[rpattabi]

It would be interesting to watch oboe. Probably, we can make pd-for-android depend on oboe (after it matures) instead of opensl? I don't know. But I really would like pd-for-android is kept up-to-date on these aspects.

[cerupcat]

Has anyone made any progress on this? I'm using opensles, but background audio is choppy and slows down. The "fake touch" solution isn't working for me.

[rpattabi]

Google's oboe tries to ensure that our app uses fast track. Integrating oboe with libpd will eliminate the need for us having to do anything special. More info libpd/libpd#284.