Alternative audio capture API

[yume-chan]

I tried to use Audio Policy API to mirror audio instead. Audio Policy API uses the same underlying API as AudioPlaybackCapture API, but requires MODIFY_AUDIO_ROUTING permission instead of showing a popup every time. (Use it to capture audio is the counterpart of #3880 (comment))

Comparing to the Remote Submix API we are currently using, Audio Policy API has both pros and cons.

Pros:

• Can choose to continue playing audio on device
• Not affected by connected headsets
• Not affected by system volume
• If needed, can capture audio from each app separately

Cons:

• Requires Android 13, in which the Shell app has MODIFY_AUDIO_ROUTING permission added
• Apps can opt out from being captured. But IMO if Netflix shows a black screen in Scrcpy, not playing its audio also makes sense

Both methods can't capture notification, ringtone and alarm sounds.

---

The code is pretty simple, but again I'm using https://github.com/Reginer/aosp-android-jar/tree/main to unhide framework APIs:

package moe.chensi.tango

import android.annotation.SuppressLint
import android.app.ActivityThread
import android.content.Context
import android.media.AudioAttributes
import android.media.AudioFormat
import android.media.AudioManager
import android.media.audiopolicy.AudioMix
import android.media.audiopolicy.AudioMixingRule
import android.media.audiopolicy.AudioPolicy
import android.os.Build
import android.system.ErrnoException
import android.system.Os
import android.system.OsConstants
import org.joor.Reflect
import org.joor.ReflectException
import java.io.FileDescriptor
import java.io.IOException
import java.nio.ByteBuffer

class Test {
    @SuppressLint("StaticFieldLeak")
    private var _systemContext: Context? = null
    private val systemContext: Context
        @SuppressLint("DiscouragedPrivateApi", "PrivateApi") get() {
            if (_systemContext == null) {
                try {
                    // Hide warnings on XiaoMi devices
                    Reflect.onClass("android.content.res.ThemeManagerStub").set("sResource", null)
                } catch (_: ReflectException) {
                }

                _systemContext = ActivityThread.systemMain().systemContext
            }
            return _systemContext as Context
        }

    private fun writeFully(fd: FileDescriptor, from: ByteBuffer) {
        // ByteBuffer position is not updated as expected by Os.write() on old Android versions, so
        // count the remaining bytes manually.
        // See <https://github.com/Genymobile/scrcpy/issues/291>.
        var remaining = from.remaining()
        while (remaining > 0) {
            try {
                val w = Os.write(fd, from)
                remaining -= w
            } catch (e: ErrnoException) {
                if (e.errno != OsConstants.EINTR) {
                    throw IOException(e)
                }
            }
        }
    }

    @SuppressLint("MissingPermission")
    private fun captureAudio(fd: FileDescriptor) {
        if (Build.VERSION.SDK_INT < Build.VERSION_CODES.TIRAMISU) {
            return
        }

//        val privileged = true
//        val encoding = AudioFormat.ENCODING_PCM_16BIT
//        val sampleRate = 16000;
//        val channelMask = AudioFormat.CHANNEL_IN_MONO
        val privileged = false
        val encoding = AudioFormat.ENCODING_PCM_16BIT
        val sampleRate = 48000;
        val channelMask = AudioFormat.CHANNEL_IN_STEREO

        val rule =
            AudioMixingRule.Builder().setTargetMixRole(AudioMixingRule.MIX_ROLE_PLAYERS).addMixRule(
                AudioMixingRule.RULE_MATCH_ATTRIBUTE_USAGE,
                AudioAttributes.Builder().setUsage(AudioAttributes.USAGE_MEDIA).build()
            ).allowPrivilegedPlaybackCapture(privileged).voiceCommunicationCaptureAllowed(true)
                .build()

        val mix = AudioMix.Builder(rule).setFormat(
            AudioFormat.Builder().setEncoding(encoding).setSampleRate(sampleRate)
                .setChannelMask(channelMask).build()
        ).setRouteFlags(AudioMix.ROUTE_FLAG_LOOP_BACK_RENDER).build()

        val policy = AudioPolicy.Builder(systemContext).addMix(mix).build()

        val audioManager = systemContext.getSystemService(AudioManager::class.java)

        val result = audioManager.registerAudioPolicy(policy)
        if (result != 0) {
            println("registerAudioPolicy failed: $result")
            return
        }

        val record = policy.createAudioRecordSink(mix)
        record.startRecording()

        val bufferSize = 1024
        val buffer = ByteBuffer.allocateDirect(bufferSize)
        while (true) {
            buffer.position(0)
            val read = record.read(buffer, bufferSize)
            buffer.limit(read)
            writeFully(fd, buffer)
        }
    }
}

If AudioMix.ROUTE_FLAG_RENDER is used instead of AudioMix.ROUTE_FLAG_LOOP_BACK_RENDER, the audio no longer plays on the device itself. If using the commented configs, it can capture audio from protected apps, but at a very low quality (it's for the live caption feature).

[rom1v]

Implemented in #5102.

[rom1v]

voiceCommunicationCaptureAllowed(true)

In my code, I did the same, but I can't capture voice communication (during a call).

[zhaogezhang]

Excuse me, does the code support wechat voice call function recently? There is no voice in 2.5 version.

[yume-chan]

I didn't test call audio. I remember seeing AudioPlaybackCapture API also sets this flag.

[sidharthv96]

If AudioMix.ROUTE_FLAG_RENDER is used instead of AudioMix.ROUTE_FLAG_LOOP_BACK_RENDER, the audio no longer plays on the device itself. If using the commented configs, it can capture audio from protected apps, but at a very low quality (it's for the live caption feature).

@yume-chan does this include the earpiece/speaker audio during calls? Or is there some workaround to capture that audio?

Both methods can't capture notification, ringtone and alarm sounds.

You mentioned we can't capture these, but VoIP call audio was not specified.

Would love to know if it would be possible at all, or if android blocks it somehow.

Capturing in call audio works (atleast in Whatsapp calls).

package moe.chensi.tango

import android.annotation.SuppressLint
import android.app.ActivityThread
import android.content.Context
import android.media.AudioAttributes
import android.media.AudioFormat
import android.media.AudioManager
import android.media.audiopolicy.AudioMix
import android.media.audiopolicy.AudioMixingRule
import android.media.audiopolicy.AudioPolicy
import android.os.Build
import android.system.ErrnoException
import android.system.Os
import android.system.OsConstants
import org.joor.Reflect
import org.joor.ReflectException
import java.io.FileDescriptor
import java.io.IOException
import java.nio.ByteBuffer

class Test {
    @SuppressLint("StaticFieldLeak")
    private var _systemContext: Context? = null
    private val systemContext: Context
        @SuppressLint("DiscouragedPrivateApi", "PrivateApi") get() {
            if (_systemContext == null) {
                try {
                    // Hide warnings on XiaoMi devices
                    Reflect.onClass("android.content.res.ThemeManagerStub").set("sResource", null)
                } catch (_: ReflectException) {
                }

                _systemContext = ActivityThread.systemMain().systemContext
            }
            return _systemContext as Context
        }

    private fun writeFully(fd: FileDescriptor, from: ByteBuffer) {
        // ByteBuffer position is not updated as expected by Os.write() on old Android versions, so
        // count the remaining bytes manually.
        // See <https://github.com/Genymobile/scrcpy/issues/291>.
        var remaining = from.remaining()
        while (remaining > 0) {
            try {
                val w = Os.write(fd, from)
                remaining -= w
            } catch (e: ErrnoException) {
                if (e.errno != OsConstants.EINTR) {
                    throw IOException(e)
                }
            }
        }
    }

    @SuppressLint("MissingPermission")
    private fun captureAudio(fd: FileDescriptor) {
        if (Build.VERSION.SDK_INT < Build.VERSION_CODES.TIRAMISU) {
            return
        }

        val privileged = true
        val encoding = AudioFormat.ENCODING_PCM_16BIT
        val sampleRate = 16000;
        val channelMask = AudioFormat.CHANNEL_IN_MONO

        val rule =
            AudioMixingRule.Builder().setTargetMixRole(AudioMixingRule.MIX_ROLE_PLAYERS).addMixRule(
                AudioMixingRule.RULE_MATCH_ATTRIBUTE_USAGE,
                AudioAttributes.Builder().setUsage(AudioAttributes.USAGE_VOICE_COMMUNICATION).addMixRule(
// Having both mix rules mean Media and Call data is captured. Albeit at a lower quality.
                AudioMixingRule.RULE_MATCH_ATTRIBUTE_USAGE,
                AudioAttributes.Builder().setUsage(AudioAttributes.USAGE_MEDIA).build()
            ).allowPrivilegedPlaybackCapture(privileged).voiceCommunicationCaptureAllowed(true)
                .build()

        val mix = AudioMix.Builder(rule).setFormat(
            AudioFormat.Builder().setEncoding(encoding).setSampleRate(sampleRate)
                .setChannelMask(channelMask).build()
        ).setRouteFlags(AudioMix.ROUTE_FLAG_LOOP_BACK_RENDER).build()

        val policy = AudioPolicy.Builder(systemContext).addMix(mix).build()

        val audioManager = systemContext.getSystemService(AudioManager::class.java)

        val result = audioManager.registerAudioPolicy(policy)
        if (result != 0) {
            println("registerAudioPolicy failed: $result")
            return
        }

        val record = policy.createAudioRecordSink(mix)
        record.startRecording()

        val bufferSize = 1024
        val buffer = ByteBuffer.allocateDirect(bufferSize)
        while (true) {
            buffer.position(0)
            val read = record.read(buffer, bufferSize)
            buffer.limit(read)
            writeFully(fd, buffer)
        }
    }
}

[cptace]

Can someone please add a bit clearer instructions as to what needs to be done to get audio mirroring when using calls? I see the code there, but with no clear instructions, sorry not a dev myself so I couldn't get this to work.