Replace the current audio driver with Oboe on Android #86776

kus04e4ek · 2024-01-04T10:16:01Z

Closes: godot-proposals#2358

Sample project 5

TODO:

https://github.com/google/oboe/blob/main/docs/GettingStarted.md#obtaining-optimal-latency
~~Get the correct number of frames per callback and for output.~~ Went with the different idea.
~~Get the correct number of frames per callback for input.~~ Went with the different idea.
Handle audio devices switching.
Rewrite SCsub.
~~Properly update device list after connecting or disconnecting devices.~~ Went with the different idea.
Delete not needed Oboe files.

Known Issues:

OpenSL not working. Tested on the virtual machine.
Bad audio quality on machines with support only for in16_t format. ~~Couldn't reproduce it on my phone.~~ (My phone supports int32_t format, I just didn't know). Tested on the virtual machine.
~~Audio not working on the physical device.~~ An issue with my phone.
Microphone not working.
Connecting a new default device will be ignored if done outside the app.
Stream closes outside the app and doesn't reopen if other stream started by the other app.

See the comment for new TODO/TODO after merge list.

AThousandShips · 2024-01-04T10:19:48Z

You will also need to add the library to the list in COPYRIGHT.txt, instructions are available there

AThousandShips · 2024-01-06T11:55:16Z

platform/android/audio_driver_oboe.h

+	std::shared_ptr<InputErrorCallback> input_error_callback;
+
+	std::shared_ptr<oboe::AudioStreamDataCallback> audio_output_callback;
+	std::shared_ptr<oboe::AudioStreamDataCallback> audio_input_callback;


We don't use the standard library normally, see here, would need to be discussed here depending on the library using it, no other engine component uses this

Yeah, I know. Oboe uses shared_ptr and regular pointers, but methods accepting regular pointers are marked as deprecated, when I think about it again, it probably will be fine to use regular pointers, it just might create problems when closing the app (Through is this important?)

Checked when shred_ptr was added, it was in the google/oboe#821 to fix an issue with the data race, without shared_ptr the data race will still occur, so I think shared_ptr should stay

kus04e4ek · 2024-01-07T13:01:38Z

Audio not working on the physical device.

Seems to be a problem with my device. Opened an issue on oboe repository.

doc/classes/AudioServer.xml

kus04e4ek · 2024-03-09T15:37:22Z

platform/android/string_android.h

+		}
+
+		jclass cCharSequence = env->GetObjectClass(source);
+		jmethodID toString = env->GetMethodID(cCharSequence, "toString", "()Ljava/lang/String;");


Camel Case to be consistent with "jni_utils.cpp", e.g.

godot/platform/android/jni_utils.cpp

Lines 198 to 199 in 5d08c26

jmethodID getName = env->GetMethodID(cclass, "getName", "()Ljava/lang/String;");

jstring clsName = (jstring)env->CallObjectMethod(cls, getName);

But that code is pretty old, maybe that code should be more consistent and not the other way around.

kus04e4ek · 2024-03-09T15:42:36Z

platform/android/java/lib/src/org/godotengine/godot/Godot.kt

@@ -336,12 +337,14 @@ class Godot(private val context: Context) : SensorEventListener {
 		}

 		if (nativeLayerInitializeCompleted && !nativeLayerSetupCompleted) {
-			nativeLayerSetupCompleted = GodotLib.setup(commandLine.toTypedArray(), tts)
+			var audioManager: AudioManager = context.getSystemService(Context.AUDIO_SERVICE) as AudioManager
+			nativeLayerSetupCompleted = GodotLib.setup(commandLine.toTypedArray(), tts, audioManager)


context.getSystemService(Context.AUDIO_SERVICE) can be called on the C++ side, but the code would be harder to understand and longer, this is just here for readability. Will change if needed

kus04e4ek · 2024-03-09T16:41:52Z

I think the PR is ready. There are just a few issues that are worth addressing:

Not all audio device types are supported when using set_device. I think it's fine to leave it like this for now? I would love to do a follow-up PR latter, but I could always finish it in this PR, if that's important. I'm just not sure yet what would be the right approach for every type. Some types might start working only when specific conditions are met, so marked those as not working for now.
Not all audio device types are tested when using set_device. I just don't have them. So, if anyone wants to help me with that, then I would be happy to accept your help. Currently, tested are: built-in_earpiece (~~not working~~ working), built-in_speaker (working), bluetooth_sco (working), bluetooth_a2dp (working), built-in_microphone (working), fm_tuner (working? Tested only on the emulator, and there was just a high-pitch sound, I guess it should be like that?), telephony (not working), remote_submix (not working), echo_reference (didn't test, but it's not expected to work, it's reserved for system apps only).
SCO support is experimental? Right now, when setting the device on one of the streams, the device on the other would be set too. I did it that way, because turning on SCO for one stream would turn on SCO for the other. Also, when turning on SCO, the default device used by Android changes, so did it for the consistency on what the default device is. I don't think that any other Audio Driver changes devices for both streams on one set_device call, so I would appreciate some feedback. Also, setting SCO just doesn't work seemingly at random times? I have suspicions, but still not sure. See this comment about SCO.

Issues above should all be fine when not using set_device, I assume that the default device picked by Android always works, so maybe it would be better to disable it for now?

And some theoretically possible feature, that I could do in a follow-up, is very limited set_device for OpenSL.

Also, #88628 could be implemented for Android too. For the same reasons as the ones why it is implemented for macOS.

EDIT:
If you're interested in testing types, then all of them can be found here: https://developer.android.com/reference/android/media/AudioDeviceInfo#TYPE_AUX_LINE
Also, when testing SCO your device might require "MODIFY_AUDIO_SETTINGS" permission, this can be given at the export preset.

kus04e4ek · 2024-03-09T17:20:27Z

doc/classes/OS.xml

@@ -653,7 +653,7 @@
 			<param index="0" name="name" type="String" />
 			<description>
 				Requests permission from the OS for the given [param name]. Returns [code]true[/code] if the permission has been successfully granted.
-				[b]Note:[/b] This method is currently only implemented on Android, to specifically request permission for [code]"RECORD_AUDIO"[/code] by [code]AudioDriverOpenSL[/code].
+				[b]Note:[/b] This method is currently only implemented on Android and is used by [code]AudioDriverOboe[/code].


I think that the specific permissions and what they are for should be put somewhere else? It would feel better if they were in AudioServer docs somewhere, where the user would be expected to look first for any audio specific questions and issues. This PR also adds one more permission "MODIFY_AUDIO_SETTINGS" for SCO support on older devices, but I'm out of ideas where to put it.

kus04e4ek · 2024-03-09T17:23:48Z

platform/android/java/lib/src/org/godotengine/godot/utils/PermissionsUtil.java

@@ -63,6 +63,7 @@ public final class PermissionsUtil {
 	public static final int REQUEST_RECORD_AUDIO_PERMISSION = 1;
 	public static final int REQUEST_CAMERA_PERMISSION = 2;
 	public static final int REQUEST_VIBRATE_PERMISSION = 3;
+	public static final int REQUEST_MODIFY_AUDIO_SETTINGS_PERMISSION = 4;


I think that it's unused, but added for consistency

kus04e4ek · 2024-03-09T17:28:30Z

servers/audio_server.h

@@ -62,6 +62,7 @@ class AudioDriver {
 	unsigned int input_size = 0;

 	void audio_server_process(int p_frames, int32_t *p_buffer, bool p_update_mix_time = true);
+	void audio_server_init_channels_and_buffers();


Exposed it, since Oboe doesn't recommend to allocate memory in callbacks, which would happen through audio_server_process if device changed

kus04e4ek · 2024-03-11T06:07:46Z

thirdparty/README.md

+## oboe
+
+- Upstream: https://github.com/google/oboe
+- Version: 1.8.2 (16d72c89be9eb8a7d617a7be531a31dba3db74f1, 2024)


Put version 1.8.2, although it's not released yet, just to have this fix: google/oboe#1968. Without this fix, on some devices, the audio would be silent or with a lot of glitches since the code checking for it was never called

Changed version from "1.8.2" to "git", but the point still stands

loliaudio · 2024-03-14T14:27:54Z

EDIT: If you're interested in testing types, then all of them can be found here: https://developer.android.com/reference/android/media/AudioDeviceInfo#TYPE_AUX_LINE Also, when testing SCO your device might require "MODIFY_AUDIO_SETTINGS" permission, this can be given at the export preset.

I would like to help testing, but I have no knowledge abou C++.
I have some of the devices listed.
Is there a way of doing it?

kus04e4ek · 2024-03-15T04:47:57Z

I would like to help testing, but I have no knowledge abou C++.
I have some of the devices listed.
Is there a way of doing it?

Thanks for replying! Here are instructions on how to build this PR and how to export with it:
~~First of all you need to install git and SCons.~~
~~Then follow the instructions here (everything before "Providing launcher icons" is required) and here.~~
~~Clone this PR's github repository: git clone https://github.com/kus04e4ek/godot.git.~~
I build with these options: scons tests=false deprecated=no minizip=no vulkan=no modules_enabled_by_default=no module_text_server_fb_enabled=yes module_gdscript_enabled=yes module_freetype_enabled=yes module_minimp3_enabled=yes platform=android generate_apk=yes target=template_debug arch=<insert your architecture>
Follow the instructions here (everything before "Providing launcher icons" is required)
Godot 4.3 dev 4 is compatible with the export template, so you can just use it as an editor.
Setup the export preset for android by overriding Custom Template/Debug, choosing the correct architecture and turning on Modify Audio Settings and Record Audio permissions. (To get your architecture you can use AIDA64)
EDIT: Was incorrect in assuming that you would have to build Godot, you can just download artifacts by clicking Details next to 🔗 GHA / 🤖 Android / Template (target=template_release) (pull_request), then choose Upload artifact and download archive from the Artifact download URL link.

kus04e4ek · 2024-03-22T13:32:56Z

Right now, when setting the device on one of the streams, the device on the other would be set too. I did it that way, because turning on SCO for one stream would turn on SCO for the other. Also, when turning on SCO, the default device used by Android changes, so did it for the consistency on what the default device is.

I misinterpreted Android docs, now output and input streams do not depend on each other. This introduced more bugs when setting output device, while input stream uses SCO. Have no ideas what the issue is, but will work on it. Default device still changes, but I don't think that's a big issue.

Setting SCO just doesn't work seemingly at random times?

Fixed it, but input stream that uses SCO is very unstable due to #86428.

bestvcboy · 2024-06-06T06:47:22Z

May I ask which version this is likely to be supported on?

kus04e4ek · 2024-06-06T14:03:20Z

May I ask which version this is likely to be supported on?

Can't really guarantee anything here, I'm not part of Godot's team, I just work on it when I can (right now I don't have time). There are still some issues and I want to refactor the code (I'll mark it as a draft, kinda forgot to do it before😅). I am also more focused on #90013 for now, which would simplify the code here and I think it should be merged first. Not many users request Oboe and there's a lot of code to go through when reviewing, so this PR is probably not a priority for members. As far as I'm aware there's no consensus on using Oboe in the first place (I plan to write a comment on Oboe's proposal page that would go through what are the advantages and disadvantages of using Oboe) (should probably mention that no one on the team seems to be against using Oboe though). I think there's currently only one member of the audio team and that's reduz, who also has other stuff to do, so the reviewing process for this PR may take a long time. Because of all of this, I don't think it should be expected to be part of Godot any time soon if it will be.
EDIT: I also forgot to mention that I want to split this PR into non-Oboe specific ones, that could be applied to OpenSL.

AThousandShips added enhancement platform:android topic:audio labels Jan 4, 2024

AThousandShips added this to the 4.x milestone Jan 4, 2024

kus04e4ek force-pushed the master branch 2 times, most recently from 195df38 to 4c46c72 Compare January 4, 2024 11:04

kus04e4ek mentioned this pull request Jan 4, 2024

Add support for more output types in AudioDriver godotengine/godot-proposals#8797

Open

kus04e4ek force-pushed the master branch 2 times, most recently from c1d2804 to 57434a1 Compare January 5, 2024 09:32

kus04e4ek mentioned this pull request Jan 6, 2024

Bring the compatibility methods from AudioDriverWASAPI into AudioDriver godotengine/godot-proposals#8815

Open

kus04e4ek force-pushed the master branch from 57434a1 to 441e82c Compare January 6, 2024 11:52

AThousandShips reviewed Jan 6, 2024

View reviewed changes

kus04e4ek force-pushed the master branch 2 times, most recently from 7026dae to 482d1ff Compare January 9, 2024 11:26

kus04e4ek force-pushed the master branch 2 times, most recently from 9c51ce1 to a69115a Compare January 27, 2024 11:09

kus04e4ek force-pushed the master branch from a69115a to a2b147d Compare March 9, 2024 15:26

kus04e4ek commented Mar 9, 2024

View reviewed changes

kus04e4ek force-pushed the master branch 2 times, most recently from 82f0379 to 30de669 Compare March 9, 2024 15:56

kus04e4ek marked this pull request as ready for review March 9, 2024 16:42

kus04e4ek requested review from a team as code owners March 9, 2024 16:42

kus04e4ek commented Mar 9, 2024

View reviewed changes

kus04e4ek commented Mar 11, 2024

View reviewed changes

kus04e4ek force-pushed the master branch from 30de669 to daefdf5 Compare March 22, 2024 13:05

kus04e4ek added 2 commits March 22, 2024 21:21

Replace the OpenSL audio driver with Oboe on Android

ad17b8d

Delete not needed Oboe files

c088787

kus04e4ek force-pushed the master branch from daefdf5 to c088787 Compare March 22, 2024 13:21

kus04e4ek marked this pull request as draft June 6, 2024 14:03

Calinou mentioned this pull request Jun 21, 2024

Replace the current audio driver with Oboe on Android godotengine/godot-proposals#2358

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Replace the current audio driver with Oboe on Android #86776

Replace the current audio driver with Oboe on Android #86776

kus04e4ek commented Jan 4, 2024 •

edited

Loading

AThousandShips commented Jan 4, 2024

AThousandShips Jan 6, 2024

kus04e4ek Jan 6, 2024

This comment was marked as outdated.

kus04e4ek Mar 9, 2024

kus04e4ek commented Jan 7, 2024

kus04e4ek Mar 9, 2024 •

edited

Loading

kus04e4ek Mar 9, 2024

kus04e4ek commented Mar 9, 2024 •

edited

Loading

kus04e4ek Mar 9, 2024

kus04e4ek Mar 9, 2024

kus04e4ek Mar 9, 2024

kus04e4ek Mar 11, 2024

kus04e4ek Mar 22, 2024

loliaudio commented Mar 14, 2024

kus04e4ek commented Mar 15, 2024 •

edited

Loading

kus04e4ek commented Mar 22, 2024

bestvcboy commented Jun 6, 2024

kus04e4ek commented Jun 6, 2024 •

edited

Loading

	jmethodID getName = env->GetMethodID(cclass, "getName", "()Ljava/lang/String;");
	jstring clsName = (jstring)env->CallObjectMethod(cls, getName);

Replace the current audio driver with Oboe on Android #86776

Are you sure you want to change the base?

Replace the current audio driver with Oboe on Android #86776

Conversation

kus04e4ek commented Jan 4, 2024 • edited Loading

AThousandShips commented Jan 4, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

This comment was marked as outdated.

Choose a reason for hiding this comment

kus04e4ek commented Jan 7, 2024

kus04e4ek Mar 9, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kus04e4ek commented Mar 9, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

loliaudio commented Mar 14, 2024

kus04e4ek commented Mar 15, 2024 • edited Loading

kus04e4ek commented Mar 22, 2024

bestvcboy commented Jun 6, 2024

kus04e4ek commented Jun 6, 2024 • edited Loading

kus04e4ek commented Jan 4, 2024 •

edited

Loading

kus04e4ek Mar 9, 2024 •

edited

Loading

kus04e4ek commented Mar 9, 2024 •

edited

Loading

kus04e4ek commented Mar 15, 2024 •

edited

Loading

kus04e4ek commented Jun 6, 2024 •

edited

Loading