Android support

[corrados]

This issue has been migrated from Sourceforge. Created: 2015-12-13
Creator: Volker Fischer

Add support for the Android operating system. The software can already be compiled for Android but the audio interface does not work right now. So the first step is to finish the audio interface implementation. As a second step, all GUI functionality must be checked that it is possible to use it with a touch screen only.

[corrados]

Some progress reported here: https://sourceforge.net/p/llcon/discussion/533517/thread/cfd165e0/#452b/6e3b

[corrados]

See here: https://www.facebook.com/groups/507047599870191/permalink/521878088387142/

[j-santander]

Is there any instructions on how to build this? or any specific branch/forked repository I could refer to? From the previous links there seems to be some implementation working to some extent. I might be able to give a hand, at least testing
Thanks

[corrados]

The code on master is from Simon. But it does not work correctly. Recently, I have started a branch where I do some testing. But also, nothing working yet.

[j-santander]

Ok, thanks. I got it to compile and started testing.

There's an mistake on android/sound.cpp: on CSound::onAudioRead() when converting from incoming float buffer to int16_t:

                pSound->vecsTmpAudioSndCrdStereo[frmNum * oboeStream->getChannelCount() + channelNum] =
                    (short) floatData[frmNum * oboeStream->getChannelCount() + channelNum] * _MAXSHORT;

should be

                pSound->vecsTmpAudioSndCrdStereo[frmNum * oboeStream->getChannelCount() + channelNum] =
                    (int16_t)(floatData[frmNum * oboeStream->getChannelCount() + channelNum] * _MAXSHORT);

otherwise the precedence of the casting operator will make the input always 0.

Changing that if I write to a file the content pSound->vecsTmpAudioSndCrdStereo vector I got a correct signal.

To continue progressing I might need to understand better how the whole things work (I'm not at all familiar with neither your code or even the oboe library). A few doubts:

• Is it assumed that the sound card is always stereo? It seems so, but the oboe library seems to be configured in Mono. Changing that I seemed to started receiving a more-or-less-ok signal on another client. What other assumptions should be made for the signal coming from CSound -> CClient (sampling rate, frame size....)?
• You got oboe setup for float samples. Is there any reason to not use int16_t samples? (we're doing the conversion by hand).
• I'm not sure how it works on the playing side. What I believe I see is that at CClient::ProcessAudioDataIntern after the recording data is processed, the same vector used for the recording data is filled with the playing data received from the network. However, my expectation is that I would be receiving an independent call to CSound::onAudioRead() for the playing channel, which might even indicate that the number of frames available for writing is not the same as the number of frames that we've written (however there seem to be an oboe builder call: setFramesPerCallback() that might help in synchronizing).
• What are the expectations for the CClient -> CSound data?
• How the expected buffer sizes work? (both for playing and recording)

In this last case, I'd go probably for a temporary circular buffer where the onAudioRead() processing of input data will write the recording data left in the buffer and the call of onAudioRead() for the output data will read. It probably would need to be mutex protected (unless we can be sure that the oboe callbacks are always run on the same thread)

[corrados]

There's an mistake on android/sound.cpp: on CSound::onAudioRead() when converting from incoming float buffer to int16_t

I'll merge from a branch soon, where the int 16 type is replaced by float.

Is it assumed that the sound card is always stereo?

Yes, the input/output audio vector for the audio interfaces is assumed to be always stereo. If you have only a mono stream available, just write the same audio data in both stereo channels.

On my branch https://github.com/corrados/jamulus/tree/android_testing I tried to use the full duplex oboe example class. But it is not yet finished. There is function onBothStreamsReady which should have the same size for input and output (which is what Jamulus expects).

[j-santander]

I've got some working code at j-santander@a3558aa
(Unfortunately I was mistakenly working from the master code, not from the android_testing branch.. and only when I was ready to submit I realized of my mistake).

In my tests I can hear other clients and they can hear me. Not sure of the overall quality of sound.

Key modification was to setup the callback only on the input stream and do explicit write to the output stream after processing the incoming data.

I'll try to have a look at the code in the android_testing branch and incorporate it.

Let me know what you think or how should I proceed.

Cheers

[corrados]

My code on the android_testing branch is not working. I was just experimenting a bit. You do not need to use any of that code. If you have something working, then you could create a pull request. I can then test it on my Android phone. But be aware that some minutes ago I applied a big change to the code on master.

[j-santander]

Created PR #661.

Thanks

[ann0see]

How’s the quality? I assume it would be best to try it with a LAN adapter

[j-santander]

How’s the quality? I assume it would be best to try it with a LAN adapter

I might be able to test with more people today.

[dzpex]

i'm interested to test the android code but i do not have experience to compile qt5 project for andoid can you help me?

[corrados]

@j-santander I tested your code and it crashed with "AudioStreamRecord: maybeConvertDeviceData() conversion size 256 too large for buffer 128". I had the same error message on my Android testing branch. This is a known bug in the Oboe library. We have to use a newer version. Just to let you know: I'll now update the Oboe Git submodule to the newest Oboe code now to avoid this issue.

[corrados]

I just updated the Oboe library version but unfortunately, it still crashes, now with a different error message: " InputEventReceiver: Attempted to finish an input event but the input event receiver has already been disposed."

[j-santander]

Oh, I'll test it more on my side. I didn't experience neither of the errors. I'm on Android 11, not sure if it has anything to do.

[j-santander]

i'm interested to test the android code but i do not have experience to compile qt5 project for andoid can you help me?

Well, I didn't even have experience on Qt. I used the Qt installer (https://www.qt.io/download-qt-installer) and made sure the Android support was marked. Using the Qt Creator, I had to set up paths to SDKs and NDKs and select the target for android.

[dzpex]

i installed all the requirements for the QT and i'm able to set the Andorid Target. But the compiler return error.....oboe/Oboe.h not found any idea?

[j-santander]

i installed all the requirements for the QT and i'm able to set the Andorid Target. But the compiler return error.....oboe/Oboe.h not found any idea?

The oboe library is stored as a git submodule, I believe you need to do:

git submodule update --init

[dzpex]

i updated the submodule but the error is the same i try to copy oboe include folder into jamulus main path or android folder but nothin

[j-santander]

I just updated the Oboe library version but unfortunately, it still crashes, now with a different error message: " InputEventReceiver: Attempted to finish an input event but the input event receiver has already been disposed."

I've been trying with different devices, and although there are problems with the sound quality, I haven't experienced any crashes like the ones you report. What Android version are you in? what terminal?

What I've tried so far:

• Google Pixel 3a with Android 11 (arm64-v8a)
• Google Pixel 3XL with Android 10 (arm64-v8a)
• BQ Aquaris X5 Plus with Android 7.1 (armeabi-v7a)
• Android Emulator on Android 7.1 (x86)

Regarding quality:

• In the BQ and the Emulator it seems I cannot play. Either because they are less powerful hardware, or the Android version. Checking into that.
• Overall architecture: What I have so far is we have the oboe stream callback on the input the stream, this calls into the CClient callback that does its stuff and replaces the input samples with the output samples in the same vector we're using as argument. After the CClient callback returns we do a non-blocking write on oboe's output stream. The concern here is that we're doing all sort of stuff that we shouldn't be doing in the oboe's callback thread (which is then the CClient callback thread that I guess allocates memory, do network operations, etc...).

[j-santander]

i updated the submodule but the error is the same i try to copy oboe include folder into jamulus main path or android folder but nothin

Ok, let's recap. What I did was:

• Install the Qt, including the Android support from the Qt installer.
• Configure the Android settings in the Qt Creator (Tools -> Options -> Devices -> Android). Solve all warnings/errors in that dialog. Tick on the "Automatically create kits for Android Toolk chains").
• Make sure the Android Kit is detected on Tools -> Options -> Kits -> Kits (I see Android Qt 5.15.1 Clang Multi-Abi). It should also show in the Qt Versions tab (Qt 5.15.1 for Android).
• Open the Jamulus.pro
• In the Projects tab -> Build & RUn It should appear the Android Qt, might be disabled. Enabled it if it is not enabled.
  Then you can connect your device and build or run the project.

Regarding the actual source. oboe is a git submodule which is located at libs/oboe. Make sure it has been checkout properly.

Hope this helps

[corrados]

I've been trying with different devices, and although there are problems with the sound quality

Have you been able to jam with other on any of these devices yet? Could you measure the overall latency you got?

What Android version are you in? what terminal?

I have a special setup: LineagesOS 16.0 (Android 9) on a Samsung Galaxy S5 (SM-G901F)

The concern here is that we're doing all sort of stuff that we shouldn't be doing in the oboe's callback thread

We actually do this for all OSs. It is not the preferred way of doing it but it works fine.

[j-santander]

I've been trying with different devices, and although there are problems with the sound quality

Have you been able to jam with other on any of these devices yet? Could you measure the overall latency you got?

I did try briefly, but I'm in the middle of trying to deploy this to my choir and the focus of sessions is to get the people onboarded.

Regarding the latency. Right now, I'm connected to a local server in my LAN:

From the Windows Client (desktop - ethernet connected) the "Configuration" reports Overall Delay 39 ms. (Ping Time 1 ms).

From my Google Pixel 3a, over WiFi, Overall Delay varies quite a lot from 55 to 80 and 90 ms. I guess it is due to the great variation on Ping Time from 2 to 33 ms.

Not sure if that's the measurement you're after.

What Android version are you in? what terminal?

I have a special setup: LineagesOS 16.0 (Android 9) on a Samsung Galaxy S5 (SM-G901F)

ok, I'm not sure if I could find a way of emulating that.

The concern here is that we're doing all sort of stuff that we shouldn't be doing in the oboe's callback thread

We actually do this for all OSs. It is not the preferred way of doing it but it works fine.

Ok, I look into that. I might need to implement a circular buffer because in some of the models it seems that I cannot write the whole batch of frames of output frames in one go.

I'll keep testing and see how far I can go.

[corrados]

I did try briefly, but I'm in the middle of trying to deploy this to my choir and the focus of sessions is to get the people onboarded.

So you intention is to bring your choir to jamulus using their Android phones? Very interesting. Please post your results here since I guess a lot of other choir members would love to use their phones for Jamulus, too.

ok, I'm not sure if I could find a way of emulating that.

No problem if not. My device is very old and I do not expect a good latency coming out of it. So just focus on the newer phones, that's fine.

[ann0see]

I think there was an attempt for iOS, but the repo disappeared. See iOS version thread on sourceforge.

[j-santander]

This all sounds very encouraging for the prospect of mobile builds - and being able to support choirs
and similar scenarios as @j-santander described!

I would love to give an Android build a test - is master the best branch to attempt a build from currently?

Sorry folks, I believe my disappearance requires a bit of explanation. As I described above, what I did over the existing code for Android that Volker and Simon had in place was just a bit of bug fixing and small adjustments. However, as I was thinking about further involvement I realized that this was uncomfortably close to my day job (not really a direct overlap, but there might be technologies and techniques in common), so before involving myself any further, I decided to clear it up with my management. They sent a request to legal department and we're still waiting for an answer.

Regarding building. All my modifications were accepted to master, but I haven't tried to build it recently. Not sure if recent changes might have broken the build.

[danryu]

Thanks @j-santander for the update.
FWIW - I don't know your exact situation, but from my experience there should be zero conflict with working on open-source projects, unless you are using company time to do so. Most progressive companies welcome their employees working on diverse projects in their own time, as it inevitably leads to an enrichment of experience and ability within the workforce.

[ann0see]

How does the UI look like on Android? I compiled Jamulus for iOS and it doesn't show anything for me. I assume, we might need QML for both operating systems?

[corrados]

The UI on Android looks the same as on the PC. No need for QML.

[ann0see]

Ok.
Also, there was no need to change the code, I assume?
On iOS is that I don't see anything but a black screen, so it's probably an iOS only problem.

I hope, my problem can be fixed in some way without having to rewrite the UI for iOS

[corrados]

Qt should give you the same GUI on all platforms.

[nefarius2001]

Hi,

• Is anyone currently working on This?
• had anyone experienced a „known to work“ configuration?

I got the code compiled and the GUI running, but not Audio transmission. Will try a different device as next step...

[nefarius2001]

I have the audio part well working on my tablet (X10605). Some UI issues are still open (mainly using the "back" button)

I am thinking about a source for "nightly" APKs, for those eager to have at least anything.
@corrados Any plans or existing infrastructure to provide "official" builds, except the released installers?

[ann0see]

I am thinking about a source for "nightly" APKs, for those eager to have at least anything.

Any plans or existing infrastructure to provide "official" builds, except the released installers?

GH actions should be able to build it automatically. Once #828 is merged and tested, it should be quite easy to add an action which automatically builds apk files.

[corrados]

That would be creat. Since the Action searches for a "r*" tag, I could easily create beta versions by simply tagging the Git repo with something like r3.6.2beta1.

[ann0see]

Yes, that’s possible but will also move the latest tag.

I tried to setup an Android build environment on a clean Debian 10 VM but didn’t succeed unfortunately.
If anybody succeeds to create a shell script which installs the dependencies and starts the build, it‘s easy to upload the apk. You will then just need to move the .apk file to /deploy in the Shell script and edit the GitHub Action.

[nefarius2001]

I tried to setup an Android build environment on a clean Debian 10 VM but didn’t succeed unfortunately.

I tried with an Ubuntu, but also no success yet.
Weird, setting it up on windows with QtDesigner was easy, now Ubuntu is quite hard...

[nefarius2001]

Until an official build is available, the APK of my pull request can be found here:
http://116.203.80.188:8000/jamulus-024ec6a8402048c13b03a31a13504feed1de3635.apk

[ann0see]

I've now created a project board on this topic, since I think it's a larger one. It would be great if we could close this issue and document needed changes in separate issues and in https://github.com/jamulussoftware/jamulus/projects/6

[ann0see]

Just a quick reminder here: Thanks to nefarius we have working .apk android builds here: https://github.com/jamulussoftware/jamulus/releases/tag/r3_7_0beta3

Once 3.7.0 is out, GitHub actions will automatically build apk files for every new release.

[hoffie]

Once Jamulus 3.7.0 is out, there will be official apk files for every new release (hopefully).

I'm wondering if we want to bless them as "official" already or rather mark them as experimental for now?

I gave them a quick try and while the UI worked, I did not manage to get usable sound out of it on a Moto G4 Play (distortion in short frequency, not the typical Wifi-related unreliability). If this is supposed to work, I should probably open a bug report about it. Until now I assumed that this was some kind of proof of concept...? :)

[ann0see]

Yes. You're right. I just edited my post. Some people have reported "usable" results: #1142

[gilgongo]

Moving this to discussion in the interests of keeping issues to actionable work items.