Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Forward audio #3757

Merged
merged 118 commits into from
Mar 10, 2023
Merged

Forward audio #3757

merged 118 commits into from
Mar 10, 2023

Conversation

rom1v
Copy link
Collaborator

@rom1v rom1v commented Feb 26, 2023

Last month, @yume-chan submitted a working PoC to capture audio on Android. Big thanks for this discovery!

Since then I have worked to properly integrate audio support in scrcpy (capture, encode, transmit, record, decode, resample, play…), and my branch reached a working state (at least on my machine) 🎉

By default, audio is enabled (if supported):

  • On Android >= 12, it works out-of-the-box;
  • On Android 11, the device screen must be unlocked (because of a dirty workaround to make the system think the app is foreground, again thank you to @yume-chan for the tip);
  • On Android <= 10, it does not work, audio is automatically disabled.

If audio capture fails, then mirroring continues with video only (since audio is enabled by default, it is not acceptable to make scrcpy fail if audio is not available), unless --require-audio is set.

To disable audio:

scrcpy --no-audio

If audio is enabled, it is also recorded when recording is enabled:

scrcpy --record=file.mp4
scrcpy --record=file.mkv

To play audio in real-time, the audio player implements drift compensation (it attempts to keep audio sample buffering to a certain level, quite low to get low-latency, but not too low to avoid underflow). It is experimental (I just tinkered with this thing today) (I've made a lot of improvements since the initial MR, and it should work pretty fine), parameters and behavior are hardcoded, some configuration variables might need to be exposed in the future (or not).

If you record, the timestamps are computed on the device, so they are not impacted by any jitter/underflow (this is the same as for the video), so the recording is always clean (if you use --record of course, not if you capture your audio output on the computer).

Like the video, it works over TCP/IP (typically Wifi).

The following options have been renamed:

  • --codec -> --video-codec
  • --bit-rate -> --video-bitrate
  • --codec-options -> --video-codec-options
  • --encoder -> --video-encoder

And similar options for audio have been added:

  • --audio-codec
  • --audio-bit-rate
  • --audio-codec-options
  • --audio-encoder

Three audio formats are supported:

  • Opus: scrcpy --audio-codec=opus
  • AAC: scrcpy --audio-codece=aac
  • Raw (uncompressed) PCM 16-bit LE: scrcpy --audio-codec=raw (more bandwidth, typically just for testing)

It is possible to list the encoders available on the device (video and audio):

scrcpy --list-encoders

If you need a specific encoder, use --audio-encoder:

scrcpy --audio-codec=aac --audio-encoder='OMX.google.aac.encoder'

When recording, raw audio is not possible (it automatically switches to OPUS). In the future, I might add audio encoding on the client-side specifically for recording, but I don't plan to make it for this audio feature.

I also implemented --audio-buffer= similar to existing --display-buffer= and --v4l2-buffer=.

This branch is in working state.

Since I have refactored A LOT of code, I may have broken things (please tell me!).

Please test, review (good luck, that's a big PR 😉), and report any problem :)

Fixes #14


old

Here is a release build for the current MR for Windows (audio.115):

Here is a release build for the current MR for Windows (audio.145):

@rom1v rom1v mentioned this pull request Feb 26, 2023
@yume-chan yume-chan mentioned this pull request Feb 27, 2023
@rom1v rom1v force-pushed the audio branch 4 times, most recently from 6e948fe to 2b2cf0a Compare February 27, 2023 20:42
@parkerlreed
Copy link

Are the BUILD instructions outdated?

(deck@steamdeck build)$ git clone https://github.com/Genymobile/scrcpy
Cloning into 'scrcpy'...
remote: Enumerating objects: 23178, done.
remote: Counting objects: 100% (3736/3736), done.
remote: Compressing objects: 100% (921/921), done.
remote: Total 23178 (delta 2443), reused 3435 (delta 2189), pack-reused 19442
Receiving objects: 100% (23178/23178), 4.74 MiB | 9.34 MiB/s, done.
Resolving deltas: 100% (14698/14698), done.
(deck@steamdeck build)$ cd scrcpy/
(deck@steamdeck scrcpy)$ git checkout audio
branch 'audio' set up to track 'origin/audio'.
Switched to a new branch 'audio'
(deck@steamdeck scrcpy)$ meson setup x --buildtype=release --strip -Db_lto=true
The Meson build system
Version: 1.0.1
Source dir: /home/deck/build/scrcpy
Build dir: /home/deck/build/scrcpy/x
Build type: native build
Project name: scrcpy
Project version: 1.25
C compiler for the host machine: cc (gcc 12.2.0 "cc (GCC) 12.2.0")
C linker for the host machine: cc ld.bfd 2.39.0
Host machine cpu family: x86_64
Host machine cpu: x86_64
Found pkg-config: /usr/bin/pkg-config (1.8.0)
Run-time dependency libavformat found: YES 59.27.100
Run-time dependency libavcodec found: YES 59.37.100
Run-time dependency libavutil found: YES 57.28.100
Run-time dependency libswresample found: YES 4.7.100
Run-time dependency sdl2 found: YES 2.24.1
Run-time dependency libavdevice found: YES 59.7.100
Run-time dependency libusb-1.0 found: YES 1.0.26
Checking for function "strdup" : YES 
Checking for function "asprintf" : YES 
Checking for function "vasprintf" : YES 
Checking for function "nrand48" : YES 
Checking for function "jrand48" : YES 
Header "sys/socket.h" has symbol "SOCK_CLOEXEC" : YES 
Configuring config.h using configuration

app/meson.build:226:0: ERROR: File src/util/average.c does not exist.

A full log can be found at /home/deck/build/scrcpy/x/meson-logs/meson-log.txt

@parkerlreed
Copy link

parkerlreed commented Feb 27, 2023

Seems to be nowhere in the sources? I don't even see a src folder

(deck@steamdeck scrcpy)$ find . -name "average.c"
(deck@steamdeck scrcpy)$ 

EDIT: Oh the folder is there just not that file in app/src/util

@rom1v
Copy link
Collaborator Author

rom1v commented Feb 27, 2023

app/meson.build:226:0: ERROR: File src/util/average.c does not exist.

Oops, it was not committed, sorry. Fixed.

@parkerlreed
Copy link

ninja: Entering directory `x'
[48/59] Generating server/scrcpy-server with a custom command

FAILURE: Build failed with an exception.

* What went wrong:
Could not open settings generic class cache for settings file '/home/deck/build/scrcpy/settings.gradle' (/home/deck/.gradle/caches/7.5/scripts/4g6ucengw406w5p3xne7pf3a4).
> BUG! exception in phase 'semantic analysis' in source unit '_BuildScript_' Unsupported class file major version 63

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 752ms
[57/59] Linking target app/scrcpy
FAILED: server/scrcpy-server 
/home/deck/build/scrcpy/server/./scripts/build-wrapper.sh /home/deck/build/scrcpy/server server/scrcpy-server release
[59/59] Linking target app/scrcpy
ninja: build stopped: subcommand failed.

@parkerlreed
Copy link

Aha java is too new (jdk 19). Trying an older version

@parkerlreed
Copy link

It's alive!

Note 10 Plus on Android 12. Working well!

@parkerlreed
Copy link

test.mp4

@reitowo
Copy link

reitowo commented Mar 3, 2023

Looking forward to this PR!

rom1v added a commit that referenced this pull request Mar 3, 2023
Rename VideoStreamer to Streamer, and extract a Codec interface which
will also support audio codecs.

PR #3757 <#3757>
rom1v added a commit that referenced this pull request Mar 3, 2023
This will allow to use "codec" for the Codec type.

PR #3757 <#3757>
rom1v added a commit that referenced this pull request Mar 3, 2023
The provided encoder name depends on the selected codec. Improve the
error message and the suggestions.

PR #3757 <#3757>
rom1v added a commit that referenced this pull request Mar 3, 2023
Since scrcpy-server is not an Android application (it's a java
executable), it has no Context.

Some features will require a Context instance to get the package name
and the UID. Add a FakeContext for this purpose.

PR #3757 <#3757>

Co-authored-by: Romain Vimont <[email protected]>
Signed-off-by: Romain Vimont <[email protected]>
rom1v added a commit that referenced this pull request Mar 3, 2023
FakeContext already provides an AttributeSource instance.

PR #3757 <#3757>

Co-authored-by: Simon Chan <[email protected]>
rom1v added a commit that referenced this pull request Mar 3, 2023
Remove duplicated constant.

PR #3757 <#3757>
rom1v added a commit that referenced this pull request Mar 3, 2023
Remove USER_ID from ServiceManager, and replace it by a constant in
FakeContext.

This is the same as android.os.Process.ROOT_UID, but this constant has
been introduced in API 29.

PR #3757 <#3757>
rom1v added a commit that referenced this pull request Mar 3, 2023
rom1v added a commit that referenced this pull request Mar 3, 2023
This will expose the correct package name and UID to the application
context.

PR #3757 <#3757>
rom1v added a commit that referenced this pull request Mar 3, 2023
Audio will be enabled by default (when supported). Add an option to
disable it.

PR #3757 <#3757>

Co-authored-by: Romain Vimont <[email protected]>
Signed-off-by: Romain Vimont <[email protected]>
rom1v added a commit that referenced this pull request Mar 3, 2023
When audio is enabled, open a new socket to send the audio stream from
the device to the client.

PR #3757 <#3757>

Co-authored-by: Romain Vimont <[email protected]>
Signed-off-by: Romain Vimont <[email protected]>
rom1v added a commit that referenced this pull request Mar 3, 2023
Create an AudioRecorder to capture the audio source REMOTE_SUBMIX.

For now, the captured packets are just logged into the console.

PR #3757 <#3757>

Co-authored-by: Romain Vimont <[email protected]>
Signed-off-by: Romain Vimont <[email protected]>
rom1v added 2 commits March 10, 2023 22:22
The audio demuxer thread is the one filling the audio buffer read by the
SDL audio thread. It is time critical to avoid buffer underflow.
Recording is background task, writing the packets to a file is not
urgent.
@rom1v rom1v merged commit 408f458 into dev Mar 10, 2023
@rom1v
Copy link
Collaborator Author

rom1v commented Mar 10, 2023

Let's merge this.

I will continue some work on the dev branch, but I can't keep a working branch with 118 commits to rebase. Basically, it works 😉

@rp1231
Copy link

rp1231 commented Mar 11, 2023

Let's merge this.

I will continue some work on the dev branch, but I can't keep a working branch with 118 commits to rebase. Basically, it works 😉

Is the mod+r Android 11 bug fixed in this commit or will it be fixed in the next one?

@rom1v
Copy link
Collaborator Author

rom1v commented Mar 11, 2023

@rp1231 I reference previous commits since the github thread is unusable: #3757 (comment) #3757 (comment)

But I've noticed a small problem in this version which didn't exist in the previous version,
When watching a youtube video, If I click on the fullscreen button, the audio breaks and distorts while the screen is rotating.

Is the mod+r Android 11 bug fixed in this commit or will it be fixed in the next one?

It is not fixed.

Since the glitch is not recorded and is reduced with more buffering, here is my hypothesis.

When buffer underflow occurs, the player inserts silence samples (it does not have any valid samples yet). Typically, buffer underflow occurs because some packets are late (due to jitter), so they are expected to arrive, but later. Therefore, inserting silence will delay the playback of future packets, and therefore I immediately shift the buffering level average by the number of silence samples inserted. If I didn't do that, then the overbuffering due to the inserted silence will be detected slowly (because the average buffering is smoothed), so it will first accelerate the playback (because underflow), then decelerate (because overbuffering will finally be detected).

But there might be another cause of buffer underflow, which is currently not taken into account at all: when the device just misses to capture samples for a period of time. This is less likely than jitter, but I suppose that this is what happens when you rotate your device during fullscreen youtube playback: the device "freezes" and don't capture audio for a short period.

In that case, considering that the missing samples will arrive (and count them for the buffering level) is a wrong prediction, which will be corrected "slowly" by the smoothed (observed) buffering level average. During this time, more underflow will occur.

One possible fix could be to use the input packet PTS to detect that samples have not been captured by the device, but another issue makes it impossible: the default OPUS encoder writes its own timestamps in the encoded packets it produces, in a way that it matches the number of actual samples (it does not use the input timestamps provided by the audio capture), so missing packets could not be detected.

The AAC encoder does not do that though (it forwards the input timestamp), so I build a specific version with more logs, to confirm my hypothesis.

EDIT: do not take this binary, see the next comment.

Please run it with:

scrcpy --audio-codec=aac -Vdebug
diff
diff --git a/app/src/audio_player.c b/app/src/audio_player.c
index 41e7565f2..ed0934691 100644
--- a/app/src/audio_player.c
+++ b/app/src/audio_player.c
@@ -144,6 +144,17 @@ sc_audio_player_frame_sink_push(struct sc_frame_sink *sink,
                                 const AVFrame *frame) {
     struct sc_audio_player *ap = DOWNCAST(sink);
 
+    assert(frame->pts != AV_NOPTS_VALUE);
+    if (ap->expected_next_pts != AV_NOPTS_VALUE) {
+        int64_t diff = ap->expected_next_pts - frame->pts;
+        if (diff) {
+            LOGI("==== unexpected PTS diff = %" PRIi64, diff);
+        }
+    }
+    int64_t frame_duration_us = INT64_C(1000000) * frame->nb_samples
+                              / ap->sample_rate;
+    ap->expected_next_pts = frame->pts + frame_duration_us;
+
     SwrContext *swr_ctx = ap->swr_ctx;
 
     int64_t swr_delay = swr_get_delay(swr_ctx, ap->sample_rate);
@@ -414,6 +425,7 @@ sc_audio_player_frame_sink_open(struct sc_frame_sink *sink,
     ap->received = false;
     ap->played = false;
     ap->underflow = 0;
+    ap->expected_next_pts = AV_NOPTS_VALUE;
 
     // The thread calling open() is the thread calling push(), which fills the
     // audio buffer consumed by the SDL audio thread.
diff --git a/app/src/audio_player.h b/app/src/audio_player.h
index 3227f2fef..224abe62d 100644
--- a/app/src/audio_player.h
+++ b/app/src/audio_player.h
@@ -35,6 +35,9 @@ struct sc_audio_player {
     // thread)
     uint32_t previous_can_write;
 
+    // Useful to detect missing packets (e.g. due to a device freeze)
+    int64_t expected_next_pts;
+
     // Resampler (only used from the receiver thread)
     struct SwrContext *swr_ctx;
 

@rom1v
Copy link
Collaborator Author

rom1v commented Mar 11, 2023

The version I built in the previous comment was on a branch with a major bug for estimation. Here is a new binary:

Also, please run with -Vverbose instead of -Vdebug as I suggested:

scrcpy --audio-codec=aac -Vverbose

@rp1231
Copy link

rp1231 commented Mar 11, 2023

I'm getting this error:

scrcpy 1.25 <https://github.com/Genymobile/scrcpy>
scrcpy: unknown option -- Vverbose

even on the previous build you posted, I got the error unknown option --Vdebug

@rp1231
Copy link

rp1231 commented Mar 11, 2023

Also wanted to mention that this problem didn't occur in the very first build you had posted (scrcpy-win64-PR3757-1)
Incase that provides some clues.

@rom1v
Copy link
Collaborator Author

rom1v commented Mar 11, 2023

Sorry, it is -Vverbose (single dash).

that this problem didn't occur in the very first build you had posted

Ok, but that could a side effect that in this first build buffering might be higher than expected.

@rom1v
Copy link
Collaborator Author

rom1v commented Mar 11, 2023

I could reproduce on a tablet:

INFO: ==== unexpected PTS diff = 0 samples
INFO: New texture: 1200x1920
INFO: ==== unexpected PTS diff = 0 samples
INFO: ==== unexpected PTS diff = 0 samples
DEBUG: [Audio] Buffer underflow, inserting silence: 46 samples
DEBUG: [Audio] Buffer underflow, inserting silence: 240 samples
DEBUG: [Audio] Buffer underflow, inserting silence: 240 samples
DEBUG: [Audio] Buffer underflow, inserting silence: 240 samples
DEBUG: [Audio] Buffer underflow, inserting silence: 240 samples
DEBUG: [Audio] Buffer underflow, inserting silence: 240 samples
DEBUG: [Audio] Buffer underflow, inserting silence: 240 samples
DEBUG: [Audio] Buffer underflow, inserting silence: 240 samples
INFO: ==== unexpected PTS diff = -313 samples
INFO: ==== unexpected PTS diff = -554 samples
DEBUG: [Audio] Buffer underflow, inserting silence: 239 samples
DEBUG: [Audio] Buffer underflow, inserting silence: 240 samples
DEBUG: [Audio] Buffer underflow, inserting silence: 240 samples
INFO: ==== unexpected PTS diff = -158 samples
INFO: ==== unexpected PTS diff = -507 samples
INFO: ==== unexpected PTS diff = -116 samples
INFO: ==== unexpected PTS diff = -29 samples
INFO: ==== unexpected PTS diff = 0 samples
INFO: ==== unexpected PTS diff = 0 samples
INFO: ==== unexpected PTS diff = 0 samples

When the tablet is rotated, everything freezes, so captured packets are not sent immediately and the audio is not captured for a few milliseconds.

On the client side, this causes buffer underflow (and glitches) as expected. The only way not to cause underflow is to increase the buffering (so that when the freeze happens, there are still enough samples in the buffer to absorb it).

@rp1231
Copy link

rp1231 commented Mar 12, 2023

@rom1v
I've noticed that the mod+r shortcut stops working on android 11 only when the youtube player is active?
Pretty strange but maybe it leads to some clues?

@rom1v
Copy link
Collaborator Author

rom1v commented Mar 12, 2023

I've noticed that the mod+r shortcut stops working on android 11 only when the youtube player is active?

mod+r requests the foreground app to switch orientation. It may refuse (typically if it does not support the other orientation, but not only).

@rp1231
Copy link

rp1231 commented Mar 12, 2023

I've noticed that the mod+r shortcut stops working on android 11 only when the youtube player is active?

mod+r requests the foreground app to switch orientation. It may refuse (typically if it does not support the other orientation, but not only).

Yep but the thing is that this issue didn't occur before on the same android 11 phone in v1.25.
Also this issue doesn't happen on android 13.

@rom1v
Copy link
Collaborator Author

rom1v commented Mar 12, 2023

this issue didn't occur before on the same android 11 phone in v1.25.

If you retry now with v1.25, does it work? I think not. It's probably due to a youtube/os update.

@rp1231
Copy link

rp1231 commented Mar 12, 2023

this issue didn't occur before on the same android 11 phone in v1.25.

If you retry now with v1.25, does it work? I think not. It's probably due to a youtube/os update.

Ok I just tried it again on v2.0 and the shortcuts started working somehow.....
I don't know how that happened.... I'll see if I can pinpoint when it happens, if it occurs again.

@directentis1
Copy link

A bit of a dumb question, but I wanted to ask:

is it possible to patch the com.android.shell package on Android 10 OS (using a custom patch with Magisk, on rooted devices) to allow the use of RECORD_AUDIO premission with the latest scrcpy?

@yume-chan
Copy link
Contributor

Processes started by root don't need any permission to capture audio. See #4127

@directentis1
Copy link

Processes started by root don't need any permission to capture audio. See #4127

Is running root directly with scrcpy a security risk, compared to running with the patched com.androi.shell package?

@megapro17
Copy link
Contributor

megapro17 commented Nov 7, 2023

Recording does not support RAW audio codec... Why?

.\scrcpy --audio-codec=raw --no-video --no-playback --record-format=mkv --record=1 -Vverbose
scrcpy v2.2 <https://github.com/Genymobile/scrcpy>
INFO: ADB device found:
INFO:     -->   (usb)           R5CW11P6N1A            device  SM_S918B
DEBUG: Device serial: R5CW11P6N1A
DEBUG: Using server (portable): C:\Users\megapro17\Desktop\Soft\scrcpy-win64-v2.2\scrcpy-server
C:\Users\megapro17\Desktop\Soft\scrcpy-win64-v2.2\scrcpy-server: 1 file pushed, 0 skipped. 15.1 MB/s (64363 bytes in 0.004s)
[server] INFO: Device: [samsung] samsung SM-S918B (Android 13)
DEBUG: Server connected
DEBUG: Starting controller thread
INFO: Recording started to matroska file: 1
DEBUG: Starting receiver thread
DEBUG: Demuxer 'audio': starting thread
ERROR: The first audio packet is not a config packet
ERROR: Recording failed to 1
DEBUG: Recorder thread ended
DEBUG: Demuxer 'audio': end of frames
ERROR: Recorder error
DEBUG: quit...
DEBUG: Receiver stopped
[server] DEBUG: Controller stopped
[server] DEBUG: Device message sender stopped
[server] DEBUG: Audio recorder stopped
[server] ERROR: Audio recording error
java.io.IOException: android.system.ErrnoException: write failed: EPIPE (Broken pipe)
        at com.genymobile.scrcpy.IO.writeFully(IO.java:33)
        at com.genymobile.scrcpy.IO.writeFully(IO.java:40)
        at com.genymobile.scrcpy.Streamer.writeDisableStream(Streamer.java:61)
        at com.genymobile.scrcpy.AudioRawRecorder.record(AudioRawRecorder.java:50)
        at com.genymobile.scrcpy.AudioRawRecorder.lambda$start$0$com-genymobile-scrcpy-AudioRawRecorder(AudioRawRecorder.java:62)
        at com.genymobile.scrcpy.AudioRawRecorder$$ExternalSyntheticLambda0.run(Unknown Source:4)
        at java.lang.Thread.run(Thread.java:1012)
Caused by: android.system.ErrnoException: write failed: EPIPE (Broken pipe)
        at libcore.io.Linux.writeBytes(Native Method)
        at libcore.io.Linux.write(Linux.java:288)
        at libcore.io.ForwardingOs.write(ForwardingOs.java:938)
        at libcore.io.BlockGuardOs.write(BlockGuardOs.java:442)
        at android.system.Os.write(Os.java:987)
        at com.genymobile.scrcpy.IO.writeFully(IO.java:25)
        ... 6 more
DEBUG: Server disconnected
DEBUG: Server terminated

Is it possible to add lossless audio recording? Any quick and dirty way

@rom1v
Copy link
Collaborator Author

rom1v commented Nov 7, 2023

Raw audio streams are not supported by mkv or mp4. It would require some lossless codec (but not implemented).

In your stack trace, you don't use the official version, right? Because if I execute with your params, I get:

$ scrcpy --audio-codec=raw --no-video --no-playback --record-format=mkv --record=1 -Vverbose
scrcpy v2.2 <https://github.com/Genymobile/scrcpy>
WARN: Recording does not support RAW audio codec

@megapro17
Copy link
Contributor

Yeah I patched an exe just for fun
Android has c2.android.flac.encoder. You just need to add another encoder and enter it everywhere?

@rom1v
Copy link
Collaborator Author

rom1v commented Nov 7, 2023

Yes, it is possible to add flac support.

On Windows, you need to build an FFmpeg with flac decoding support: https://github.com/rom1v/scrcpy-deps/blob/231e4e819127dc55f1bf394943ba9887c1cb6224/build_ffmpeg_windows.sh#L40-L41

And in scrcpy, you add flac everywhere in addition to opus and aac. Here is the commit for aac: 4601735

PR welcome 😉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

9 participants