Add v4l2loopback support #2268

rom1v · 2021-04-19T18:35:29Z

This is a follow up for v4l2loopback support (#2232 #2233) implemented by @martinellimarco.

I refactored scrcpy to support this feature properly, then I rebased @martinellimarco's work to use the refactored codebase.

The main purpose is to make the v4l2 component consume frames directly from the decoder, in order to avoid decoding the video stream twice if mirroring display is enabled:

                      screen/display
                    /
            decoder
          /         \
         /            v4l2_sink
  stream
         \
           recorder

          \--------/  \-------------/
         packet sinks   frame sinks

Here is a summary of the changes:

Remove --render-expired-frames option (see 8bae1f6 for details)
Remove compat with old FFmpeg decodeing and codec params API (deprecated since 2016)
Make video_buffer not consume the input frame (to make possible to send it to several sinks)
Add two abstractions (traits):
- sc_packet_sink: a component which receives packets. It is implemented by the decoder and the recorder, and the stream now push its packets to the registered packet sinks (without depending on the concrete sink types)
- sc_frame_sink: a component which receives frames. It is implemented by the screen (and now v4l2), and the decoder now pushes its frames to the registered frame sinks (without depending on the concrete sink types)
Add a new mode for --lock-video-orientation: initial. This locks the video in the orientation the device happens to be on start and avoids to pass an explicit value. This mode is set by default when v4l2 sink is enabled.
Rework the v4l2 sink component to implement sc_frame_sink, and be registered to the decoder.
Use a video_buffer to always send the very last frame, like for the display (it may skip frames, it's on purpose, is it a problem?)
Add some documentation in the README

Here is a usage summary:

sudo apt install v4l2loopback-dkms   # install v4l2loopback
sudo modprobe v4l2loopback           # create a video device
scrcpy --v4l2-sink /dev/video1       # mirror + v4l2
scrcpy --v4l2-sink /dev/video1 -N    # v4l2 only
ffplay -i /dev/video1                # test

The video stream could also be captured from OBS.

Thank you for your reviews/feedbacks.

The recorder thread wrote the whole content except the trailer, which was odd.

This flag forced the decoder to wait for the previous frame to be consumed by the display. It was initially implemented as a compilation flag for testing, not intended to be exposed at runtime. But to remove ifdefs and to allow users to test this flag easily, it had finally been exposed by commit ebccb9f. In practice, it turned out to be useless: it had no practical impact, and it did not solve or mitigate any performance issues causing frame skipping. But that added some complexity to the codebase: it required an additional condition variable, and made video buffer calls possibly blocking, which in turn required code to interrupt it on exit. To prepare support for multiple sinks plugged to the decoder (display and v4l2 for example), the blocking call used for pacing the decoder output becomes unacceptable, so just remove this useless "feature".

The new API has been introduced in 2016 in libavcodec 57.xx, it's very old. This will avoid to maintain two code paths for decoding.

The new API has been introduced in 2016 in libavformat 57.xx, it's very old. This will avoid to maintain two code paths for codec parameters.

The video buffer took ownership of the producer frame (so that it could swap frames quickly). In order to support multiple sinks plugged to the decoder, the decoded frame must not be consumed by the display video buffer. Therefore, move the producer and consumer frames out of the video buffer, and use FFmpeg AVFrame refcounting to share ownership while avoiding copies.

This will allow to get the parent of an embedded struct.

This trait will allow to abstract the concrete sink types from the packet producer (stream.c).

This will make further commits more readable.

The fact that the recorder uses a separate thread is an internal detail, so the functions _start(), _stop() and _join() should not be exposed. Instead, start the thread on _open() and _stop()+_join() on close(). This paves the way to expose the recorder as a packet sink trait.

Make recorder implement the packet sink trait. This will allow the stream to push packets without depending on the concrete sink type.

This will make further commits more readable.

Make decoder implement the packet sink trait. This will allow the stream to push packets without depending on the concrete sink type.

Now that decoder and recorder implement the packet sink trait, make stream push packets to the sinks without depending on the concrete sink types.

This trait will allow to abstract the concrete sink types from the frame producer (decoder.c).

Make screen implement the frame sink trait. This will allow the decoder to push frames without depending on the concrete sink type.

Now that screen implements the packet sink trait, make decoder push packets to the sinks without depending on the concrete sink types.

The video buffer is now an internal detail of the screen component. Since the screen is plugged to the decoder via the frame sink trait, the decoder does not access to the video buffer anymore.

Now that screen is both the owner and the listener of the video buffer, execute the code directly without callbacks.

The destruction order is important, but tricky, because the screen is open/close by the decoder, but destroyed by scrcpy.c on the main thread. Add assertions to guarantee that the screen is not destroyed before being closed.

The screen may not be destroyed immediately on close to avoid undefined behavior, because it may still receive events from the decoder. But the visual window must still be closed immediately.

Only initialize ops and parameters copy from recorder_init(). It was inconsistent to initialize some fields from _init() and some others from _open().

There are many initializations in recorder_open(). Handle RAII-like deinitialization using gotos.

EAGAIN was only handled on receive_frame. In practice, it should not be necessary, since one packet always contains one frame. But just in case.

Add a function to know if a string list, using some separator, contains a specific string.

The AVOutputFormat name is a string list: it contains names (possibly only one) separated by '.'

Add a new mode to the --lock-video-orientation option, to lock the initial orientation of the device. This avoids to pass an explicit value (0, 1, 2 or 3) and think about which is the right one.

If the option is set without argument, lock the initial device orientation (as if the value "initial" was passed).

sugoides · 2021-07-06T07:46:38Z

Also OBS virtual cam can also be used as source in OBS Studio, so I thought that it can also be used like v4l2

@rom1v

sugoides · 2021-07-06T07:52:01Z

Or also using pyvirtualcamera?
@rom1v

Sebastiangperez · 2021-12-28T21:01:57Z

Hi guys , when i use this command :
scrcpy --v4l2-sink /dev/video0
this is the message that i got.
scrcpy: unrecognized option '--v4l2-sink'

rom1v · 2021-12-28T21:10:37Z

Your scrcpy version is too old.

Sebastiangperez · 2021-12-28T21:31:27Z

Your scrcpy version is too old.

SDL 2.0.14
libavcodec 58.91.100
libavformat 58.45.100
libavutil 56.51.100

Sorry , i have a snap package installed and also via apt , it was messing both.

rom1v · 2021-12-28T21:40:46Z

This feature has been introduced in scrcpy v1.18.

You can build/install the latest version very easily: https://github.com/Genymobile/scrcpy/blob/master/BUILD.md#simple

Sebastiangperez · 2021-12-28T21:43:18Z

now i getting this

scrcpy 1.21 https://github.com/Genymobile/scrcpy
INFO: Video orientation is locked for v4l2 sink. See --lock-video-orientation.
/usr/local/share/scrcpy/scrcpy-server: 1 file pushed. 2.5 MB/s (40067 bytes in 0.015s)
[server] INFO: Device: samsung SM-G955F (Android 9)
INFO: Renderer: opengl
INFO: OpenGL version: 4.6.0 NVIDIA 470.86
INFO: Trilinear filtering enabled
INFO: Initial texture: 1080x2216
ERROR: Failed to open output device: /dev/video0
ERROR: Could not open frame sink 1
ERROR: Could not open decoder sinks
ERROR: Could not open packet sink 0
ERROR: Could not open stream sinks
WARN: Device disconnected
WARN: Killing the server...

rom1v · 2021-12-28T21:44:25Z

How did you create the v4l2 loopback device?

https://github.com/Genymobile/scrcpy#v4l2loopback

Sebastiangperez · 2021-12-28T21:45:34Z

Yes , i did

v4l2-ctl -d /dev/video0 -l

User Controls

                keep_format 0x0098f900 (bool)   : default=0 value=0
          sustain_framerate 0x0098f901 (bool)   : default=0 value=0
                    timeout 0x0098f902 (int)    : min=0 max=100000 step=1 default=0 value=0
           timeout_image_io 0x0098f903 (bool)   : default=0 value=0

➜ ~ v4l2-ctl --list-devices
Dummy video device (0x0000) (platform:v4l2loopback-000):
/dev/video0

rom1v · 2021-12-28T21:49:03Z

ERROR: Failed to open output device: /dev/video0

Might be a permission problem.

What are the results of:

ls -lh /dev/video0
id

Sebastiangperez · 2021-12-28T21:51:27Z

Is on root user and video group
crw-rw----+ 1 root video 81, 0 dic 28 18:48 /dev/video0

aliyazdi75 · 2022-01-03T12:39:29Z

Hi @rom1v, I also faced this issue, is there any workaround to fix this?

Quackdoc · 2022-01-04T19:36:22Z

try running scrcpy with root

rom1v · 2022-01-04T19:42:41Z

No, you should not run scrcpy as root.

Check with id if your user belongs to the video group.

aliyazdi75 · 2022-01-04T19:52:22Z

@rom1v
How can I check that? Also this is my id:

uid=1000(aliyazdi75) gid=1000(aliyazdi75) groups=1000(aliyazdi75),4(adm),24(cdrom),27(sudo),30(dip),46(plugdev),119(lpadmin),130(lxd),131(sambashare),998(docker)

rom1v · 2022-01-04T19:55:51Z

So your user is not in the video group (it's not listed).

sudo adduser $USER video

then reboot

aliyazdi75 · 2022-01-04T20:13:45Z

Thanks, the video group added but I got another error now:

scrcpy 1.21 <https://github.com/Genymobile/scrcpy>
INFO: Video orientation is locked for v4l2 sink. See --lock-video-orientation.
/usr/local/share/scrcpy/scrcpy-server: 1 file pushed. 1.3 MB/s (40067 bytes in 0.029s)
[server] INFO: Device: samsung SM-G955F (Android 9)
INFO: Renderer: opengl
INFO: OpenGL version: 3.0 Mesa 20.0.8
INFO: Trilinear filtering enabled
INFO: Initial texture: 1080x2216
ERROR: Failed to open output device: /dev/video10
ERROR: Could not open frame sink 1
ERROR: Could not open decoder sinks
ERROR: Could not open packet sink 0
ERROR: Could not open stream sinks
WARN: Device disconnected
WARN: Killing the server...

rom1v · 2022-01-04T20:24:26Z

ERROR: Failed to open output device: /dev/video10

Looks like the very same error. Did you reboot after adding the video group?

aliyazdi75 · 2022-01-04T20:27:21Z

@rom1v yes I did and this is the new id:

uid=1000(aliyazdi75) gid=1000(aliyazdi75) groups=1000(aliyazdi75),4(adm),24(cdrom),27(sudo),30(dip),44(video),46(plugdev),119(lpadmin),130(lxd),131(sambashare),998(docker)

and v4l2-ctl --list-devices for the new one I created:

scrcpy virtual camera (platform:v4l2loopback-000):
	/dev/video10

aliyazdi75 · 2022-01-04T20:31:06Z

No:

[video4linux2,v4l2 @ 0x7f03d4000bc0] Not a video capture device.0   
/dev/video10: No such device

and

VLC media player 3.0.16 Vetinari (revision 3.0.16-0-g5e70837d8d)
[00005648bdbd4a00] main libvlc: Running vlc with the default interface. Use 'cvlc' to use vlc without interface.
Qt: Session management error: Could not open network socket
[00007fc654001120] v4l2 demux error: cannot open device '/dev/video10': Operation not permitted
[00007fc654001120] v4l2 demux error: cannot open device '/dev/video10': Operation not permitted
[00007fc654002790] v4l2 stream error: cannot open device '/dev/video10': Operation not permitted
QObject::~QObject: Timers cannot be stopped from another thread

rom1v · 2022-01-04T20:32:04Z

Try to output a file to v4l2:

ffmpeg -re -i video.mp4 -f v4l2 /dev/videoNNN

and play it:

ffplay /dev/videoNNN

rom1v · 2022-01-04T20:32:50Z

v4l2 demux error: cannot open device '/dev/video10': Operation not permitted

You definitely have permissions issues with your v4l2 device.

aliyazdi75 · 2022-01-04T20:35:12Z

The previous message:

Try to output a file to v4l2:
ffmpeg -re -i video.mp4 -f v4l2 /dev/videoNNN
and play it:
ffplay /dev/videoNNN

Worked as expected.

v4l2 demux error: cannot open device '/dev/video10': Operation not permitted

You definitely have permissions issues with your v4l2 device.

How can I fix this?

aliyazdi75 · 2022-01-05T09:48:29Z

@rom1v Sorry I found the problem. It was because I was using snap version and it needed to get access to the camera by running this command: sudo snap connect scrcpy:camera This snap version is making me ccrasy, I'm switching to the build version. But it's really annoying for the updates. Is it possible to update apt version?

nikp123 · 2024-01-21T16:06:50Z

For those experiencing an issue where it "fails to encode", you can try lowering the horizontal resolution by adding: -m400 or such.

Uzver123 · 2024-02-05T13:35:01Z

Is there something similar for Windows? I mean a way to grab the video from ScrCpy without showing the ScrCpy window?

rom1v · 2024-02-05T13:41:09Z

@Uzver123 No

Uzver123 · 2024-02-05T16:11:42Z

@Uzver123 No

Could you make something like webcam splitter to create virtual webcam, i am interested to using ScrCpy to capture phone's camera and output as webcam, any video capture program can connect to webcam.

rom1v added 28 commits April 19, 2021 20:19

Write trailer from recorder thread

e3da97a

The recorder thread wrote the whole content except the trailer, which was odd.

Remove compat with old FFmpeg decoding API

adaa889

The new API has been introduced in 2016 in libavcodec 57.xx, it's very old. This will avoid to maintain two code paths for decoding.

Remove compat with old FFmpeg codec params API

7bb17e1

The new API has been introduced in 2016 in libavformat 57.xx, it's very old. This will avoid to maintain two code paths for codec parameters.

Add container_of() macro

321bf79

This will allow to get the parent of an embedded struct.

Add packet sink trait

1116502

This trait will allow to abstract the concrete sink types from the packet producer (stream.c).

Reorder recorder functions

fb07f4a

This will make further commits more readable.

Expose recorder as packet sink

71327e8

Make recorder implement the packet sink trait. This will allow the stream to push packets without depending on the concrete sink type.

Reorder decoder functions

f82aab2

This will make further commits more readable.

Expose decoder as packet sink

f27403e

Make decoder implement the packet sink trait. This will allow the stream to push packets without depending on the concrete sink type.

Make stream push packets to sinks

eb97139

Now that decoder and recorder implement the packet sink trait, make stream push packets to the sinks without depending on the concrete sink types.

Add frame sink trait

bb6ac2b

This trait will allow to abstract the concrete sink types from the frame producer (decoder.c).

Expose screen as frame sink

6d8f0a2

Make screen implement the frame sink trait. This will allow the decoder to push frames without depending on the concrete sink type.

Make decoder push frames to sinks

cec5bcb

Now that screen implements the packet sink trait, make decoder push packets to the sinks without depending on the concrete sink types.

Move video_buffer to screen

8ede4b3

The video buffer is now an internal detail of the screen component. Since the screen is plugged to the decoder via the frame sink trait, the decoder does not access to the video buffer anymore.

Remove video_buffer callbacks

7309a57

Now that screen is both the owner and the listener of the video buffer, execute the code directly without callbacks.

Assert screen closed on destroy

c9a5611

The destruction order is important, but tricky, because the screen is open/close by the decoder, but destroyed by scrcpy.c on the main thread. Add assertions to guarantee that the screen is not destroyed before being closed.

Hide the window immediately on close

99c6a76

The screen may not be destroyed immediately on close to avoid undefined behavior, because it may still receive events from the decoder. But the visual window must still be closed immediately.

Initialize recorder fields on open

49582e4

Only initialize ops and parameters copy from recorder_init(). It was inconsistent to initialize some fields from _init() and some others from _open().

Handle errors using gotos in recorder_open()

c8eb5cc

There are many initializations in recorder_open(). Handle RAII-like deinitialization using gotos.

Handle EAGAIN on send_packet in decoder

066f4f5

EAGAIN was only handled on receive_frame. In practice, it should not be necessary, since one packet always contains one frame. But just in case.

Fix recorder comment

b2e67b5

Add strlist_contains()

9d3bf09

Add a function to know if a string list, using some separator, contains a specific string.

Use strlist_contains() to find a muxer

195073d

The AVOutputFormat name is a string list: it contains names (possibly only one) separated by '.'

Add --lock-video-orientation=initial

a963d89

Add a new mode to the --lock-video-orientation option, to lock the initial orientation of the device. This avoids to pass an explicit value (0, 1, 2 or 3) and think about which is the right one.

Make --lock-video-orientation argument optional

8605696

If the option is set without argument, lock the initial device orientation (as if the value "initial" was passed).

rom1v added v4l2 webcam labels Apr 19, 2021

rom1v mentioned this pull request Oct 3, 2021

Add HID keyboard simulation via USB HID over AoAv2 #2632

Closed

rom1v mentioned this pull request Oct 27, 2021

Capture with OBS without scrcpy window #467

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add v4l2loopback support #2268

Add v4l2loopback support #2268

rom1v commented Apr 19, 2021 •

edited

Loading

sugoides commented Jul 6, 2021

sugoides commented Jul 6, 2021

Sebastiangperez commented Dec 28, 2021

rom1v commented Dec 28, 2021

Sebastiangperez commented Dec 28, 2021 •

edited

Loading

rom1v commented Dec 28, 2021

Sebastiangperez commented Dec 28, 2021

rom1v commented Dec 28, 2021 •

edited

Loading

Sebastiangperez commented Dec 28, 2021 •

edited

Loading

rom1v commented Dec 28, 2021

Sebastiangperez commented Dec 28, 2021 •

edited

Loading

aliyazdi75 commented Jan 3, 2022

Quackdoc commented Jan 4, 2022

rom1v commented Jan 4, 2022

aliyazdi75 commented Jan 4, 2022

rom1v commented Jan 4, 2022

aliyazdi75 commented Jan 4, 2022

rom1v commented Jan 4, 2022

aliyazdi75 commented Jan 4, 2022

aliyazdi75 commented Jan 4, 2022

rom1v commented Jan 4, 2022

rom1v commented Jan 4, 2022

aliyazdi75 commented Jan 4, 2022

aliyazdi75 commented Jan 5, 2022

nikp123 commented Jan 21, 2024

Uzver123 commented Feb 5, 2024

rom1v commented Feb 5, 2024

Uzver123 commented Feb 5, 2024

Add v4l2loopback support #2268

Add v4l2loopback support #2268

Conversation

rom1v commented Apr 19, 2021 • edited Loading

sugoides commented Jul 6, 2021

sugoides commented Jul 6, 2021

Sebastiangperez commented Dec 28, 2021

rom1v commented Dec 28, 2021

Sebastiangperez commented Dec 28, 2021 • edited Loading

rom1v commented Dec 28, 2021

Sebastiangperez commented Dec 28, 2021

rom1v commented Dec 28, 2021 • edited Loading

Sebastiangperez commented Dec 28, 2021 • edited Loading

rom1v commented Dec 28, 2021

Sebastiangperez commented Dec 28, 2021 • edited Loading

aliyazdi75 commented Jan 3, 2022

Quackdoc commented Jan 4, 2022

rom1v commented Jan 4, 2022

aliyazdi75 commented Jan 4, 2022

rom1v commented Jan 4, 2022

aliyazdi75 commented Jan 4, 2022

rom1v commented Jan 4, 2022

aliyazdi75 commented Jan 4, 2022

aliyazdi75 commented Jan 4, 2022

rom1v commented Jan 4, 2022

rom1v commented Jan 4, 2022

aliyazdi75 commented Jan 4, 2022

aliyazdi75 commented Jan 5, 2022

nikp123 commented Jan 21, 2024

Uzver123 commented Feb 5, 2024

rom1v commented Feb 5, 2024

Uzver123 commented Feb 5, 2024

rom1v commented Apr 19, 2021 •

edited

Loading

Sebastiangperez commented Dec 28, 2021 •

edited

Loading

rom1v commented Dec 28, 2021 •

edited

Loading

Sebastiangperez commented Dec 28, 2021 •

edited

Loading

Sebastiangperez commented Dec 28, 2021 •

edited

Loading