Scrcpy Virtual Display Mirroring Issue and Resolution for Android Devices

[mengyanshou]

Environment

• OS: macOS
• Scrcpy version: 2.5
• Installation method: brew
• Device model: Xiaomi14 MEIZU 21
• Android version: 14(both)

Describe the bug

I encountered a problem similar to one in another issue, related issue #4598

Some devices have virtual displays, such as Samsung's Dex, Smartisan's TNT. When using scrcpy to mirror these devices, the screen touch events become ineffective.

I have two devices, Xiaomi 14 and Meizu 21, both are Android 14, and the same issue exists.

If I cast to a display with an ID of 2 or 3,

when the target display is a VirtualDisplay (I haven't tested with a physical display yet), the screen control events do not work.

But if I direct all the events' display ID to the display ID created by the scrcpy-server's VirtualDisplay,

everything works fine.

The scrcpy-server created the virtual display here:

try {
    Rect videoRect = screenInfo.getVideoSize().toRect();
    virtualDisplay = ServiceManager.getDisplayManager()
            .createVirtualDisplay("scrcpy", videoRect.width(), videoRect.height(), device.getDisplayId(), surface);
    Ln.d("Display: using DisplayManager API");
} catch (Exception displayManagerException) {
    try {
        display = createDisplay();
        setDisplaySurface(display, surface, videoRotation, contentRect, unlockedVideoRect, layerStack);
        Ln.d("Display: using SurfaceControl API");
    } catch (Exception surfaceControlException) {
        Ln.e("Could not create display using DisplayManager", displayManagerException);
        Ln.e("Could not create display using SurfaceControl", surfaceControlException);
        throw new AssertionError("Could not create display");
    }
}

Here are my operations:

1. Create a virtual display using Android API

Assuming the obtained display ID is 2.

2. Run Task on the specified display

am display move-stack taskId $displayId

or

adb shell am start-activity -W -S --activityType 0 --activity-brought-to-front --display 2 -n com.nightmare.sula/.MainActivity
This will run the Android task on this virtual display.

3. Use scrcpy --display-id 2 to mirror

The screen can be rendered normally, but the events do not take effect.

I made the following modifications to the scrcpy server code:

Device.java

public final class Device {
    private final int displayId;
    private int inputDisplayId;
    // code

    public void setInputDisplayId(int id) {
        inputDisplayId = id;
    }
    // code

    public Device(Options options) throws ConfigurationException {
        displayId = options.getDisplayId();
        inputDisplayId = displayId;
    }

    public boolean injectEvent(InputEvent event, int injectMode) {
        return injectEvent(event, inputDisplayId, injectMode);
    }

    public static boolean injectKeyEvent(int action, int keyCode, int repeat, int metaState, int displayId, int injectMode) {
        long now = SystemClock.uptimeMillis();
        KeyEvent event = new KeyEvent(now, now, action, keyCode, repeat, metaState, KeyCharacterMap.VIRTUAL_KEYBOARD, 0, 0,
                InputDevice.SOURCE_KEYBOARD);
        return injectEvent(event, displayId, injectMode);
    }

    public boolean injectKeyEvent(int action, int keyCode, int repeat, int metaState, int injectMode) {
        // notice the inputDisplayId
        return injectKeyEvent(action, keyCode, repeat, metaState, inputDisplayId, injectMode);
    }

    public boolean pressReleaseKeycode(int keyCode, int injectMode) {
        // notice the inputDisplayId
        return pressReleaseKeycode(keyCode, inputDisplayId, injectMode);
    }
}

ScreenCapture.java

package com.genymobile.scrcpy;

import com.genymobile.scrcpy.wrappers.ServiceManager;
import com.genymobile.scrcpy.wrappers.SurfaceControl;

import android.graphics.Rect;
import android.hardware.display.VirtualDisplay;
import android.os.Build;
import android.os.IBinder;
import android.view.Surface;

public class ScreenCapture extends SurfaceCapture implements Device.RotationListener, Device.FoldListener {

    @Override
    public void start(Surface surface) {
        ScreenInfo screenInfo = device.getScreenInfo();
        Rect contentRect = screenInfo.getContentRect();

        // does not include the locked video orientation
        Rect unlockedVideoRect = screenInfo.getUnlockedVideoSize().toRect();
        int videoRotation = screenInfo.getVideoRotation();
        int layerStack = device.getLayerStack();

        if (display != null) {
            SurfaceControl.destroyDisplay(display);
            display = null;
        }
        if (virtualDisplay != null) {
            virtualDisplay.release();
            virtualDisplay = null;
        }

        try {
            Rect videoRect = screenInfo.getVideoSize().toRect();
            virtualDisplay = ServiceManager.getDisplayManager()
                    .createVirtualDisplay("scrcpy", videoRect.width(), videoRect.height(), device.getDisplayId(), surface);
            Ln.e("Display: using DisplayManager API virtualDisplay id -> " + virtualDisplay.getDisplay().getDisplayId());
            /// add line to set inputDisplayId
            device.setInputDisplayId(virtualDisplay.getDisplay().getDisplayId());
        } catch (Exception displayManagerException) {
            try {
                display = createDisplay();
                setDisplaySurface(display, surface, videoRotation, contentRect, unlockedVideoRect, layerStack);
                Ln.e("Display: using SurfaceControl API");
            } catch (Exception surfaceControlException) {
                Ln.e("Could not create display using DisplayManager", displayManagerException);
                Ln.e("Could not create display using SurfaceControl", surfaceControlException);
                throw new AssertionError("Could not create display");
            }
        }
    }
}

Now when I mirror the virtual display 2 again, the events work.

And again, it's very strange that

Using this scrcpy-server

SCRCPY_SERVER_PATH=server/build/scrcpy-server scrcpy --no-audio --display-id=0
It works normally when the display ID is 0, this has only been tested on Android 14, I will test more Android versions with a virtual machine later.

This is my description of this issue, I hope to get some response, if possible, I can submit a PR, thank you.

Other

Now, I can briefly introduce why I use a virtual display.

Virtual displays are now extensively used by me in Android app flow and a Launcher.

What is app flow?

An app is running on device A, now with the help of scrcpy-server, I can very smoothly run this app alone on device B.

In my own app called Uncon,

I used scrcpy-server, and then according to its protocol, used native Android code for rendering. After comparison, the efficiency is higher than SDL2 rendering.

Android Launcher?
Some Android devices (USB2.0) cannot output the screen through DP.

So I made such a try.

Create a virtual display.
Move the Launcher to the virtual display.
Through scrcpy --display-id $displayId.
This way, people can see this desktop launcher on the PC.

It looks like this.

2024-07-26.17.37.19_2.mp4

Ignore those errors.

Currently, this Launcher is running on this device.

And at this time, it does not interfere with the use of other functions of the device, which is the best.

Finally, thank you for your contribution to this project.

[rom1v]

Thank you for the post 👍

In the future, it might help to run scrcpy with a specific app (running on a virtual display). (I guess)

As you noticed, creating a new virtual display results in a new display id:

scrcpy/server/src/main/java/com/genymobile/scrcpy/video/ScreenCapture.java

Lines 52 to 53 in 071d459

	virtualDisplay = ServiceManager.getDisplayManager()
	.createVirtualDisplay("scrcpy", videoRect.width(), videoRect.height(), device.getDisplayId(), surface);

So there is the displayId of the display to mirror, and the displayId of the virtual display.

In the scrcpy main use case (simple screen mirroring), input events must use the display id of the display to mirror (in practice it's 0). If I use the one returned by virtualDisplay.getDisplay().getDisplayId() to inject events, it does not work, they are not injected.

So IIUC, we should use the original displayId only if it's 0, and the one from the virtual display otherwise?

[mengyanshou]

I think it is feasible. I am about to submit a PR to use the virtual display's id for control events when the display id is not 0.

[rom1v]

I am about to submit a PR to use the virtual display's id for control events when the display id is not 0.

But is it the correct condition?

[mengyanshou]

In fact, in the actual test, when the displayId is 0, the event specified as the id of the virtual display created, it also works, and this has only been tested on Android 14
Or is it controlled by a switch? For example, --dispatch-to-virtual

[rom1v]

In fact, in the actual test, when the displayId is 0, the event specified as the id of the virtual display created, it also works, and this has only been tested on Android 14

It seems it does not work on a Pixel 8 with Android 14.

Here is the diff I tested:

diff

diff --git a/server/src/main/java/com/genymobile/scrcpy/device/Device.java b/server/src/main/java/com/genymobile/scrcpy/device/Device.java
index 5a1083fde..eee35254e 100644
--- a/server/src/main/java/com/genymobile/scrcpy/device/Device.java
+++ b/server/src/main/java/com/genymobile/scrcpy/device/Device.java
@@ -65,6 +65,8 @@ public final class Device {
      */
     private final int displayId;
 
+    private int inputDisplayId;
+
     /**
      * The surface flinger layer stack associated with this logical display
      */
@@ -246,7 +248,8 @@ public final class Device {
     }
 
     public boolean injectKeyEvent(int action, int keyCode, int repeat, int metaState, int injectMode) {
-        return injectKeyEvent(action, keyCode, repeat, metaState, displayId, injectMode);
+        Ln.d("injectKeyEvent inputDisplayId=" + inputDisplayId);
+        return injectKeyEvent(action, keyCode, repeat, metaState, inputDisplayId, injectMode);
     }
 
     public static boolean pressReleaseKeycode(int keyCode, int displayId, int injectMode) {
@@ -255,7 +258,7 @@ public final class Device {
     }
 
     public boolean pressReleaseKeycode(int keyCode, int injectMode) {
-        return pressReleaseKeycode(keyCode, displayId, injectMode);
+        return pressReleaseKeycode(keyCode, inputDisplayId, injectMode);
     }
 
     public static boolean isScreenOn() {
@@ -401,4 +404,8 @@ public final class Device {
         DisplayInfo displayInfo = ServiceManager.getDisplayManager().getDisplayInfo(displayId);
         return displayInfo.getRotation();
     }
+
+    public void setInputDisplayId(int id) {
+        inputDisplayId = id;
+    }
 }
diff --git a/server/src/main/java/com/genymobile/scrcpy/video/ScreenCapture.java b/server/src/main/java/com/genymobile/scrcpy/video/ScreenCapture.java
index fbeca2af0..d56d168aa 100644
--- a/server/src/main/java/com/genymobile/scrcpy/video/ScreenCapture.java
+++ b/server/src/main/java/com/genymobile/scrcpy/video/ScreenCapture.java
@@ -52,6 +52,9 @@ public class ScreenCapture extends SurfaceCapture implements Device.RotationList
             virtualDisplay = ServiceManager.getDisplayManager()
                     .createVirtualDisplay("scrcpy", videoRect.width(), videoRect.height(), device.getDisplayId(), surface);
             Ln.d("Display: using DisplayManager API");
+            int displayId = virtualDisplay.getDisplay().getDisplayId();
+            Ln.d("====" + displayId);
+            device.setInputDisplayId(displayId);
         } catch (Exception displayManagerException) {
             try {
                 display = createDisplay();

[mengyanshou]

I just noticed this code, so indeed, when displayId is 0, the events injected by injectEvent should also correspond to displayId 0

public static boolean injectEvent(InputEvent inputEvent, int displayId, int injectMode) {
    if (!supportsInputEvents(displayId)) {
        throw new AssertionError("Could not inject input event if !supportsInputEvents()");
    }
    if (displayId != 0 && !InputManager.setDisplayId(inputEvent, displayId)) {
        return false;
    }

    return ServiceManager.getInputManager().injectInputEvent(inputEvent, injectMode);
}

[mengyanshou]

I'll try not to affect the original functionality of scrcpy. First, I've added an option dispatch_to_vd to the scrcpy-server part. In the ScreenCapture.start function call:

if (device.isDispatchToVD()) {
    device.setInputDisplayId(virtualDisplay.getDisplay().getDisplayId());
}

Is this feasible?

This is the full diff

diff --git a/server/src/main/java/com/genymobile/scrcpy/Device.java b/server/src/main/java/com/genymobile/scrcpy/Device.java
index 8d0ee231..3969657f 100644
--- a/server/src/main/java/com/genymobile/scrcpy/Device.java
+++ b/server/src/main/java/com/genymobile/scrcpy/Device.java
@@ -60,6 +60,7 @@ public final class Device {
      * Logical display identifier
      */
     private final int displayId;
+    private int inputDisplayId;
 
     /**
      * The surface flinger layer stack associated with this logical display
@@ -68,8 +69,15 @@ public final class Device {
 
     private final boolean supportsInputEvents;
 
+    /**
+     * Whether to dispatch input events to the virtual display
+     */
+    private final boolean dispatchToVD;
+
     public Device(Options options) throws ConfigurationException {
         displayId = options.getDisplayId();
+        inputDisplayId = displayId;
+        dispatchToVD = options.getDispatchToVD();
         DisplayInfo displayInfo = ServiceManager.getDisplayManager().getDisplayInfo(displayId);
         if (displayInfo == null) {
             Ln.e("Display " + displayId + " not found\n" + LogUtils.buildDisplayListMessage());
@@ -168,6 +176,14 @@ public final class Device {
         return displayId;
     }
 
+    public int getInputDisplayId() {
+        return inputDisplayId;
+    }
+
+    public void setInputDisplayId(int id) {
+        inputDisplayId = id;
+    }
+
     public synchronized void setMaxSize(int newMaxSize) {
         maxSize = newMaxSize;
         screenInfo = ScreenInfo.computeScreenInfo(screenInfo.getReverseVideoRotation(), deviceSize, crop, newMaxSize, lockVideoOrientation);
@@ -218,11 +234,14 @@ public final class Device {
         return supportsInputEvents;
     }
 
+    public  boolean isDispatchToVD() {
+        return dispatchToVD;
+    }
+
     public static boolean injectEvent(InputEvent inputEvent, int displayId, int injectMode) {
         if (!supportsInputEvents(displayId)) {
             throw new AssertionError("Could not inject input event if !supportsInputEvents()");
         }
-
         if (displayId != 0 && !InputManager.setDisplayId(inputEvent, displayId)) {
             return false;
         }
@@ -231,7 +250,7 @@ public final class Device {
     }
 
     public boolean injectEvent(InputEvent event, int injectMode) {
-        return injectEvent(event, displayId, injectMode);
+        return injectEvent(event, inputDisplayId, injectMode);
     }
 
     public static boolean injectKeyEvent(int action, int keyCode, int repeat, int metaState, int displayId, int injectMode) {
@@ -242,7 +261,7 @@ public final class Device {
     }
 
     public boolean injectKeyEvent(int action, int keyCode, int repeat, int metaState, int injectMode) {
-        return injectKeyEvent(action, keyCode, repeat, metaState, displayId, injectMode);
+        return injectKeyEvent(action, keyCode, repeat, metaState, inputDisplayId, injectMode);
     }
 
     public static boolean pressReleaseKeycode(int keyCode, int displayId, int injectMode) {
@@ -251,7 +270,7 @@ public final class Device {
     }
 
     public boolean pressReleaseKeycode(int keyCode, int injectMode) {
-        return pressReleaseKeycode(keyCode, displayId, injectMode);
+        return pressReleaseKeycode(keyCode, inputDisplayId, injectMode);
     }
 
     public static boolean isScreenOn() {
diff --git a/server/src/main/java/com/genymobile/scrcpy/Options.java b/server/src/main/java/com/genymobile/scrcpy/Options.java
index 9b1d8d8d..b8242aab 100644
--- a/server/src/main/java/com/genymobile/scrcpy/Options.java
+++ b/server/src/main/java/com/genymobile/scrcpy/Options.java
@@ -47,6 +47,7 @@ public class Options {
     private boolean listDisplays;
     private boolean listCameras;
     private boolean listCameraSizes;
+    private boolean dispatchToVD = false;
 
     // Options not used by the scrcpy client, but useful to use scrcpy-server directly
     private boolean sendDeviceMeta = true; // send device name and size
@@ -194,6 +195,10 @@ public class Options {
         return listEncoders || listDisplays || listCameras || listCameraSizes;
     }
 
+    public boolean getDispatchToVD() {
+        return dispatchToVD;
+    }
+
     public boolean getListEncoders() {
         return listEncoders;
     }
@@ -370,6 +375,9 @@ public class Options {
                 case "list_camera_sizes":
                     options.listCameraSizes = Boolean.parseBoolean(value);
                     break;
+                case "dispatch_to_vd":
+                    options.dispatchToVD = Boolean.parseBoolean(value);
+                    break;
                 case "camera_id":
                     if (!value.isEmpty()) {
                         options.cameraId = value;
diff --git a/server/src/main/java/com/genymobile/scrcpy/ScreenCapture.java b/server/src/main/java/com/genymobile/scrcpy/ScreenCapture.java
index 090c96f0..d5222723 100644
--- a/server/src/main/java/com/genymobile/scrcpy/ScreenCapture.java
+++ b/server/src/main/java/com/genymobile/scrcpy/ScreenCapture.java
@@ -49,6 +49,9 @@ public class ScreenCapture extends SurfaceCapture implements Device.RotationList
             virtualDisplay = ServiceManager.getDisplayManager()
                     .createVirtualDisplay("scrcpy", videoRect.width(), videoRect.height(), device.getDisplayId(), surface);
             Ln.d("Display: using DisplayManager API");
+            if (device.isDispatchToVD()) {
+                device.setInputDisplayId(virtualDisplay.getDisplay().getDisplayId());
+            }
         } catch (Exception displayManagerException) {
             try {
                 display = createDisplay();

[eiyooooo]

Please note that only touch events or other screen control events confirmed to be non-functional need to redirect to mirrorVD.
pressReleaseKeycode or other normal key events should always be injected to the original DisplayId.

[mengyanshou]

Please note that only touch events or other screen control events confirmed to be non-functional need to redirect to mirrorVD. pressReleaseKeycode or other normal key events should always be injected to the original DisplayId.

Ok received, I have been busy these days, I will request PR later

[mengyanshou]

I have submitted the pull request, please check if there are any errors #5214

[rom1v]

Please test #5370.

[mengyanshou]

add dispatch_to_vd option to server #5214

see #5370 (comment)