diff --git a/DEVELOP.md b/DEVELOP.md new file mode 100644 index 0000000000..34cc0671f2 --- /dev/null +++ b/DEVELOP.md @@ -0,0 +1,238 @@ +# scrcpy for developers + +## Overview + +This application is composed of two parts: + - the server (`scrcpy-server.jar`), to be executed on the device, + - the client (the `scrcpy` binary), executed on the host computer. + +The client is responsible to push the server to the device and start its +execution. + +Once the client and the server are connected to each other, the server initially +sends device information (name and initial screen dimensions), then starts to +send a raw H.264 video stream of the device screen. The client decodes the video +frames, and display them as soon as possible, without buffering, to minimize +latency. The client is not aware of the device rotation (which is handled by the +server), it just knows the dimensions of the video frames. + +The client captures relevant keyboard and mouse events, that it transmits to the +server, which injects them to the device. + + + +## Server + + +### Privileges + +Capturing the screen requires some privileges, which are granted to `shell`. + +The server is a Java application (with a [`public static void main(String... +args)`][main] method), compiled against the Android framework, and executed as +`shell` on the Android device. + +[main]: server/src/main/java/com/genymobile/scrcpy/Server.java#L61 + +To run such a Java application, the classes must be [_dexed_][dex] (typically, +to `classes.dex`). If `my.package.MainClass` is the main class, compiled to +`classes.dex`, pushed to the device in `/data/local/tmp`, then it can be run +with: + + adb shell CLASSPATH=/data/local/tmp/classes.dex \ + app_process / my.package.MainClass + +_The path `/data/local/tmp` is a good candidate to push the server, since it's +readable and writable by `shell`, but not world-writable, so a malicious +application may not replace the server just before the client executes it._ + +Instead of a raw _dex_ file, `app_process` accepts a _jar_ containing +`classes.dex` (e.g. an [APK]). For simplicity, and to benefit from the gradle +build system, the server is built to an (unsigned) APK (renamed to +`scrcpy-server.jar`). + +[dex]: https://en.wikipedia.org/wiki/Dalvik_(software) +[apk]: https://en.wikipedia.org/wiki/Android_application_package + + +### Hidden methods + +Albeit compiled against the Android framework, [hidden] methods and classes are +not directly accessible (and they may differ from one Android version to +another). + +They can be called using reflection though. The communication with hidden +components is provided by [_wrappers_ classes][wrappers] and [aidl]. + +[hidden]: https://stackoverflow.com/a/31908373/1987178 +[wrappers]: https://github.com/Genymobile/scrcpy/blob/079d750d41b7849eb1b9caaa6151ef2429581584/server/src/main/java/com/genymobile/scrcpy/wrappers +[aidl]: https://github.com/Genymobile/scrcpy/blob/079d750d41b7849eb1b9caaa6151ef2429581584/server/src/main/aidl/android/view + + +### Threading + +The server uses 2 threads: + + - the **main** thread, encoding and streaming the video to the client; + - the **controller** thread, listening for _control events_ (typically, + keyboard and mouse events) from the client. + +Since the video encoding is typically hardware, there would be no benefit in +encoding and streaming in two different threads. + + +### Screen video encoding + +The encoding is managed by [`ScreenEncoder`]. + +The video is encoded using the [`MediaCodec`] API. The codec takes its input +from a [surface] associated to the display, and writes the resulting H.264 +stream to the provided output stream (the socket connected to the client). + +[`ScreenEncoder`]: https://github.com/Genymobile/scrcpy/blob/079d750d41b7849eb1b9caaa6151ef2429581584/server/src/main/java/com/genymobile/scrcpy/ScreenEncoder.java +[`MediaCodec`]: https://developer.android.com/reference/android/media/MediaCodec.html +[surface]: https://github.com/Genymobile/scrcpy/blob/079d750d41b7849eb1b9caaa6151ef2429581584/server/src/main/java/com/genymobile/scrcpy/ScreenEncoder.java#L63-L64 + +On device [rotation], the codec, surface and display are reinitialized, and a +new video stream is produced. + +New frames are produced only when changes occur on the surface. This is good +because it avoids to send unnecessary frames, but there are drawbacks: + + - it does not send any frame on start if the device screen does not change, + - after fast motion changes, the last frame may have poor quality. + +Both problems are [solved][repeat] by the flag +[`KEY_REPEAT_PREVIOUS_FRAME_AFTER`][repeat-flag]. + +[rotation]: https://github.com/Genymobile/scrcpy/blob/079d750d41b7849eb1b9caaa6151ef2429581584/server/src/main/java/com/genymobile/scrcpy/ScreenEncoder.java#L89-L92 +[repeat]: https://github.com/Genymobile/scrcpy/blob/079d750d41b7849eb1b9caaa6151ef2429581584/server/src/main/java/com/genymobile/scrcpy/ScreenEncoder.java#L125-L126 +[repeat-flag]: https://developer.android.com/reference/android/media/MediaFormat.html#KEY_REPEAT_PREVIOUS_FRAME_AFTER + + +### Input events injection + +_Control events_ are received from the client by the [`EventController`] (run in +a separate thread). There are 5 types of input events: + - keycode (cf [`KeyEvent`]), + - text (special characters may not be handled by keycodes directly), + - mouse motion/click, + - mouse scroll, + - custom command (e.g. to switch the screen on). + +All of them may need to inject input events to the system. To do so, they use +the _hidden_ method [`InputManager.injectInputEvent`] (exposed by our +[`InputManager` wrapper][inject-wrapper]). + +[`EventController`]: https://github.com/Genymobile/scrcpy/blob/079d750d41b7849eb1b9caaa6151ef2429581584/server/src/main/java/com/genymobile/scrcpy/EventController.java#L70 +[`KeyEvent`]: https://developer.android.com/reference/android/view/KeyEvent.html +[`MotionEvent`]: https://developer.android.com/reference/android/view/MotionEvent.html +[`InputManager.injectInputEvent`]: https://android.googlesource.com/platform/frameworks/base/+/oreo-release/core/java/android/hardware/input/InputManager.java#857 +[inject-wrapper]: https://github.com/Genymobile/scrcpy/blob/079d750d41b7849eb1b9caaa6151ef2429581584/server/src/main/java/com/genymobile/scrcpy/wrappers/InputManager.java#L27 + + + +## Client + +The client relies on [SDL], which provides cross-platform API for UI, input +events, threading, etc. + +The video stream is decoded by [libav] (FFmpeg). + +[SDL]: https://www.libsdl.org +[libav]: https://www.libav.org/ + +### Initialization + +On startup, in addition to _libav_ and _SDL_ initialization, the client must +push and start the server on the device, and open a socket so that they may +communicate. + +Note that the client-server roles are expressed at the application level: + + - the server _serves_ video stream and handle requests from the client, + - the client _controls_ the device through the server. + +However, the roles are inverted at the network level: + + - the client opens a server socket and listen on a port before starting the + server, + - the server connects to the client. + +This role inversion guarantees that the connection will not fail due to race +conditions, and avoids polling. + +Once the server is connected, it sends the device information (name and initial +screen dimensions). Thus, the client may init the window and renderer, before +the first frame is available. + +To minimize startup time, SDL initialization is performed while listening for +the connection from the server (see commit [90a46b4]). + +[90a46b4]: https://github.com/Genymobile/scrcpy/commit/90a46b4c45637d083e877020d85ade52a9a5fa8e + + +### Threading + +The client uses 3 threads: + + - the **main** thread, executing the SDL event loop, + - the **decoder** thread, decoding video frames, + - the **controller** thread, sending _control events_ to the server. + + +### Decoder + +The [decoder] runs in a separate thread. It uses _libav_ to decode the H.264 +stream from the socket, and notifies the main thread when a new frame is +available. + +There are two [frames] simultaneously in memory: + - the **decoding** frame, written by the decoder from the decoder thread, + - the **rendering** frame, rendered in a texture from the main thread. + +When a new decoded frame is available, the decoder _swaps_ the decoding and +rendering frame (with proper synchronization). Thus, it immediatly starts +to decode a new frame while the main thread renders the last one. + +[decoder]: https://github.com/Genymobile/scrcpy/blob/079d750d41b7849eb1b9caaa6151ef2429581584/app/src/decoder.c +[frames]: https://github.com/Genymobile/scrcpy/blob/079d750d41b7849eb1b9caaa6151ef2429581584/app/src/frames.h + + +### Controller + +The [controller] is responsible to send _control events_ to the device. It runs +in a separate thread, to avoid I/O on the main thread. + +On SDL event, received on the main thread, the [input manager][inputmanager] +creates appropriate [_control events_][controlevent]. It is responsible to +convert SDL events to Android events (using [convert]). It pushes the _control +events_ to a blocking queue hold by the controller. On its own thread, the +controller takes events from the queue, that it serializes and sends to the +client. + +[controller]: https://github.com/Genymobile/scrcpy/blob/079d750d41b7849eb1b9caaa6151ef2429581584/app/src/controller.h +[controlevent]: https://github.com/Genymobile/scrcpy/blob/079d750d41b7849eb1b9caaa6151ef2429581584/app/src/controlevent.h +[inputmanager]: https://github.com/Genymobile/scrcpy/blob/079d750d41b7849eb1b9caaa6151ef2429581584/app/src/inputmanager.h +[convert]: https://github.com/Genymobile/scrcpy/blob/079d750d41b7849eb1b9caaa6151ef2429581584/app/src/convert.h + + +### UI and event loop + +Initialization, input events and rendering are all [managed][scrcpy] in the main +thread. + +Events are handled in the [event loop], which either updates the [screen] or +delegates to the [input manager][inputmanager]. + +[scrcpy]: https://github.com/Genymobile/scrcpy/blob/079d750d41b7849eb1b9caaa6151ef2429581584/app/src/scrcpy.c +[event loop]: https://github.com/Genymobile/scrcpy/blob/079d750d41b7849eb1b9caaa6151ef2429581584/app/src/scrcpy.c#L38 +[screen]: https://github.com/Genymobile/scrcpy/blob/079d750d41b7849eb1b9caaa6151ef2429581584/app/src/screen.h + + +## Hack + +For more details, go read the code! + +If you find a bug, or have an awesome idea to implement, please discuss and +contribute ;-) diff --git a/README.md b/README.md index e2ac7eb082..c34d3b6ddb 100644 --- a/README.md +++ b/README.md @@ -1,8 +1,8 @@ # scrcpy -This project displays screens of Android devices connected on USB, and allow to -control them using the host mouse and keyboard. It does not require any _root_ -access. It works on _GNU/Linux_, _Windows_ and _Mac OS_. +This application provides display and control of Android devices connected on +USB. It does not require any _root_ access. It works on _GNU/Linux_, _Windows_ +and _Mac OS_. ![screenshot](assets/screenshot-debian-600.jpg) @@ -185,6 +185,23 @@ If several devices are listed in `adb devices`, you must specify the _serial_: | enable/disable FPS counter (on stdout) | `Ctrl`+`i` | +## Why _scrcpy_? + +A colleague challenged me to find a name as unpronounceable as [gnirehtet]. + +[`strcpy`] copies a **str**ing; `scrcpy` copies a **scr**een. + +[gnirehtet]: https://github.com/Genymobile/gnirehtet +[`strcpy`]: http://man7.org/linux/man-pages/man3/strcpy.3.html + + +## Developers + +Read the [developers page]. + +[developers page]: DEVELOP.md + + ## Licence Copyright (C) 2018 Genymobile