Add --hid-capture & --hid-replay options

[Rob--W]

This PR introduces the --hid-record and --hid-replay flags to support the use case of capturing events and replaying it (#4468). I have designed it such that it can also replay from a stream, currently through a fifo pipe to support the use case of targeting multiple devices at once (#4179). FIFO pipes are UNIX only, but the implementation is easily extensible to support streaming over a local socket (e.g. TCP, which would also work on Windows).

Documentation and unit tests are included.

Tested as follows:

• The file format is plain text, line-based [timestamp] [accessory id] [hex-encoded space-separated event data]. This simple format is chosen to make it easier to edit the event data manually if desired. Blank lines and lines with # are ignored, everything else is rejected.
• The parsing and serialization of the file format is fully covered by unit tests.
• I have manually tested the functionality with real devices connected to a Linux laptop via USB.
• Test devices were: Google Pixel 8, Samsung Galaxy A52s, Samsung Galaxy A53.
• Test scenarios: record to one file, replay from one file, record (overwrite) existing file, replay from non-existing file, replay and record in one command, replay from a FIFO pipe, where input is recorded to. In all cases I also tested the effect of interrupting the replay early, with the expected behavior (program closes without issues).
• I made sure that there are no memory leaks or memory safety issue by compiling with ASAN:
  meson setup x --buildtype=debug --strip -Db_lto=true -Dprebuilt_server=/usr/share/scrcpy/scrcpy-server -Db_sanitize=address,undefined && ninja -Cx

A note on the design:

• The implementation utilizes up to three extra threads to minimize the impact on the main and event thread (aoa).
• To record events, one thread is used (to perform I/O - write to file).
• To replay data, two threads are used: one to perform I/O (read from file/stream) and another to replay (waiting as needed to ensure that enough time has elapsed to replay the event). I chose two separate threads to make sure that the I/O and replay-sleep do not block each other from progressing as quick as possible. For ordinary local files, this does not really matter, but for data streamed from a fifo pipe (or in the future: sockets), blocking the pipe or socket could result in undesired interruptions or blocked pipes or closed sockets.

[Rob--W]

@rom1v Note: I have tested it extensively on Linux, but I haven't tried cross-compiling to Windows yet. I'll check whether it can compile successfully on Windows after your code review.

[Rob--W]

@rom1v The branch currently has merge conflicts due to changes in the past two months. Before I rebase, I'd like to ask whether you have unsubmitted feedback or any other blockers to merging the PR.

[rom1v]

First of all, thank you for your PR. It's a lot of work, with a lot attention to details (as per your PR description) ❤️

Sorry for the delay (the UHID feature took priority 😉).

Streaming

Initially, I did not consider the "live" recording/replay of events, so loading the whole file (which should be small) then replaying it would be ok. But I like the idea to support this use case 👍

Replay

During the replay, keyboard and mouse should be disabled (a mouse move during a replay breaks everything).
I think that replay does not even require mirroring, and there is even no need for a window (like OTG).
This could be a separate code path which just configures AOA/HID and replays synchronously (with some feedback in the console).

For me, the most important use case for this feature is to enable USB debugging on a device with a broken screen (but other people might have other use cases).

Threading

To minimize timing errors, the wait should be as close as possible as the input injection. So IMO, the thread waiting during ms_to_sleep (btw we should use µS instead of ms) should not post an event to another thread (sc_aoa_push_hid_event()) but should call sc_aoa_send_hid_event() directly immediately. In other words, the thread waiting for the deadline should be the thread which writes AOA over USB directly.

The thread which parses the file might be a separate thread posting the parsed events in a queue along with their timings, as you said:

I chose two separate threads to make sure that the I/O and replay-sleep do not block each other from progressing as quick as possible.

TBH, I think it might as well be the same thread, since the "injection" needs to wait for I/O to get the data anyway, so unless parsing takes a long time (in which case pipelining could improve the progress), I'm not sure it could make a difference. But why not.

Format

I think a binary format would be simpler. Parsing text in C is a PITA. I already parse some text because I have no choice in adb_parser.c, but if we can avoid that (more complexity, more code to maintain, more bugs…), that would be great. In another language (like Rust or Python), parsing text would be ok, but in C I really really prefer binary.

What do you think?

And I'm not sure there is an important need to edit HID event data manually in text.

What I have in mind is something like:

mouse:    |xx xx xx xx xx xx xx|02|xx xx xx xx|
keyboard: |xx xx xx xx xx xx xx|01|xx xx xx xx xx xx xx xx|
           \------------------/ ^^ \---------------------/
                 timestamp    device    HID event

(the size of the HID event might be deduced from the device)

I think we should add a protocol version at the beginning of the file, to be able to make changes later but still be able to replay old recorded files.

Also, a minor detail: the timestamps should start at 0 in the recorded file.

AOA/UHID

When you implemented the PR, HID was only supported via AOA. Now there is UHID.
But I think the feature should still target AOA specifically, because timings are captured and replayed on the client side (UHID would require to capture and replay on the server side, so the code could not be common).

---

Your contribution is great! I'm sorry that the codebase changed a lot since your PR was written (causing conflicts).

Also, I'm aware that changing from a text format to a binary format is quite a big change, but the binary version should be simpler (I'd really prefer not to maintain more code which parses text in C when this is not absolutely necessary).

Thank you again 😉

[Podingo1224]

how this work please please ? I absolutely need it please

[Rob--W]

Because this PR has not been merged you would have to check out the branch locally and compile the code yourself.

Or you could wait until I found time to update the patch to implement the changes requested by the maintainer of this project.

[Der-Lappen]

I have an issue using the functionality and cannot get this to work. I recorded some mouse activity with

./run x --otg --hid-record=onyx.log

and then tried to replay.
Unfortunately, i get this error:

ERROR: Invalid HID replay data in onyx.log

This is the log file created with the tool. Any ideas what might cause the issue?
onyx.log