Add HID keyboard simulation via USB HID over AoAv2

[AlynxZhou]

Add HID keyboard simulation via USB HID over AoAv2, which provides better input experience if connected with USB cable.

Currently this needs --serial to provide USB serial, we may update it to fetch serial via adb automatically.

A new option --input-mode is added, by default is auto, try HID first and fallback to old input injection, user can set hid or inject manually, it will fail if user's selection not work.

Please add your comments if you have any advice and I'll try to update it.

Fixes #279.

[AlynxZhou]

Oh sorry I forgot Windows, I'll do it after I have my dinner.

[rom1v]

Thank you for your work 👍

I will try to review this week-end (this might take some time, it's a big change/PR 😉).

[AlynxZhou]

I've built libusb for windows, however it seems cannot open and compare serial for my phone, maybe some driver is needed but I don't know, windows is really a mess...

[AlynxZhou]

Good news: I build windows version successfully with libusb dependencies.

Bad news: libusb unable to open Android phone on windows, I've searched a lot and only find a relevant page: https://libusb-devel.narkive.com/vENuKzdR/getting-the-serial-number-of-the-camera-fails, seems that windows has some limitation of accessing multi end point device, and libusb needs to visit end point 0 directly which windows does not allow.

So I think we have nothing to do with it, it will fallback to inject mode on windows, so just leave it here should be fine, and Linux users need this feature most (so they can use apps that do not have Linux version but Android version).

Please take your time to review it and I am waiting for advice! Thanks!

[rom1v]

Bad news: libusb unable to open Android phone on windows, I've searched a lot and only find a relevant page: https://libusb-devel.narkive.com/vENuKzdR/getting-the-serial-number-of-the-camera-fails, seems that windows has some limitation of accessing multi end point device, and libusb needs to visit end point 0 directly which windows does not allow.

OK, I never managed to use AOA on Windows either. It is totally ok to implement it for Linux only (at least as a first step). Moreover, there is already another feature (V4L2) available only on Linux. I suggest you keep your commit Add Windows build support for libusb locally, it might be useful later, but drop it from this PR for now.

Also, I have a little request (I didn't review yet, but looked at commit titles): could you rebase -i to clean up some history. For example, Move HID keyboard setup code into functions: this code could have been written directly "into functions" I guess. Or Make sdl_keymod_to_hid_modifiers() more readable (make it more readable directly). And other spurious changes to make clean commits… Thank you :)

[AlynxZhou]

Bad news: libusb unable to open Android phone on windows, I've searched a lot and only find a relevant page: https://libusb-devel.narkive.com/vENuKzdR/getting-the-serial-number-of-the-camera-fails, seems that windows has some limitation of accessing multi end point device, and libusb needs to visit end point 0 directly which windows does not allow.

OK, I never managed to use AOA on Windows either. It is totally ok to implement it for Linux only (at least as a first step). Moreover, there is already another feature (V4L2) available only on Linux. I suggest you keep your commit Add Windows build support for libusb locally, it might be useful later, but drop it from this PR for now.

Also, I have a little request (I didn't review yet, but looked at commit titles): could you rebase -i to clean up some history. For example, Move HID keyboard setup code into functions: this code could have been written directly "into functions" I guess. Or Make sdl_keymod_to_hid_modifiers() more readable (make it more readable directly). And other spurious changes to make clean commits… Thank you :)

Will do it tomorrow, it's midnight 3:00 and I am gonna sleep after fighting against windows for whole night 😹

[AlynxZhou]

Squashed into 2 commits.

I think it's better to keep the windows build commit here, since the code itself is correct -- it's not our problem but libusb's problem, once it works correctly (maybe hard though), our code will work.

Plus if I dropped build support for libusb on Windows, I need to add a lot of #ifdef _WIN32 (or something similiar) to build release on Windows (since no libusb is here and aoa_hid hid_keyboard needs it when compiling so the only way will be disable them, too). So I think keep it will be better than drop it.

[AlynxZhou]

😄 I have to rebase it again because I found that I mis-spell AOAv2 into AoAv2.

[chldbwnstm]

Can we please simulate mouse click as well?

[AlynxZhou]

I tried locally, but the position syncing is bad, and position is needed for detect what has been clicked. And it should not be better then current experience.

…

On Mon, Sep 20, 2021, 13:07 humblebee ***@***.***> wrote: Can we please simulate mouse click as well? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#2632 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACANVR6IXSEZGT3F4ZAVTKTUC26SZANCNFSM5ECBSPKQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[rom1v]

@AlynxZhou I don't know if you received my e-mail this week-end (probably went into your spams, like often with gmail recipients). (If you missed it, you didn't miss a lot, I said I had no time yet.)

Anyway, here is some quick feedback.

Keyboard

I did a quick test on my devices, when I use HID (either in French or English locale) from my laptop (a French AZERTY keyboard), it injects keys as if my keyboard was QWERTY (so numbers, accented characters, some letters… are incorrect).

I'm not sure how to solve this problem if we inject scancodes. Any idea?

Input mode selection

(Maybe a good idea is like usbaudio, if only one device with adb, use it, others ask for serial?)

I'd really prefer to be explicit. Otherwise, people will never know immediately which method is used and why/why not.

After reflection, I confirm that I would really prefer the mode to be explicit. It greatly impacts how the injection works, so it should not just be the consequence of some heuristic/fallback.

If it's explicitly selected, scrcpy should fail IMO, not fallback.

Then what should be the default option?

IMO, the default should be the Android text injection: it's the most basic, but works the same way everywhere:

• Linux, Windows and macOS
• over USB and wirelessly / TCP/IP

HID injection is better, but only over USB and only on Linux, and it might cause significant differences (hopefully for the better, but it can also impact things like the AZERTY/QWERTY issue I encounter).

So IMO it should be enabled explicitly with scrcpy --input-method=hid.

Auto device selection

For now, as we discussed, HID can only be enabled if a serial is explicitly passed.

I'm not asking you to do anything about it, I'm just listing some ideas/solutions.

We could use adb get-serialno and match by serial (as already said earlier, like in usbaudio):

$ adb get-serialno
05f5e60a0ae518e5

Or with adb-devpath to find the vid:pid:

$ adb get-devpath
usb:1-4
$ cat /sys/bus/usb/devices/1-4/id{Vendor,Product}
18d1
4ee2

Filtering by adb interfaces could also help: https://github.com/rom1v/usbaudio/blob/6f46c43b9a04249f5de61255b834dc7ebd56421c/src/aoa.c#L85-L117

Please don't do anything about it, I must rebase my old work to read the output of a process: #279 (comment)

[AlynxZhou]

@AlynxZhou I don't know if you received my e-mail this week-end (probably went into your spams, like often with gmail recipients). (If you missed it, you didn't miss a lot, I said I had no time yet.)

Gmail puts your email into junk mail directly, sad.

Anyway, here is some quick feedback.

Keyboard

I did a quick test on my devices, when I use HID (either in French or English locale) from my laptop (a French AZERTY keyboard), it injects keys as if my keyboard was QWERTY (so numbers, accented characters, some letters… are incorrect).

Hmm, I only use QWERTY keyboard so I did not test this, however, I have some thoughts about it. I suggest you run it in verbose mode, and see if the key in logs is the same as what you typed (I think it should be the same).

Then I suggest you to try to change keyboard layout in your Android IME, for example I am using Gboard, I can add QWERTY and AZERTY in Alphabet, maybe this is the problem, you have to set it manually because HID has no info about keyboard layout.

I personally only have on tweak of my keyboard that I swapped Control and Caps Lock with udev rules, and it works fine on Android (because this has been handled before SDL). I think the problem is that by default Android IME uses QWERTY as keyboard layout. But I cannot test it, I need your help.

I'm not sure how to solve this problem if we inject scancodes. Any idea?

I think this is not problem of those codes, they just forward scancodes...

Input mode selection

(Maybe a good idea is like usbaudio, if only one device with adb, use it, others ask for serial?)

I'd really prefer to be explicit. Otherwise, people will never know immediately which method is used and why/why not.

After reflection, I confirm that I would really prefer the mode to be explicit. It greatly impacts how the injection works, so it should not just be the consequence of some heuristic/fallback.

If it's explicitly selected, scrcpy should fail IMO, not fallback.

Then what should be the default option?

IMO, the default should be the Android text injection: it's the most basic, but works the same way everywhere:

* Linux, Windows and macOS

* over USB and wirelessly / TCP/IP

HID injection is better, but only over USB and only on Linux, and it might cause significant differences (hopefully for the better, but it can also impact things like the AZERTY/QWERTY issue I encounter).

So IMO it should be enabled explicitly with scrcpy --input-method=hid.

I prefer the current logic, but if you think the old way is better for default, it's OK, should be easily changed in one commit I think.

Auto device selection

For now, as we discussed, HID can only be enabled if a serial is explicitly passed.

I'm not asking you to do anything about it, I'm just listing some ideas/solutions.

We could use adb get-serialno and match by serial (as already said earlier, like in usbaudio):

$ adb get-serialno
05f5e60a0ae518e5

Or with adb-devpath to find the vid:pid:

$ adb get-devpath
usb:1-4
$ cat /sys/bus/usb/devices/1-4/id{Vendor,Product}
18d1
4ee2

Filtering by adb interfaces could also help: https://github.com/rom1v/usbaudio/blob/6f46c43b9a04249f5de61255b834dc7ebd56421c/src/aoa.c#L85-L117

Please don't do anything about it, I must rebase my old work to read the output of a process: #279 (comment)

I did nothing about auto selection, the only thing that might be related is that I added a adb_getserialno function (which cannot return serial because we cannot read output of process), and I never use it, I forget to remove it because it's harmless. It could be removed with a single commit.

It's mid night 2:00 and my brain is not so clear, my words might be confusing, please add comment if you have any questions about my reply, I'll reply after waking up 😄.

[AlynxZhou]

Actually I am not sure what a AZERTY keyboard is, when you press A on your keyboard, does it send scancode for A? Or send scancode Q (because the key is Q on QWERTY) and your system convert it into keycode A? If it sends scancode A, all things should works fine, if it sends scancode Q, Android is responsible for convert it into keycode A like what your system do. But anyway this should have nothing related to our code, it just forward scancode.

Please have a test for setting your IME to AZERTY and see if it works fine.

[rom1v]

when you press A on your keyboard, does it send scancode for A?

It generates SDL_SCANCODE_Q for physical key code (event->keysym.scancode) and SDLK_a for virtual key code (event->keysym.sym).

Changing the device keyboard layout seems to have no impact.

Maybe the layout is part of some USB descriptor? (I didn't read the specs)

[rom1v]

Maybe the layout is part of some USB descriptor?

https://www.usb.org/document-library/device-class-definition-hid-111

bCountryCode p22-23

[AlynxZhou]

Maybe the layout is part of some USB descriptor?

https://www.usb.org/document-library/device-class-definition-hid-111

bCountryCode p22-23

However AOA does not use HID descriptor...

[AlynxZhou]

when you press A on your keyboard, does it send scancode for A?

It generates SDL_SCANCODE_Q for physical key code (event->keysym.scancode) and SDLK_a for virtual key code (event->keysym.sym).

Hmm, if you got SDL_SCANCODE_Q, I think send Q to Android should be the correct behavior, because your keyboard is sending Q to your system.

Changing the device keyboard layout seems to have no impact.

What's your Android IME? Could you please tell me what you did to change it? Thanks.

Maybe the layout is part of some USB descriptor? (I didn't read the specs)

For AOA we only have Report Descriptor in documents.

I guess your keyboard is a normal QWERTY keyboard and just swap keycaps? The layout is converted by system.

Another way is that you use an OTG cable to connect your keyboard directly... my code should works the same as connecting keyboard to Android physically.

[AlynxZhou]

OK, I find where to change the physical keyboard in Android, seems not the IME, for my Galaxy S9+ is Settings -> General management -> Language and input -> Physical Keyboard -> Gboard - Multilingual typing (maybe change other IME if you don't use Gboard).

After choose "French, Azerty style", when I press Q on my QWERTY keyboard, A is shown on screen. This is not related with my Linux's keyboard layout (I am still QWERTY on my computer). You may check this.

Those settings only appear when Android detected a physical keyboard (OTG or AOA).

[rom1v]

Indeed 👍 (I don't use GBoard, but OpenBoard, but it seems independent anyway)

[AlynxZhou]

Should be in document, I am not sure whether Android can change layout automatically when you plugged in your keyboard via OTG, but seems we need to set it manually for once in AOA.

[AlynxZhou]

See new commits about remove adb_get_serialno and use legacy way as default input mode, if you still have any questions, please add your comment.

[krishtoautomate]

Hi, will this otg keyboard work for ios as well?

[rom1v]

No.

[castillofrancodamian]

How do I change the keyboard language? I already tried changing the language of the physical keyboard on Android but it still doesn't work.

[ghost]

How do I change the keyboard language? I already tried changing the language of the physical keyboard on Android but it still doesn't work.

Hi, what phone do you use? Go to settings in phone. Type "physical" in search bar. Click on physical keyboard settings. You'll see keyboard shortcuts settings. Reply to me if not working.

[castillofrancodamian]

@Channpyae It's a Nokia 2.3. I did exactly that and it's still with the English keyboard (I guess).

[ghost]

@Channpyae It's a Nokia 2.3. I did exactly that and it's still with the English keyboard (I guess).

I'm facing the same issue, now. I tried multiple methods like installing physical keyboard layouts on both my system and phone(Android 12, Samsung) but doesn't work. It seems scrcpy don't take unicode characters other than English; based on terminal outputs. Sorry, man. I can't help.

[rom1v]

What do you test exactly? scrcpy in otg mode or the current PR about uhid?

(physical keyboard settings have no impact on the default injection mode)

[castillofrancodamian]

@rom1v I use scrcpy --otg --hid-keyboard --hid-mouse .