Android support based on scrcpy

[RaiMan]

My suggestion is to start basic implementations against scrcpy in the class:
org.sikuli.script.support.Devices.Andoid.Device
which implements IScreen and IRobot - so everything for now is in one place.

Feel free, to create necessary supporting classes in the package org.sikuli.script.support.Devices.Andoid

Any comments, suggestions, ... please in this discussion issue.

[TeofilisMartisius]

Hi,

Sorry for a long delay, I was unable to work on my hobby projects.

I made some small progress. I managed to turn scrcpy into a shared library (modified meson build). Then I managed to produce a Java/JNI wrapper for scrcpy library using Bytedeco JavaCPP and invoke scrcpy from Java as a shared library.

So far nothing much works and the code is a mess, but it demonstrates the concept should work. Next thing to do is to export more functions from scrcpy library and to try to get some screen grabs from Java. I'll continue to work on this as I have time.

UPDATE. I can see the screen running form Java via JavaCPP via libscrcpy, everything starts fine. Now I need to tweak scrcpy to decode h264 video stream into my Java bitmap instead of displaying it, and I need to tweak scrcpy to take events from Java code instead of mouse on window. Then clean up the code. Good progress so far.

Teofilis

[RaiMan]

No hurry ;-)
Anyways this would be a very welcome goody for some people out there.

[TeofilisMartisius]

Ok, progress update.

I have refactored scrcpy enough to be able to hook into video feed. Scrcpy is using ffmpeg library under the hood, and ffmpeg is available on Java as well with Bytedeco mappings. Long story short, now I have a Java process that is able to invoke scrcpy and get the video feed IN JAVA, as a BufferedImage.

I guess it's time to clean up the code and to get in touch with scrcpy guys and ask them if they would consider taking my changes upstream and making scrcpy into a library. I don't want to maintain this as a fork of scrcpy forever.

Teofilis

[RaiMan]

Ok, now I finally understand your concept.

Wish you all the best, to get the people at scrcpy convinced.

[TeofilisMartisius]

Ok, just FYI, I committed my code which uses scrcpy from Java, just to give you an idea of what's possible: https://github.com/TeofilisMartisius/scrcpy-java

I'm still waiting for more detailed feedback from scrcpy team with regards to merging my PR or whatever effort is needed to turn scrcpy into a library.

Next I'll do event passing (keys, mouse) Java => scrcpy => Android. Then some performance checks. And then try to integrate it all with SikuliX. Hopefully at some point the PR will get merged with upstream scrcpy, if not I can maintain the fork for now.

[RaiMan]

Thanks. Will have a look.

Just to track your activities on scrcpy: Genymobile/scrcpy#2595

[TeofilisMartisius]

Update. I did some more work cleaning up https://github.com/TeofilisMartisius/scrcpy-java Screen capture and Mouse clicks now work (drag still doesn't, and neither does keyboard).

Next thing I'll start wrapping SikuliX IScreen around a Scrcpy session and see how that goes.

Also, I'm using native libraries and for now the only platform supported is x86_64 Linux. If this takes off, we need to support more platforms...

Also, no more feedback on upstreaming this to scrcpy.

[RaiMan]

Thanks for the update.

[TeofilisMartisius]

Ok, I wrote initial implementation of Screen and Robot for Scrcpy library. I can use SikuliX on Android with very limited set of features: https://github.com/TeofilisMartisius/SikuliX1/tree/android206

I'm having problems recognizing text. I have implemented a scrcpy version that simply replays an earlier recorded screen capture (for easier testing). When replaying https://github.com/TeofilisMartisius/scrcpy-java/blob/master/scrcpy-demo/src/main/resources/recorded.mkv it fails to find text "Hello". It looks like it does OCR the entire screen successfully (Finder.java line ~777, wordsFound does contain "Hello"). But when trying to OCR the words again in line 802, it comes out blank. I think Tesseract is having trouble recognizing the word in a cropped image...

P.S. Are there any unit/integration tests for SikuliX? I mean using a recorded videos it should be possible to implement some.

EDIT. That issue with recognizing "Hello" seems to be tesseract failing to recognize closely cropped text. It's not related to Android/scrcpy integration. I'll dig some more.

[RaiMan]

Thanks for the update.

P.S. Are there any unit/integration tests for SikuliX?

Not officially. I made some experiences with different setups based on standard unittest, but it is not part of the current repo.
... but I am open for any suggestions.

That issue with recognizing "Hello" seems to be tesseract failing to recognize closely cropped text

... my experience too: text areas need some surrounding background. Something, that can surely be optimized in my implementation.

[RaiMan]

I now have a Nexus 7 (Android 6.0.1) ready.
I have allowed to access it over USB.

How can I test your implementation?

[TeofilisMartisius]

First, I'd say run vanilla scrcpy to verify that it works.

Right. I assume you're on Linux? I only have 64bit scrcpy library bundled with the software. It should be inside the scrcpy-platform-1.0-SNAPSHOT.jar

Oh right, you'll probably need to build that from source, or else if it fails, I have to do a release and give you binaries.

Then, on SikuliX branch I would run ScrcpyScreenTry to test the SikuliX API. There are two Scrcpy implementations:

IScrcpy scrcpy = new Scrcpy(Scrcpy.defaultOptions());
which uses real scrcpy

or

IScrcpy scrcpy = new ScrcpyRecorded("recorded.mkv");
Which just replays a recorded video.

Good luck, and drop me an email if you get stuck.

Teofilis