[Feature Request] Camera frames access by native extension

[itlancer]

Feature Description

AIR should provide some way to get camera frames by native extension.
In many cases for image processing, computer vision and OpenCV usage you need fastest way to get "image bytes" from camera. Most of such operations should be done by native extensions. And getting frames from AS3 Camera and then pass it to native extension, then convert to bytes and so on cause huge performance overhead. Especially for low-performance Android devices.
Moreover Camera usage with AS3 side for Android cause terrible performance: #1980
May be it related that AIR still uses legacy Android Camera API https://developer.android.com/guide/topics/media/camera, not CameraX https://developer.android.com/training/camerax or Camera2 https://developer.android.com/reference/android/hardware/camera2/package-summary.

If we get some way to get frames from camera by native extension it could increase overall performance for such applications.

For example, with Android native extension it could be new callback "on new image availability" which available through FlashRuntimeExtensions. With such callback we could get https://developer.android.com/reference/android/media/Image or frame YUV_420_888 bytes.

May be access to camera object from native side also could be useful. In such way we could also configure other camera parameters like torch/flash, auto focus and other. https://developer.android.com/reference/android/hardware/camera2/CameraManager

Related issues:
#1895
#1348
#1344

Known Workarounds

Write own native extension for camera usage.

[ajwfrost]

Hi

Just reviewing this one, based on the other comments/updates made to linked issues recently .. and am curious: if you're looking to get hold of camera frames from within an ANE, then why do you not just directly request the camera via the ANE i.e. use the Android/Java code in the ANE to take a picture or kick off a video recording?

thanks

[itlancer]

@ajwfrost
Firstly let me describe my use case.
I need to display on screen camera view, process camera images (by native extension) and overlay some graphics over faces (such as masks). All in real-time.

Sometimes we are working with camera just from native extension (without AIR Camera).
But there is a challenge to display camera view from native extension with other DisplayList and Stage3D. Especially on Android (performance issues) and using BitmapData transferring from ANE->AIR. We trying to optimize it by using Stage3D and passing VideoTexture from native extension to it. But still have some limitations (and it could works only for Android right now).

Basically all these issues around "how to improve performance?".

1. Camera usage cause terrible performance for Android: [Android] Camera usage cause terrible performance #1980
2. We need to make ARGB -> RGB conversion for camera images bytes. This feature could help: [Feature Request] Getting RGB (not ARGB) pixels from Camera #3048
3. Camera access from Worker also could help. Right now it impossible.
4. Working with native extension from Worker could help. Right now it impossible.
5. Overlay something over StageWebView and other "native" layers could help. "Combining" Stage3D/DisplayList and "native" render.
   Discussed here: Edge-based WebView2 - Details please #1170 (comment)