Infer asset videos

[clydebailey]

I am trying to read a stream originating from a video file and display the inferences in android surface view. Any ideas how to implement the processor.getVideoSurfaceOutput().setSurface(surfaceTexture) function to output the inferred image to the screen?

`

static {
    // Load all native libraries needed by the app.
    System.loadLibrary("mediapipe_jni");
    System.loadLibrary("opencv_java3");
}

// {@link SurfaceTexture} where the camera-preview frames can be accessed.
private SurfaceTexture surfaceTexture;
// {@link SurfaceView} that displays the camera-preview frames processed by a MediaPipe graph.
private SurfaceView previewDisplayView;
// Creates and manages an {@link EGLContext}.
private EglManager eglManager;
// Sends camera-preview frames into a MediaPipe graph for processing, and displays the processed
// frames onto a {@link Surface}.
private FrameProcessor processor;
// Converts the GL_TEXTURE_EXTERNAL_OES texture from Android camera into a regular texture to be
// consumed by {@link FrameProcessor} and the underlying MediaPipe graph.
private ExternalTextureConverter converter;
// ApplicationInfo for retrieving metadata defined in the manifest.
private ApplicationInfo applicationInfo;
// Handles camera access via the {@link CameraX} Jetpack support library.
private CameraXPreviewHelper cameraHelper;
private AutoFitTextureView videoTexture;
private FrameLayout recorderSurface;

MediaPlayer player;
private String fileUri;

@Override
protected void onCreate(Bundle savedInstanceState) {
    super.onCreate(savedInstanceState);
    setContentView(getContentViewLayoutResId());

    try {
        applicationInfo =
                getPackageManager().getApplicationInfo(getPackageName(), PackageManager.GET_META_DATA);
    } catch (PackageManager.NameNotFoundException e) {
        Log.e(TAG, "Cannot find application info: " + e);
    }

    videoTexture = findViewById(R.id.preview);
    videoTexture.setSurfaceTextureListener(this);

    fileUri = getIntent().getStringExtra("FILE_URI");

// setupPreviewDisplayView();

    // Initialize asset manager so that MediaPipe native libraries can access the app assets, e.g.,
    // binary graphs.
    AndroidAssetUtil.initializeNativeAssetManager(this);
    eglManager = new EglManager(null);
    processor =
            new FrameProcessor(
                    this,
                    eglManager.getNativeContext(),
                    BINARY_GRAPH_NAME,
                    INPUT_VIDEO_STREAM_NAME,
                    OUTPUT_VIDEO_STREAM_NAME);
    processor
            .getVideoSurfaceOutput()
            .setFlipY(FLIP_FRAMES_VERTICALLY);

    PermissionHelper.checkAndRequestCameraPermissions(this);
    AndroidPacketCreator packetCreator = processor.getPacketCreator();
    Map<String, Packet> inputSidePackets = new HashMap<>();
    processor.addPacketCallback(
            OUTPUT_LANDMARKS_STREAM_NAME,
            (packet) -> {
                Log.e(TAG, "Received multi-hand landmarks packet.");
                Log.v(TAG, packet.toString());
                byte[] landmarksRaw = PacketGetter.getProtoBytes(packet);
                try {
                    LandmarkProto.NormalizedLandmarkList landmarks = LandmarkProto.NormalizedLandmarkList.parseFrom(landmarksRaw);
                    if (landmarks == null) {
                        Log.v(TAG, "[TS:" + packet.getTimestamp() + "] No iris landmarks.");
                        return;
                    }
                    // Note: If eye_presence is false, these landmarks are useless.
                    Log.v(
                            TAG,
                            "[TS:"
                                    + packet.getTimestamp()
                                    + "] #Landmarks for iris: "
                                    + landmarks.getLandmarkCount());
                    Log.v(TAG, getLandmarksDebugString(landmarks));
                } catch (InvalidProtocolBufferException e) {
                    Log.e(TAG, "Couldn't Exception received - " + e);
                    return;
                }
            });

}

@Override
protected void onResume() {
    super.onResume();
}

private void mediaPlay(SurfaceTexture surfaceTexture){
    Surface surface = new Surface(surfaceTexture);
    Uri filUri = Uri.fromFile(new File(this.fileUri));
    player = MediaPlayer.create(this,filUri);
    player.setSurface(surface);
    player.setOnCompletionListener(new MediaPlayer.OnCompletionListener() {
        @Override
        public void onCompletion(MediaPlayer mp) {
            finishWithSuccess();
        }
    });

    player.start();
}

private void finishWithSuccess(){
    Intent intent = new Intent();
    setResult(RESULT_OK,intent);
    finish();
}

protected int getContentViewLayoutResId() {
    return R.layout.activity_pose_estimation;
}

private static String getLandmarksDebugString(LandmarkProto.NormalizedLandmarkList landmarks) {
    int landmarkIndex = 0;
    String landmarksString = "";
    for (LandmarkProto.NormalizedLandmark landmark : landmarks.getLandmarkList()) {
        landmarksString +=
                "\t\tLandmark["
                        + landmarkIndex
                        + "]: ("
                        + landmark.getX()
                        + ", "
                        + landmark.getY()
                        + ", "
                        + landmark.getZ()
                        + ")\n";
        ++landmarkIndex;
    }
    return landmarksString;
}



@Override
public void onSurfaceTextureAvailable(SurfaceTexture surfaceTexture, int width, int height) {

    mediaPlay(surfaceTexture);
    converter = new ExternalTextureConverter(eglManager.getContext(), 2);
    converter.setFlipY(FLIP_FRAMES_VERTICALLY);
    converter.setConsumer(processor);
    converter.setSurfaceTextureAndAttachToGLContext(surfaceTexture,1080,1920);

}

@Override
public void onSurfaceTextureSizeChanged(SurfaceTexture surfaceTexture, int width, int height) {

}

@Override
public boolean onSurfaceTextureDestroyed(SurfaceTexture surfaceTexture) {

    return false;
}

@Override
public void onSurfaceTextureUpdated(SurfaceTexture surfaceTexture) {


}`

[clydebailey]

I am trying to use onNextFrame to infer on a javacv frame. Is there a conversion for mat/bitmap/javacv frame to mediapipe texture frame?

[clydebailey]

Used this reference to solve my problem.
#310
This comment in specific #310 (comment)

Started off with using javacv FFmpegFrameGrabber. Grab each frame and convert to bitmap then convert the bitmap to textureframe and feed it to the mediapipe processor.

Thanks Mediapipe Team

[google-ml-butler]

Are you satisfied with the resolution of your issue?
Yes
No

[clydebailey]

Hey guys! I am still perplexed with an issue and reopening this thread.

Problem Statement: I am running mediapipe pose on a bunch of bitmap images but I am getting the same inference for each image.

How:

1. Grabbed images from video using javacv ffmpeg grabber and store bitmaps in a hashmap with the frame number as a key
   grabber = new FFmpegFrameGrabber(fileUri); frame = grabber.grabImage(); Bitmap bitmap = converterToBitmap.convert(frame);

2. Setup Texture Renderer
   public void setupRenderer(){ renderer = new TextureRenderer(); renderer.setFlipY(FLIP_FRAMES_VERTICALLY); renderer.setup(); }

3. Send bitmaps for processing
   renderer.render(outputFrame.getTextureName(), bitmap); outputFrame.setTimestamp(timestamp); outputFrame.setInUse(); processor.onNewFrame(outputFrame);

I have cross verified whether I am sending the right bitmap image to the processor and it looks fine. Is there a queue for the processor that needs to be cleared or something. I have also tried to remove the packets by calling packet.release() method

@kuaashish can you help out here?? TIA

[kuaashish]

Hi @jiuqiant,
Could you please look into this issue here.

[kuaashish]

Hello @clydebailey,
We are upgrading the MediaPipe Legacy Solutions to new MediaPipe solutions However, the libraries, documentation, and source code for all the MediapPipe Legacy Solutions will continue to be available in our GitHub repository and through library distribution services, such as Maven and NPM.

You can continue to use those legacy solutions in your applications if you choose. Though, we would request you to check new MediaPipe solutions which can help you more easily build and customize ML solutions for your applications. These new solutions will provide a superset of capabilities available in the legacy solutions. Thank you

[github-actions]

This issue has been marked stale because it has no recent activity since 7 days. It will be closed if no further activity occurs. Thank you.

[github-actions]

This issue was closed due to lack of activity after being marked stale for past 7 days.

[google-ml-butler]

Are you satisfied with the resolution of your issue?
Yes
No