React Native - Face Landmark Detection not working with Tensor3D

[Patel-Preet]

System information

• Have I written custom code (as opposed to using a stock example script provided in TensorFlow.js): Combination
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Windows 11
• Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: Samsung Galaxy
• TensorFlow.js installed from (npm or script link): npm
• TensorFlow.js version (use command below): "^4.10.0"
• Browser version: NA
• Tensorflow.js Converter Version: "^3.20.0"

Deps:
"react": "18.2.0",
"react-native": "0.72.3",
"@tensorflow-models/face-landmarks-detection": "^1.0.5",
"@tensorflow/tfjs": "^4.10.0",
"@tensorflow/tfjs-core": "^3.20.0",
"@tensorflow/tfjs-react-native": "^0.8.0",
"@tensorflow/tfjs-backend-webgl": "^3.20.0",
"expo-gl": "^13.0.1",

Describe the current behavior
According to the docs https://github.com/tensorflow/tfjs-models/tree/master/face-landmarks-detection/src/tfjs the estimateFaces takes Tensor3D as paramater but when executed it throws an error
Error: pixels passed to tf.browser.fromPixels() must be either an HTMLVideoElement, HTMLImageElement, HTMLCanvasElement, ImageData in browser, or OffscreenCanvas, ImageData in webworker or {data: Uint32Array, width: number, height: number}, but was Tensor

#3186 (comment)
#4530 (comment)
#7529
https://js.tensorflow.org/api_react_native/0.5.0/#decodeJpeg (Returns Tensor3D)

Describe the expected behavior
It should have returned the face landmarks.

Standalone code to reproduce the issue

import * as tf from '@tensorflow/tfjs';
import '@tensorflow/tfjs-core';
import * as faceLandmarksDetection from '@tensorflow-models/face-landmarks-detection';
import {
  bundleResourceIO,
  decodeJpeg ,
} from '@tensorflow/tfjs-react-native';
import 'expo-gl';
// import '@tensorflow/tfjs-backend-webgl';
import * as FileSystem from 'expo-file-system';

 const createFaceMesh = async (imageURI) => {
        //https://github.com/tensorflow/tfjs/issues/4530#issuecomment-760301595
        // tf.removeBackend('cpu')
        // await tf.setBackend('webgl')
        await tf.ready();
        const model = faceLandmarksDetection.SupportedModels.MediaPipeFaceMesh;
        const modelJson = require('../../assets/model.json');
        const modelWeights = require('../../assets/group1-shard1of1.bin');
        const detectorConfig = {
          runtime: 'tfjs',
          landmarkModelUrl : bundleResourceIO(modelJson, modelWeights)
        };
        let detector = await faceLandmarksDetection.createDetector(model, detectorConfig);
    
        const estimationConfig = {flipHorizontal: false, staticImageMode: true};

        const imgB64 = await FileSystem.readAsStringAsync(imageURI, {
          encoding: FileSystem.EncodingType.Base64,
        });
        const imgBuffer = tf.util.encodeString(imgB64, 'base64').buffer;
        const raw = new Uint8Array(imgBuffer)  
        const imageTensor = decodeJpeg(raw);

        console.log(imageTensor)
      
        let faceMesh = await detector.estimateFaces( imageTensor, estimationConfig);
        console.log(faceMesh);
    }

Other info / logs Include any logs or source code that would be helpful to
diagnose the problem. If including tracebacks, please include the full
traceback. Large logs and files should be attached.

 WARN  cpu backend was already registered. Reusing existing backend factory.
 WARN  The kernel '_FusedMatMul' for backend 'cpu' is already registered
 WARN  The kernel 'Abs' for backend 'cpu' is already registered
 WARN  The kernel 'Acos' for backend 'cpu' is already registered
 WARN  The kernel 'Acosh' for backend 'cpu' is already registered
 WARN  The kernel 'Add' for backend 'cpu' is already registered
 WARN  The kernel 'AddN' for backend 'cpu' is already registered
 WARN  The kernel 'All' for backend 'cpu' is already registered
 WARN  The kernel 'Any' for backend 'cpu' is already registered
 WARN  The kernel 'ArgMax' for backend 'cpu' is already registered
 WARN  The kernel 'ArgMin' for backend 'cpu' is already registered
 WARN  The kernel 'Asin' for backend 'cpu' is already registered
 WARN  The kernel 'Asinh' for backend 'cpu' is already registered
 WARN  The kernel 'Atan' for backend 'cpu' is already registered
 WARN  The kernel 'Atan2' for backend 'cpu' is already registered
 WARN  The kernel 'Atanh' for backend 'cpu' is already registered
 WARN  The kernel 'AvgPool' for backend 'cpu' is already registered
 WARN  The kernel 'AvgPool3D' for backend 'cpu' is already registered
 WARN  The kernel 'AvgPool3DGrad' for backend 'cpu' is already registered
 WARN  The kernel 'AvgPoolGrad' for backend 'cpu' is already registered
 WARN  The kernel 'BatchMatMul' for backend 'cpu' is already registered
 WARN  The kernel 'FusedBatchNorm' for backend 'cpu' is already registered
 WARN  The kernel 'BatchToSpaceND' for backend 'cpu' is already registered
 WARN  The kernel 'Bincount' for backend 'cpu' is already registered
 WARN  The kernel 'BroadcastArgs' for backend 'cpu' is already registered
 WARN  The kernel 'Cast' for backend 'cpu' is already registered
 WARN  The kernel 'Ceil' for backend 'cpu' is already registered
 WARN  The kernel 'ClipByValue' for backend 'cpu' is already registered
 WARN  The kernel 'Complex' for backend 'cpu' is already registered
 WARN  The kernel 'ComplexAbs' for backend 'cpu' is already registered
 WARN  The kernel 'Concat' for backend 'cpu' is already registered
 WARN  The kernel 'Conv2D' for backend 'cpu' is already registered
 WARN  The kernel 'Conv2DBackpropFilter' for backend 'cpu' is already registered
 WARN  The kernel 'Conv2DBackpropInput' for backend 'cpu' is already registered
 WARN  The kernel 'Conv3D' for backend 'cpu' is already registered
 WARN  The kernel 'Conv3DBackpropFilterV2' for backend 'cpu' is already registered
 WARN  The kernel 'Conv3DBackpropInputV2' for backend 'cpu' is already registered
 WARN  The kernel 'Cos' for backend 'cpu' is already registered
 WARN  The kernel 'Cosh' for backend 'cpu' is already registered
 WARN  The kernel 'CropAndResize' for backend 'cpu' is already registered
 WARN  The kernel 'Cumprod' for backend 'cpu' is already registered
 WARN  The kernel 'Cumsum' for backend 'cpu' is already registered
 WARN  The kernel 'DenseBincount' for backend 'cpu' is already registered
 WARN  The kernel 'DepthToSpace' for backend 'cpu' is already registered
 WARN  The kernel 'DepthwiseConv2dNative' for backend 'cpu' is already registered
 WARN  The kernel 'DepthwiseConv2dNativeBackpropFilter' for backend 'cpu' is already registered
 WARN  The kernel 'DepthwiseConv2dNativeBackpropInput' for backend 'cpu' is already registered
 WARN  The kernel 'Diag' for backend 'cpu' is already registered
 WARN  The kernel 'Dilation2D' for backend 'cpu' is already registered
 WARN  The kernel 'Dilation2DBackpropFilter' for backend 'cpu' is already registered
 WARN  The kernel 'Dilation2DBackpropInput' for backend 'cpu' is already registered
 WARN  The kernel 'Einsum' for backend 'cpu' is already registered
 WARN  The kernel 'Elu' for backend 'cpu' is already registered
 WARN  The kernel 'EluGrad' for backend 'cpu' is already registered
 WARN  The kernel 'Equal' for backend 'cpu' is already registered
 WARN  The kernel 'Erf' for backend 'cpu' is already registered
 WARN  The kernel 'Exp' for backend 'cpu' is already registered
 WARN  The kernel 'ExpandDims' for backend 'cpu' is already registered
 WARN  The kernel 'Expm1' for backend 'cpu' is already registered
 WARN  The kernel 'FFT' for backend 'cpu' is already registered
 WARN  The kernel 'Fill' for backend 'cpu' is already registered
 WARN  The kernel 'FlipLeftRight' for backend 'cpu' is already registered
 WARN  The kernel 'Floor' for backend 'cpu' is already registered
 WARN  The kernel 'FloorDiv' for backend 'cpu' is already registered
 WARN  The kernel 'FusedConv2D' for backend 'cpu' is already registered
 WARN  The kernel 'FusedDepthwiseConv2D' for backend 'cpu' is already registered
 WARN  The kernel 'GatherNd' for backend 'cpu' is already registered
 WARN  The kernel 'GatherV2' for backend 'cpu' is already registered
 WARN  The kernel 'Greater' for backend 'cpu' is already registered
 WARN  The kernel 'GreaterEqual' for backend 'cpu' is already registered
 WARN  The kernel 'Identity' for backend 'cpu' is already registered
 WARN  The kernel 'IFFT' for backend 'cpu' is already registered
 WARN  The kernel 'Imag' for backend 'cpu' is already registered
 WARN  The kernel 'IsFinite' for backend 'cpu' is already registered
 WARN  The kernel 'IsInf' for backend 'cpu' is already registered
 WARN  The kernel 'IsNan' for backend 'cpu' is already registered
 WARN  The kernel 'LeakyRelu' for backend 'cpu' is already registered
 WARN  The kernel 'Less' for backend 'cpu' is already registered
 WARN  The kernel 'LessEqual' for backend 'cpu' is already registered
 WARN  The kernel 'LinSpace' for backend 'cpu' is already registered
 WARN  The kernel 'Log' for backend 'cpu' is already registered
 WARN  The kernel 'Log1p' for backend 'cpu' is already registered
 WARN  The kernel 'LogicalAnd' for backend 'cpu' is already registered
 WARN  The kernel 'LogicalNot' for backend 'cpu' is already registered
 WARN  The kernel 'LogicalOr' for backend 'cpu' is already registered
 WARN  The kernel 'LRN' for backend 'cpu' is already registered
 WARN  The kernel 'LRNGrad' for backend 'cpu' is already registered
 WARN  The kernel 'Max' for backend 'cpu' is already registered
 WARN  The kernel 'Maximum' for backend 'cpu' is already registered
 WARN  The kernel 'MaxPool' for backend 'cpu' is already registered
 WARN  The kernel 'MaxPool3D' for backend 'cpu' is already registered
 WARN  The kernel 'MaxPool3DGrad' for backend 'cpu' is already registered
 WARN  The kernel 'MaxPoolGrad' for backend 'cpu' is already registered
 WARN  The kernel 'MaxPoolWithArgmax' for backend 'cpu' is already registered
 WARN  The kernel 'Mean' for backend 'cpu' is already registered
 WARN  The kernel 'Min' for backend 'cpu' is already registered
 WARN  The kernel 'Minimum' for backend 'cpu' is already registered
 WARN  The kernel 'MirrorPad' for backend 'cpu' is already registered
 WARN  The kernel 'Mod' for backend 'cpu' is already registered
 WARN  The kernel 'Multinomial' for backend 'cpu' is already registered
 WARN  The kernel 'Multiply' for backend 'cpu' is already registered
 WARN  The kernel 'Neg' for backend 'cpu' is already registered
 WARN  The kernel 'NonMaxSuppressionV3' for backend 'cpu' is already registered
 WARN  The kernel 'NonMaxSuppressionV4' for backend 'cpu' is already registered
 WARN  The kernel 'NonMaxSuppressionV5' for backend 'cpu' is already registered
 WARN  The kernel 'NotEqual' for backend 'cpu' is already registered
 WARN  The kernel 'OneHot' for backend 'cpu' is already registered
 WARN  The kernel 'OnesLike' for backend 'cpu' is already registered
 WARN  The kernel 'Pack' for backend 'cpu' is already registered
 WARN  The kernel 'PadV2' for backend 'cpu' is already registered
 WARN  The kernel 'Pow' for backend 'cpu' is already registered
 WARN  The kernel 'Prelu' for backend 'cpu' is already registered
 WARN  The kernel 'Prod' for backend 'cpu' is already registered
 WARN  The kernel 'RaggedTensorToTensor' for backend 'cpu' is already registered
 WARN  The kernel 'Range' for backend 'cpu' is already registered
 WARN  The kernel 'Real' for backend 'cpu' is already registered
 WARN  The kernel 'RealDiv' for backend 'cpu' is already registered
 WARN  The kernel 'Reciprocal' for backend 'cpu' is already registered
 WARN  The kernel 'Relu' for backend 'cpu' is already registered
 WARN  The kernel 'Relu6' for backend 'cpu' is already registered
 WARN  The kernel 'Reshape' for backend 'cpu' is already registered
 WARN  The kernel 'ResizeBilinear' for backend 'cpu' is already registered
 WARN  The kernel 'ResizeBilinearGrad' for backend 'cpu' is already registered
 WARN  The kernel 'ResizeNearestNeighbor' for backend 'cpu' is already registered
 WARN  The kernel 'ResizeNearestNeighborGrad' for backend 'cpu' is already registered
 WARN  The kernel 'Reverse' for backend 'cpu' is already registered
 WARN  The kernel 'RotateWithOffset' for backend 'cpu' is already registered
 WARN  The kernel 'Round' for backend 'cpu' is already registered
 WARN  The kernel 'Rsqrt' for backend 'cpu' is already registered
 WARN  The kernel 'ScatterNd' for backend 'cpu' is already registered
 WARN  The kernel 'SearchSorted' for backend 'cpu' is already registered
 WARN  The kernel 'Select' for backend 'cpu' is already registered
 WARN  The kernel 'Selu' for backend 'cpu' is already registered
 WARN  The kernel 'Sigmoid' for backend 'cpu' is already registered
 WARN  The kernel 'Sign' for backend 'cpu' is already registered
 WARN  The kernel 'Sin' for backend 'cpu' is already registered
 WARN  The kernel 'Sinh' for backend 'cpu' is already registered
 WARN  The kernel 'Slice' for backend 'cpu' is already registered
 WARN  The kernel 'Softmax' for backend 'cpu' is already registered
 WARN  The kernel 'Softplus' for backend 'cpu' is already registered
 WARN  The kernel 'SpaceToBatchND' for backend 'cpu' is already registered
 WARN  The kernel 'SparseFillEmptyRows' for backend 'cpu' is already registered
 WARN  The kernel 'SparseReshape' for backend 'cpu' is already registered
 WARN  The kernel 'SparseSegmentMean' for backend 'cpu' is already registered
 WARN  The kernel 'SparseSegmentSum' for backend 'cpu' is already registered
 WARN  The kernel 'SparseToDense' for backend 'cpu' is already registered
 WARN  The kernel 'SplitV' for backend 'cpu' is already registered
 WARN  The kernel 'Sqrt' for backend 'cpu' is already registered
 WARN  The kernel 'Square' for backend 'cpu' is already registered
 WARN  The kernel 'SquaredDifference' for backend 'cpu' is already registered
 WARN  The kernel 'Step' for backend 'cpu' is already registered
 WARN  The kernel 'StridedSlice' for backend 'cpu' is already registered
 WARN  The kernel 'StringNGrams' for backend 'cpu' is already registered
 WARN  The kernel 'StringSplit' for backend 'cpu' is already registered
 WARN  The kernel 'StringToHashBucketFast' for backend 'cpu' is already registered
 WARN  The kernel 'Sub' for backend 'cpu' is already registered
 WARN  The kernel 'Sum' for backend 'cpu' is already registered
 WARN  The kernel 'Tan' for backend 'cpu' is already registered
 WARN  The kernel 'Tanh' for backend 'cpu' is already registered
 WARN  The kernel 'Tile' for backend 'cpu' is already registered
 WARN  The kernel 'TopK' for backend 'cpu' is already registered
 WARN  The kernel 'Transform' for backend 'cpu' is already registered
 WARN  The kernel 'Transpose' for backend 'cpu' is already registered
 WARN  The kernel 'Unique' for backend 'cpu' is already registered
 WARN  The kernel 'Unpack' for backend 'cpu' is already registered
 WARN  The kernel 'UnsortedSegmentSum' for backend 'cpu' is already registered
 WARN  The kernel 'ZerosLike' for backend 'cpu' is already registered
 LOG  {"dataId": {"id": 216}, "dtype": "int32", "id": 216, "isDisposedInternal": false, "kept": false, "rankType": "3", "shape": [2208, 2944, 3], "size": 19501056, "strides": [8832, 3]}
 WARN  Possible Unhandled Promise Rejection (id: 0):
Error: pixels passed to tf.browser.fromPixels() must be either an HTMLVideoElement, HTMLImageElement, HTMLCanvasElement, ImageData in browser, or OffscreenCanvas, ImageData in webworker or {data: Uint32Array, width: number, height: number}, but was Tensor

[gaikwadrahul8]

Hi, @Patel-Preet

Thank you for bringing this issue to our attention, if possible could you please help me with your package.json file and model zip file so I'll try to replicate the same issue from my end ?

It would be great if you guide me with steps to replicate the same issue from my end. Thank you!

[Patel-Preet]

Hi @gaikwadrahul8,

Sorry for the delayed response. I have attached the package.json and model zip file as requested. The above code should be enough to replicate the issue. However, let me know if you need additional details.

package.zip

tfjs-model_face_landmarks_detection_face_mesh_1.zip

[Patel-Preet]

After updating the deps to 4.10.0 the new errors is
Error: Argument 'x' passed to 'conv2d' must be a Tensor or TensorLike, but got 'Tensor' at convertToTensor
There are a few comments addressing this, mainly resolved by using JSC. #7056
I want to know @gaikwadrahul8 , Is there any other way to resolve this without disabling hermes?

[paradite]

After updating the deps to 4.10.0 the new errors is Error: Argument 'x' passed to 'conv2d' must be a Tensor or TensorLike, but got 'Tensor' at convertToTensor There are a few comments addressing this, mainly resolved by using JSC. #7056 I want to know @gaikwadrahul8 , Is there any other way to resolve this without disabling hermes?

Hi, I have a walkaround for the issue here: #6526 (comment)

I have been using it in production for a while now so it is working. I am also working on a potential fix for this issue.

[gaikwadrahul8]

Hi, @Patel-Preet

Apologize for the delayed response and could you please refer above comment from @paradite with this workaround and I see a PR #7947 which is submitted by @paradite which may take care of this issue after merging that PR if I'm not wrong. Thank you!

[github-actions]

This issue has been marked stale because it has no recent activity since 7 days. It will be closed if no further activity occurs. Thank you.

[github-actions]

This issue was closed due to lack of activity after being marked stale for past 7 days.

[google-ml-butler]

Are you satisfied with the resolution of your issue?
Yes
No