Face Landmark Detection return empty array

[Lonoyaa]

Please make sure that this is a bug. As per our
GitHub Policy,
we only address code/doc bugs, performance issues, feature requests and
build/installation issues on GitHub. tag:bug_template

System information

• Have I written custom code (as opposed to using a stock example script provided in TensorFlow.js): Yes
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): ubuntu 22.04.1
• Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: Huawei Nova 4
• TensorFlow.js installed from (npm or script link): yarn add @tensorflow/tfjs
• TensorFlow.js version (use command below):3.11.0
• Browser version:-
• Tensorflow.js Converter Version:3.11.0

Describe the current behavior
FaceLandmarkDetection.estimateFace(inputTensor3D) return empty array
Describe the expected behavior
estimateFace return face array contain box and keypoints
Standalone code to reproduce the issue

"dependencies": {

"@mediapipe/face_detection": "^0.4.1646425229",
"@mediapipe/face_mesh": "^0.4.1633559619",
"@react-native-async-storage/async-storage": "^1.15.5",
"@shopify/react-native-skia": "0.1.175",
"@tensorflow-models/face-detection": "^1.0.1",
"@tensorflow-models/face-landmarks-detection": "^1.0.2",
"@tensorflow/tfjs": "^3.11.0",
"@tensorflow/tfjs-backend-cpu": "~3.11.0",
"@tensorflow/tfjs-backend-webgl": "~3.11.0",
"@tensorflow/tfjs-converter": "^3.11.0",
"@tensorflow/tfjs-core": "3.11.0",
"@tensorflow/tfjs-react-native": "^0.8.0",
"@types/react-native": "^0.71.3",
"deprecated-react-native-prop-types": "^4.0.0",
"expo-asset": "^7.0.0",
"expo-camera": "^13.2.1",
"expo-gl": "^10.2.1",
"expo-gl-cpp": "10.4.1",
"expo-image-loader": "2.1.1",
"react": "18.2.0",
"react-native": "0.71.4",
"react-native-canvas": "^0.1.38",
"react-native-fs": "^2.14.1",
"react-native-gesture-handler": "^2.9.0",
"react-native-reanimated": "2",
"react-native-unimodules": "^0.13.3",
"react-native-vision-camera": "^2.15.4",
"react-native-webview": "^11.26.1",
"unimodules-barcode-scanner-interface": "^6.1.0"

},"

const image = require('./assets/Face.jpeg');
const imageAssetPath = Image.resolveAssetSource(image);
const response = await fetch(imageAssetPath.uri, {}, { isBinary: true });
const imageDataArrayBuffer = await response.arrayBuffer();
const imageData1 = new Uint8Array(imageDataArrayBuffer);
const imageTensor1 = decodeJpeg(imageData1);
const test = await modelReady?.estimateFaces(imageTensor1)
console.log('test0',test)

Other info / logs Include any logs or source code that would be helpful to
diagnose the problem. If including tracebacks, please include the full
traceback. Large logs and files should be attached.

At the top, I call tf.setBackend('cpu') before tf.ready(), not sure if it's related

[google-ml-butler]

Are you satisfied with the resolution of your issue?
Yes
No

[Lonoyaa]

I rebuild the project with expo and manage to use rn-webgl as backend then it sucesses to return keypoint array. If I switch to cpu backend then it doesn't work.

So, it's not possible to use cpu backend with react native (managed by react-native-cli)? Or is there any advices?

[gaikwadrahul8]

Hi, @Lonoyaa

Apologize for the delayed response and It seems like Face Landmarks Detection pre-trained model will work on tfjs-backend-webgl as runtime and I see in documentation it has mentioned, In the runtime-backend dropdown, choose 'tfjs-webgl' here and in the peer dependencies of face-landmarks-detection it has used below script tag so I'm not sure at the moment whether is it possible to use cpu backend with react native but we'll update you soon. Thank you!

CC :@Linchenn

Via script tags:

<!-- Require the peer dependencies of face-landmarks-detection. -->
<script src="https://cdn.jsdelivr.net/npm/@tensorflow/tfjs-core"></script>
<script src="https://cdn.jsdelivr.net/npm/@tensorflow/tfjs-converter"></script>

<!-- You must explicitly require a TF.js backend if you're not using the TF.js union bundle. -->
<script src="https://cdn.jsdelivr.net/npm/@tensorflow/tfjs-backend-webgl"></script>

<script src="https://cdn.jsdelivr.net/npm/@tensorflow-models/face-detection"></script>

<script src="https://cdn.jsdelivr.net/npm/@tensorflow-models/face-landmarks-detection"></script>

Via npm:

yarn add @tensorflow/tfjs-core, @tensorflow/tfjs-converter
yarn add @tensorflow/tfjs-backend-webgl
yarn add @tensorflow-models/face-detection
yarn add @tensorflow-models/face-landmarks-detection

[Lonoyaa]

Hi, @gaikwadrahul8
Thank you for your answer, it works after I change the backend to rn-webgl.

[gaikwadrahul8]

Hi, @Lonoyaa

Apologize for the delayed response and you're welcome, Good to hear that your issue got resolved so please feel free to close this issue if this issue got resolved for you. Thank you!

[google-ml-butler]

Are you satisfied with the resolution of your issue?
Yes
No