Video inference using NodeJS #108
Replies: 6 comments 19 replies
-
|
That's exactly why I don't explicitly enable video optimizations just for if (input && this.config.videoOptimized && (
(typeof HTMLImageElement !== 'undefined' && input instanceof HTMLImageElement)
|| (typeof Image !== 'undefined' && input instanceof Image)
|| (typeof Image !== 'undefined' && input instanceof ImageData)
|| (typeof ImageBitmap !== 'undefined' && image instanceof ImageBitmap))
) {
log('disabling video optimization');
previousVideoOptimized = this.config.videoOptimized;
this.config.videoOptimized = false;
}so if you parse the video stream and read frames in |
Beta Was this translation helpful? Give feedback.
-
That's great! Do you think it'd be helpful if I create a demo of this and create a PR. |
Beta Was this translation helpful? Give feedback.
-
go for it! the only thing i'm worried about extra dependencies - you need a library to parse video in |
Beta Was this translation helpful? Give feedback.
-
|
Hey, could you point me to a library to do this? I'm finding it surprisingly hard to find one. node-webcam library as of now only captures one image, camera-capture library gives me an unreadable stream |
Beta Was this translation helpful? Give feedback.
-
|
accessing webcam from
so lets look at
having said all that, i actually don't have an environment with nodejs and webcam, so this is written from the head without ANY testing - take it a hint more than as a solution! const util = require('util');
const nodeWebCam = require('node-webcam');
const tf = require('@tensorflow/tfjs-node');
const Human = require('@vladmandic/human/dist/human.node.js').default;
// options for node-webcam
const optionsCamera = {
callbackReturn: 'buffer', // this means whatever `fswebcam` writes to disk, no additional processing so it's fastest
saveShots: false, // don't save processed frame to disk, note that temp file is still created by fswebcam thus recommendation for tmpfs
};
// options for human
const optionsHuman = {
backend: 'tensorflow',
modelBasePath: 'file://node_modules/@vladmandic/human/models/',
};
const camera = nodeWebCam.create(optionsCamera);
const capture = util.promisify(camera.capture);
const human = new Human(optionsHuman);
const results = [];
const buffer2tensor = human.tf.tidy((buffer) => {
const decode = human.tf.node.decodeImage(buffer, 3);
let expand;
if (decode.shape[2] === 4) { // input is in rgba format, need to convert to rgb
const channels = human.tf.split(decode, 4, 2); // tf.split(tensor, 4, 2); // split rgba to channels
const rgb = human.tf.stack([channels[0], channels[1], channels[2]], 2); // stack channels back to rgb and ignore alpha
expand = human.tf.reshape(rgb, [1, decode.shape[0], decode.shape[1], 3]); // move extra dim from the end of tensor and use it as batch number instead
} else {
expand = human.tf.expandDims(decode, 0); // inpur ia rgb so use as-is
}
const cast = human.tf.cast(expand, 'float32');
return cast;
});
async function process() {
// trigger next frame every 5 sec
// triggered here before actual capture and detection since we assume it will complete in less than 5sec
// so it's as close as possible to real 5sec and not 5sec + detection time
// if there is a chance of race scenario where detection takes longer than loop trigger, then trigger should be at the end of the function instead
setTimeout(() => process(), 5000);
const buffer = await capture(); // gets the (default) jpeg data from from webcam
const tensor = buffer2tensor(buffer); // create tensor from image buffer
const res = await human.detect(tensor); // run detection
// do whatever here with the res
// or just append it to results array that will contain all processed results over time
results.push(res);
// alternatively to triggering every 5sec sec, simply trigger next frame as fast as possible
// setImmediate(() => process());
}
console.log('Human:', human.version);
console.log('TFJS:', tf.version_core);
process();note#1: if it were up to me, i'd look into using software that can has broad format and device support and then re-broadcast stream in a well-known format (not write to disk), for example note#2: for a typical IP camera (not WebCam) that uses RTSP protocol (such as most security cameras), I actually wrote a utility that takes that reads stream from IP camera and creates WebRTC stream that Human can connect to (I've added WebRTC support recently) |
Beta Was this translation helpful? Give feedback.
-
|
Hey, apologies for the late reply and thank you for the detailed answer! I was not able to get the node-webcam working continuously. The webcam would turn off and turn on between each frame, seriously affecting the fps. I will look into the vlc solution and get back. Could you help me with some hints as to how to decode the stream from nodeJS?. Can I use the buffer2tensor method you have written in your solution for each frame? That is great. As of now, I do need it for webcam, but that's good to know ! |
Beta Was this translation helpful? Give feedback.
-
|
more i think about it, it's still a lot of work to decode actual stream and continously output frames it might be easier to use same so you could:
then in the nodejs:
processing should also set a this is all a theory since i don't have a system where i can play with |
Beta Was this translation helpful? Give feedback.
-
|
Sounds good!. I will try this out in the coming week!. Thank you for the detailed solution |
Beta Was this translation helpful? Give feedback.
-
|
Hey, so turns out fswebcam's --loop takes only intervals in seconds. So using |
Beta Was this translation helpful? Give feedback.
-
|
does regarding other tools, |
Beta Was this translation helpful? Give feedback.
-
|
Unfortunately, it doesn't. I'll raise a request for the feature on the repository :) Oh yes, I was following that lead last week. Will check it out. Thank you! |
Beta Was this translation helpful? Give feedback.
-
|
Take a look at both options: |
Beta Was this translation helpful? Give feedback.
-
|
I think the challenge will be finding a way to get continuous frames instead of a h264/h265 encoded stream. I'll try to find out if there is a way to do that |
Beta Was this translation helpful? Give feedback.
-
|
yup. i still think that worst case scenario, you can use |
Beta Was this translation helpful? Give feedback.
-
|
i just did a quick prototype using https://github.com/vladmandic/human/blob/main/demo/node-video.js main trick was to use motion jpeg as output format which is then easily parsed for frame start/end markers which gives jpeg per each frame. |
Beta Was this translation helpful? Give feedback.
-
|
Great, looks good. I can just swap in I also had a few more questions I was hoping you could help with!
|
Beta Was this translation helpful? Give feedback.
-
Rotation is anyhow disabled in NodeJS due to missing function
Correct. And none,
for multiple faces, the client demo also, behind the scenes, match function is just a loop around |
Beta Was this translation helpful? Give feedback.
-
|
btw, i'm still fine-tuning descriptor precision, but it depends a lot on face image pre-processing, so if you want to contribute and experiment on finding best options, take a look at |
Beta Was this translation helpful? Give feedback.
-
|
Oh yeah, I was reading this https://github.com/vladmandic/human/wiki/Embedding#face-image-pre-processing. I don't have any expertise but I will try my best to look into this :). Can I ask what ideas you were considering exploring? |
Beta Was this translation helpful? Give feedback.
-
|
on input image, few examples that are already in the code, but disabled as i need to test all combinations:
on output descriptor, |
Beta Was this translation helpful? Give feedback.
-
|
Regarding the face descriptor size, the way is to change the architecture and retrain the required layers, correct? Or is there a different way? |
Beta Was this translation helpful? Give feedback.
-
|
clearly, that's the best way
if reduce operation is decent, then resulting descriptor is only 256-elements and still maintains most of it's precision |
Beta Was this translation helpful? Give feedback.
-
I was thinking of writing a video demo using nodeJS as client side. Do you know of a tool to capture video stream from nodeJS and that can be fed into human for detection? I could only find tools that have callback on each frame capture and from what I understood about the API, to leverage the aggressive caching I need to pass in a HTMLVideoElement. I might be wrong so it'd be great if you could point out how this could be done!
Beta Was this translation helpful? Give feedback.
All reactions