Exploring FFPMEG + node

[shiffman]

See #2

[joeyklee]

Awesome - I'm going to working on some of the UI behaviors in the electron app template and hopefully our efforts can meet in the middle ;)

[shiffman]

I now have node hooked up to ffmpeg and am able to execute a script to unpack a video into all of its frames. Next step is to repack frames to video.

Does it make sense for me to create a quick web interface and process the image using tf.js in the browser or should I investigate BodyPix + tfjs-node? @joeyklee let me know if you have an opinion.

[shiffman]

Thinking about this a bit more maybe I want to have a browser interface b/c I want to use canvas to draw on the image and that is likely easier without me worrying about a headless browser/canvas instance?

[joeyklee]

@shiffman - yes I was thinking the same thing. Does it make sense to load the video and run bodyPix on the video in the browser and then save each frame to the file system and then pack them up?

[joeyklee]

So far, what I've got ready for an interface looks like below... when you click "start processing" it would trigger the segmentation process.

I've not worked in Electron before, so still figuring out a few things for file uploads, but things are getting closer in #3

[shiffman]

@joeyklee yes! I now have a function that runs two ffmpeg commands, one to unpack all the frames and one to repack it as a video. I also need to run a command to extract the audio and re-add the audio. I also need to investigate retaining video quality / framerate or other important details from the original video.

Let me know when you want to try to integrate these two together?

async function processVideo(file) {
  // Make the directory
  fs.mkdirSync('frames');

  // unpack frames
  const command1 = `ffmpeg -i ${file} -qscale:v 2 frames/out%03d.jpg`;
  const response1 = await exec(command1);
  console.log(response1);

  // Apply obfuscation

  // repack frames
  const command2 = `ffmpeg -start_number 0 -i 'frames/out%3d.jpg' out.mp4`;
  const response2 = await exec(command2);
  console.log(response2);
}

[joeyklee]

@shiffman - oh cool! this makes sense. I will try to integrate those functions!

[shiffman]

Great! This is probably at its stopping point right now so I could merge it and then we integrate these bits into the electron stuff you are working on?

[joeyklee]

@shiffman - that sounds great! Please go ahead.

[shiffman]

Ok, merging this in! I refactored it so that the ffmpeg node script is the root of the repo with the p5.js demo in a public folder. Obviously it's all disconnected and I could redo it to all work together but I will wait until you fold in your stuff!