Can this be used for connecting three.js with TensorFlow.js?

[EliasHasle]

tfjs is a JS version of TensorFlow, and successor to the experimental deeplearn.js. When WebGL is available, tfjs will accelerate operation using the GPU. tfjs has functions to create a tensor from a pixel source (with similar restrictions as texImage2D), and to create an image from a tensor.

I would like to combine it with three.js, e.g. to capture "sensory" data from three.js graphics and, conversely, augment three.js graphics with data from tfjs.

Do you think this library would be suitable for such communication, when the goal is increased performance? Or is it better to stick with communication through canvas textures?

[greggman]

Why don't you just try it?

I have no idea if tensorflow supports WebGL1. You'll need to prevent it from trying to use WebGL2 since this library doesn't support WebGL2 by using

  virtualWebGL.setup({
    disableWebGL2: true,
  });

[EliasHasle]

OK. Thanks. I may actually try it (or rather make some limited experiments with it). I suspect many small overheads from the virtualization will add up to something that may be just as bad as the expected overhead from communicating through canvas textures. But it would depend on the application. When (if) I have done some experiments, I will post an update here.

This work is very interesting anyway. 👍

[ghost]

@EliasHasle Did you ever experiment with something like this?

[EliasHasle]

@EliasHasle Did you ever experiment with something like this?

No, regrettably.