[Question] What kind of information the eight neural descriptors for each point represent?

[ttsesm]

Do you have any idea regarding what kind of information the neural descriptors at each point represent?

I've read the paper but it was not that clear about the information that are learned in these descriptors.

Also do you have any idea how lighting information (e.g. light sources position, intensity, distribution) could be possibly added as extra parameters?

Furthermore, to my understanding by using your model someone could render any map other than just the RGB values is that correct? For example imagine that someone has the irrandiance map of the scene the in principle he could render the corresponding irrandiance values for unseen views right?

[seva100]

Hi @ttsesm, thank you for your interest in our work.

We did not aim for the interpretability of the neural descriptors. These are the 8-dimensional "latent" vectors which the network uses as a memory. Certainly, this representation is richer than the RGB colors of points (otherwise, the Pix2Pix ablation would perform at a similar level, but it's much worse -- you can find the comparison in the paper or at the project page). They also seem to be capturing some view-dependent effects -- this is e.g. seen on our supplementary video at 1:50). As usual with the latent variables, these are hard to interpret; one may try to get some understanding by applying some subspace learning methods (PCA like we do at 1:00, t-SNE, etc.).

We have some ideas about relighting extensions, but I am not sure we can comment on that. You can check out the related concurrent work https://github.com/LansburyCH/relightable-nr.

About your last question: surely you can try to model any kind of mapping, not only RGB, for example normal mapping, as long as you have the ground truth for it. Seems like you can render the diffuse irradiance mapping, as well as some other view-independent information, but you'll need to modify the algorithm somehow to directly model view-dependent and lighting-dependent output if needed.

[ttsesm]

Hi @seva100, thanks a lot for the insightful feedback.

To be honest I am still in the process to fully understand the whole thing that is happening under the hood but in any case it seems really interesting.