extrinsic optimize

[fanxinyun1991]

Did you test any real case scene? I got a video, and obtain extrinsics using colmap. With the extrinsic optimizer open, the inference results are not good. The inference images looks like covered in smoking. (while in the same case, barf obtain a good result.)

[Tom94]

Hi there!

Our extrinsics optimizer is experimental (not used in any paper results) and definitely not as thoroughly evaluated as those of other publications.

While it helps us obtain slightly more accurate fits in our own real-world scenes (for example the robot factory from the top of the README), colmap is already really close to optimal in these.

For what it's worth, another experiment that we've performed is verifying that it can recover from slightly perturbed camera poses in synthetic scenes.

Some more technical details for those readers familiar with BARF and other works: we parameterize rotation gradients in terms of moment vectors---imagine the camera as a sphere with mass and the loss gradients as imparting angular momentum onto it. This gives Adam a really intuitive physical analog and turns out to be equivalent to certain log-space rotation formulations. (And more importantly for us: it's a very simple mathematical formulation/implementation akin to rigid body simulations.)

However, we have not investigated and/or tuned this formulation in great depth. As with some of the other features that go beyond the convex hull of the paper: here be dragons. :)

[fanxinyun1991]

Thank you for your reply and appreciate your perfact work.

is there any different with barf , in principle? using the same data, we got good results with barf, but not with NGP. I worried that NGP optimize too fast, and completed the training before the extrinsic converge.

Only increase training step do not got any improvement. When I perturbed camera poses larger than 0.05, I got bad results.

Do you have any plan to add experiments in real-world scene? Improve the proformence in scenes with inaccurate camera poses?

[Tom94]

@ashawkey just discovered a downstream bug in tiny-cuda-nn that probably negatively affected the extrinsics/distortion optimization. The bug didn't affect any other aspects of instant-ngp (thankfully).

It is fixed now (commit 409613a), so it may be worth trying again.

Another tip: it may prove insightful to enable Debug Visualization -> [ ] Visualize Cameras, which will let you see the optimized camera position and orientation in relation to the prescribed position from the training data.

[roangel]

I have another question closely related to this one @Tom94. Let's say I have a perfectly trained nerf, and then let's say I have one single picture I want to figure out the extrinsics for. I start training where everything is frozen except the extrinsics. Then, I perturb slightly the initial pose of the image (change a bit the original transform in transforms.json) and train for some steps. The result looks actually more like noise than converging to anything close to its ground-truth. I freeze everything with this:

    testbed.nerf.training.optimize_extrinsics = True
    testbed.nerf.training.optimize_extra_dims = False
    testbed.nerf.training.optimize_exposure = False
    testbed.nerf.training.optimize_distortion = False
    testbed.nerf.training.optimize_focal_length = False
    testbed.nerf.training.include_sharpness_in_error = False
    testbed.shall_train_encoding = False
    testbed.shall_train_network = False

The image I chose for this is even an image that is present in the original training dataset so I'd expect that the training is able to recover the pose rather fast.

Is there something I'm doing wrong? Thanks for the help!

[Tom94]

I don't see a fundamental issue with this approach -- I'm surprised to hear it doesn't work, given the setup you describe.

There's code in testbed_nerf.cu that perturbs all the training images (search for float perturb_amount = 0.0f; and increase it to 0.01 or even 0.1) and it seems to be able to reliably recover in my tests using the lego scene.

[roangel]

Thanks for the reply! I even froze the rotation by commenting out the gradient parts through the rotations, and only perturb the translation, and still doesn't recover properly. It's weird because when training the nerf I do allow for extrinsics optimization and it actually doesn't vary much, which means that it should recover a single image pose perturbed very well

[roangel]

Also, I am reading the trained transforms with testbed.nerf.training.get_camera_extrinsics(0) from the python API. I also tried getting it from testbed.nerf.training.transforms but python complains about the type xforms not existing. It's the get_camera_extrinsics the proper day of doing it?