SymForce + PyTorch project ideas

[ameysutavani]

Hello all,

I wanted to start a discussion about SymForce + PyTorch project ideas and find use-cases from the community that might improve/enable new functionality in robotics applications.

For conversation starters, here are some of the ideas that I find interesting:

• Taking inspiration from GradSLAM: https://gradslam.github.io/, it would be a nice experiment to add factor-graph based cost functions to a fully differentiable slam pipeline. Something like this can allow to fully flow the geometric residues and covariances all the way up to the feature extraction layers. Maybe adding custom geometric factors can lead to better convergences than chamfer or quadric losses. One could essentially learn features using factors as a training signal.

• We can also take a look at some massive sfm problems that can benefit from parallelization on gpu. Pytorch can possibly be a (simpler) lower hanging fruit than going down the CUDA route.

• We can also come up with a couple of slam + prediction problems that may benefit from a common graphical representation in torch. Something on the lines of prediction aided slam.

• I am also curious about gaussian belief propagation: https://gaussianbp.github.io/ and it's parallels with factor graphs and transformers. Can be an interesting idea to leverage that to guide the gradient descent in slam and sfm problems.

I am very curious to learn about even more ideas and applications from the symforce community that would benefit from the above integration. Looking forward to gathering a pool of ideas here and then we can use this space to roadmap them according to feasibility, impact and added value.

CC: @hmartiro @aaron-skydio

[hayk-skydio]

Hey @ameysutavani! Sorry for the very late reply on my part. We're super excited to have you contribute in this area. There are a lot of promising goals in the space of SymForce + PyTorch, and also more generally in the space of performant SLAM.

I believe the first bullet is the best place to start out, by building a differentiable SLAM example using SymForce to generate a PyTorch graph and on-manifold derivatives and then using it to learn the task end-to-end. @aaron-skydio just made a barebones PyTorch backend for generating free functions, which is the best starting point here. More codegen work is needed to generate the geometry and camera classes, which would make PyTorch usage much more convenient. We would be very happy for you to start down this path.

Beyond that, I'm also quite interested in ideas around parallelizing large-scale SfM or GBP. My knowledge consists mostly of methods that parallelize these methods on CPU, so I'd be curious if you have any references for doing GPU acceleration of sparse SLAM problems. Another exciting project would be to attempt a directly SLAM system using SymForce + PyTorch, that may map nicely onto the GPU.

Tagging people for additional thoughts: @xipeng-wang-skydio @ryan-skydio @zachteed

[ameysutavani]

@hayk-skydio Great to hear that! And I am eager to hear for other members as well!

I guess I can start going thorough the relevant symforce code to get comfortable with how the code generation is happening (I am new to jinja, but want to learn, and it always helps to have a goal point in mind). If you or @aaron-skydio could help point me towards instructions to setup the a barebones dev environment, I can start chipping away.

Maybe, we should also set some bi-weekly or weekly cadence to track these directions and planned work?

[aaron-skydio]

Sure, personally I'm a fan of conda but whatever python environment you like should do. I probably recommend building symforce from source with CMake as described here, since that will give you the most full-featured dev environment. Since you probably won't be modifying the C++ components of symforce, you could probably get away with installing symforce with pip install -e . as described here, which is simpler but has some limitations (probably most relevant for you is that it doesn't let you run all the tests).

To get familiar with the code generation, you can start looking at the generated sym package, which comes from the templates here and is assembled by the codegen test here (which primarily calls cam_package_codegen.generate for Python). We'll have the preliminary pytorch backend published soon so you can start building on top of it.

[ameysutavani]

Thanks @aaron-skydio. I will try this out.

[aaron-skydio]

Hey @ameysutavani, just a heads up that the initial PyTorch backend is now merged here if you want to start playing with it: a5341d8

It currently doesn't support geo or cam types - you can work around this by using their storage vectors in the interfaces to your generated functions instead of the actual types, but it would also be nice to add support for the classes in the interfaces to generated functions

[ameysutavani]

Thanks @aaron-skydio. I can start playing playing around on top of it.

[ameysutavani]

@hmartiro @aaron-skydio One more good reason for pursuing factor graph computations on GPU is that this will also unlock extra capabilities of off-the-shelf single board computers like Nvidia Jetson family, which might have anemic CPUs but decent GPUs or DLAs.

Again, it is very application dependent if this really gives any speedups and profiling is probably the only way to be sure. But this might be one more "potentially good side effect" to track.