Example coarse alignment

[McHaillet]

Hi Alister!

This is the example I added for an IMOD style coarse alignment of tilt-series. It basically sets the 0 degree tilt as a reference, and then finds neighboring tilt shifts in both the positive and negative tilting direction. The coarse shifts are then used as the initial point for optimization for refining the tilt-series alignment. However, in this example the coarse alignment now gives a much better tomogram than after optimization.

Code-wise, I now added the coarse alignment to tsa_real_data_multiregion.py, let me know if that is okay for you. Otherwise I can also move it to a separate example file. Secondly, I used a tilt-series that I had stored locally. I don't know if you have some online reference data to use for this. I can revert the IMAGE_FILE and parameters back to what it was before.

As I told you I found the idea of an automated tilt-series alignment in libtilt very interesting (but currently not much time to mess around with it due to phd thesis). I like your setup with a torch optimization on patches, although quite different from AreTomo. Some observations that I had while experimenting with the code.

• The problem with the optimization partly seems that the dft based image shifts, although very accurate, loop the data around and create some strong edges, I think these can be quite detrimental to the alignment (see below). Perhaps, a solution could be that the dft shift code optionally outputs a binary mask with the areas that contain lost information?
• The example now only optimizes shifts. For tilt-angle offset AreTomo uses image stretching and to optimize an axis-angle it uses common-lines. I guess these things could also be calculated before starting the optimization.

IMAGE: Large y-shift loops the bottom part of the image to the top.

[alisterburt]

@McHaillet you have no idea how happy it makes me to see you playing with libtilt :-)

Let's put this in its own example coarse_alignment.py and maybe add stretching there at a later point? If we end up pushing this further we can then put everything in its own repo and have some reference data auto-downloaded there.

I'm a bit surprised how well this performs even without the stretching perpendicular to the tilt-axis, very nice :-)

Agree that the wraparound is a problem - the patch idea was just something I was playing with and I'm not sure it's better, I thought it might lead to more robust estimates 😆

[alisterburt]

Perhaps, a solution could be that the dft shift code optionally outputs a binary mask with the areas that contain lost information?

Nice idea, I'd err on the side of not overcomplicating the simple shift code and instead having a separate thing that can provide the masks if they're needed... in the motion correction code I have a soft edged circle in the center of the patch, we could also use a similar strategy here as a first step?

[alisterburt]

The example now only optimizes shifts. For tilt-angle offset AreTomo uses image stretching and to optimize an axis-angle it uses common-lines. I guess these things could also be calculated before starting the optimization.

Yep totally agree again - now that we have the coarse alignment we know the area common to each image for common line calculation so that's super useful :-)

[alisterburt]

but currently not much time to mess around with it due to phd thesis

I feel you, good luck getting it all written up! :-)

[McHaillet]

@McHaillet you have no idea how happy it makes me to see you playing with libtilt :-)

Well, thank you for putting in the effort to start an open source community cryo-em library! The code base seems really nice

Let's put this in its own example coarse_alignment.py and maybe add stretching there at a later point? If we end up pushing this further we can then put everything in its own repo and have some reference data auto-downloaded there.

I moved it to a new file!

Just want to mention that in the tsa_real_data_multiregion.py, the shifts for the matrix are commented and a projection_model_optimiser.step() is missing in the optimisation loop. I can make a PR to update it if you want?

[McHaillet]

Nice idea, I'd err on the side of not overcomplicating the simple shift code and instead having a separate thing that can provide the masks if they're needed... in the motion correction code I have a soft edged circle in the center of the patch, we could also use a similar strategy here as a first step?

Agreed, for now its probably fine.

[McHaillet]

I think the most important things for this aretomo-like workflow are the following (perhaps could make issue for these)

• some function find_shift(image1, image2) that uses the 2d cross correlation and extract a position from it. Ideally with upsampling to get a float point estimate of the shift, like in scikit-image phase_cross_correlation
• a version of project_fourier or project_real that works from 2d to 1d for the common line matching (at least that is how they do it in aretomo)
• and some code that can stretch along a tilt-axis at arbitrary angle in a 2d image, I guess that could be done with an affine matrix that has a rotation and a scaling along a single axis

[alisterburt]

🥰 love to see it!

Have opened a few issues for the missing pieces - let's merge this here
#64 #65 #66