Can not import alignment during running ts_aretomo

[Huang-XN]

Hi Alister,

I have got error messages when running ts_aretomo.

The error messages were:

angpix = 1.7005
mask = False
alignz = 1200
axis_iter = 0
axis_batch = 0
min_fov = 0
axis = null
patches = null
delete_intermediate = False
exe = AreTomo2
device_list = {  }
perdevice = 3
workers = {  }
settings = warp_tiltseries.settings
input_data = {  }
input_data_recursive = True
input_processing = ./tilt_stack_processing/tiltstack
output_processing = ./alignment

No alternative input specified, will use input parameters from warp_tiltseries.settings
File search will be relative to /home/huangxn/Desktop/Playground/protein/data/cryodrgn_10499/warp_2/tomostar
65 files found
Parsing previous results for each item, if available...
65/65, previous metadata found for 0                                                            
Connecting to workers...
Connected to 3 workers
Current tilt axis angle: 85.700 °
Running AreTomo with final average tilt axis angle:
0/65
Failed to import alignments:
all tilt angles are zero in /home/huangxn/Desktop/Playground/protein/data/cryodrgn_10499/warp_2/alignment/tiltstack/00296/00296_Imod/00296_st.tlt

The command I used was:

WarpTools ts_aretomo \
          --settings warp_tiltseries.settings \
          --alignz 1200 \
          --exe AreTomo2 \
          --angpix 1.7005 \
          --input_data_recursive \
          --input_processing ./tilt_stack_processing/tiltstack \
          --output_processing ./alignment \
          --perdevice 3

I ran this on Ubuntu 22.04.4LTS.

Steps before alignment were follow the offcial tutorial.
warp_tiltseries.settings.txt
00296.log

Thank you for your help

Best,
Huang

[alisterburt]

Hi @Huang-XN - is AreTomo2 on your PATH? What do the worker logs say?

AreTomo doesn't like running with multiple processes per GPU in our experience so you might also want to drop --perdevice 3

[alisterburt]

Sorry, just saw you already provided the worker log and it looks ok... could you set the environment variable WARP_DEBUG=1 and rerun?

Do the output files for this TS look okay?

I assume you're having the same issue across multiple TS?

[Huang-XN]

Sorry, just saw you already provided the worker log and it looks ok... could you set the environment variable WARP_DEBUG=1 and rerun?

Do the output files for this TS look okay?

I assume you're having the same issue across multiple TS?

Thank you. I will try with WARP_DEBUG=1 and tell you later on.

Sorry, I have no idea of how to tell if the output is good. I am the very beginner for this.

You are right. I have run this program across multiple TS and got the same issue for multiple TS.

[Huang-XN]

Hi @alisterburt, I have run the ts_aretomo with WARP_DEBUG=1 again. It still gave me the same error message.

Running command ts_aretomo with:
angpix = 1.7005
mask = False
alignz = 1200
axis_iter = 0
axis_batch = 0
min_fov = 0
axis = null
patches = null
delete_intermediate = False
exe = AreTomo2
device_list = {  }
perdevice = 3
workers = {  }
settings = warp_tiltseries.settings
input_data = {  }
input_data_recursive = True
input_processing = ./tilt_stack_processing/tiltstack
output_processing = ./alignment

No alternative input specified, will use input parameters from warp_tiltseries.settings
File search will be relative to /home/huangxn/Desktop/Playground/protein/data/cryodrgn_10499/warp_2/tomostar
65 files found
Parsing previous results for each item, if available...
65/65, previous metadata found for 0                                                                            
Connecting to workers...
Connected to 3 workers
Current tilt axis angle: 85.700 °
Running AreTomo with final average tilt axis angle:
0/65
Failed to import alignments:
all tilt angles are zero in /home/huangxn/Desktop/Playground/protein/data/cryodrgn_10499/warp_2/alignment/tiltstack/TS_84/TS_84_Imod/TS_84_st.tlt
1/65, 03:06:33 remaining                                                                                        
Failed to import alignments:
all tilt angles are zero in /home/huangxn/Desktop/Playground/protein/data/cryodrgn_10499/warp_2/alignment/tiltstack/TS_12/TS_12_Imod/TS_12_st.tlt
2/65, 03:23:25 remaining                                                                                        
Failed to import alignments:
all tilt angles are zero in /home/huangxn/Desktop/Playground/protein/data/cryodrgn_10499/warp_2/alignment/tiltstack/TS_72/TS_72_Imod/TS_72_st.tlt
3/65, 03:27:20 remaining                                                                                        
Failed to import alignments:
all tilt angles are zero in /home/huangxn/Desktop/Playground/protein/data/cryodrgn_10499/warp_2/alignment/tiltstack/TS_10/TS_10_Imod/TS_10_st.tlt
4/65, 03:00:44 remaining                                                                                        
Failed to import alignments:
all tilt angles are zero in /home/huangxn/Desktop/Playground/protein/data/cryodrgn_10499/warp_2/alignment/tiltstack/TS_05/TS_05_Imod/TS_05_st.tlt
5/65, 02:56:34 remaining  

The command:

WarpTools ts_aretomo \
          --settings warp_tiltseries.settings \
          --alignz 1200 \
          --exe AreTomo2 \
          --angpix 1.7005 \
          --input_data_recursive \
          --input_processing ./tilt_stack_processing/tiltstack \
          --output_processing ./alignment \
          --perdevice 3

log files:
TS_84.log
TS_72.log
TS_12.log

tomostar:
TS_84.tomostar.txt

The output looks as follow:

I have tried mannually importing alignments with ts_import_alignments command. But it still failed. It said could not find ***.xf file.
ts_import_alignments was as follow:

WarpTools ts_import_alignments \
--settings warp_tiltseries.settings \
--alignments ./alignment/tiltstack \
----angpix 1.7005

Notice:
I just renamed the file. The files are identical to what I have used before.

[alisterburt]

Hi @Huang-XN - which version of AreTomo are you running? Are the tilt angles actually all zero in the referenced .tlt file? What does the .aln file say?

It seems that AreTomo is running to completion but producing strange output and I'm not sure why. Does running the same AreTomo outside of Warp produce sane results?

I will not be able to help too actively as I am away for the next few weeks taking holidays - good luck with your further debugging. You may find https://warpem.com/reference/warptools/custom_tilt_series_alignments/ useful too

[Huang-XN]

I used AreTomo2. Version is

AreTomo2 version 1.1.2
Built on Feb 20 2024

Log files told that tilt angles were actually 0.

I have run AreTomo2 outside the WarpTools and imported results.
Thanks for your help.

[jowagner91]

I have run into the same problem. The referenced .tlt file only contains 0s. If I reconstruct the tomograms regardless they look good. So for my part, I have been just ignoring it.
If you run AreTomo2 outside of warp with the same inputs the .tlt file will contain angels but only the ones you feed in from the .rawtlt from what I saw in my limited testing. It seems to me that the --TiltCor flag is not working correctly. I am currently using version 1.1.2 with a fresh installation from today.

[alisterburt]

@jowagner91 thanks for the input - very weird. Do you have the commit hash for the broken version of AT2? I can't reproduce this

[alisterburt]

confirmed that AreTomo2 v1.1.2 with commit hash 241c54de174aa4bf1ad208e4dcd2ba581e9e9a32 is producing tlt files filled with zeros

[jowagner91]

Just to confirm that I understand the Aretomo2 document and warp settings currently: warp runs it with - - TiltCorr set to 1 0 which means that it does correction both for aligment and reconstruction but we overwrite the angle to be 0? So no correct either for aligment or reconstruction?

[alisterburt]

Honestly unsure - AreTomo seems to change what existing flags mean across versions and the documentation doesn't always keep up. E.g. we don't use the 'warp compatible' output at all and it doesn't work as expected in our hands