cc.gr.diff_nhood_enrichment stopping at 50 permutations or killing RAM

[LukasHats]

Report

Dear @marcovarrone,

sorry to post again, thanks to your help I could make everything work but I am now stuck at the cc.gr.diff_nhood_enrichment.

I have a new M3 Macbook with 36GB RAM, but when I set n_jobs=2 or higher, I get a system warning after a few minutes, that I ran out of RAM, even when only VSCode is opened. The odd thing is, that my activity monitor is not even showing huge amounts of CPU usage, so it might not even be really detected?

When I set n_jobs=1 it does not complain but always stops after around 50 permutations and then just runs forever without increasing the %.

cc.gr.diff_nhood_enrichment(
    adata,
    cluster_key='cellcharter_CN11',
    condition_key='disease2',
    library_key='image_ID',
    pvalues=True,
    n_jobs=1,
    n_perms=100
)

0%|          | 0/100 [00:00<?, ?it/s]WARNING:root:In order to use sagenet models, please install pytorch geometric (see https://pytorch-geometric.readthedocs.io/) and 
 captum (see https://github.com/pytorch/captum).
WARNING:root:mvTCR is not installed. To use mvTCR models, please install it first using "pip install mvtcr"
WARNING:root:multigrate is not installed. To use multigrate models, please install it first using "pip install multigrate".
 48%|████▊     | 48/100 [05:20<06:02,  6.96s/it]

adata

AnnData object with n_obs × n_vars = 1018580 × 33
    obs: 'Object', 'area', 'Y_centroid', 'X_centroid', 'axis_major_length', 'axis_minor_length', 'eccentricity', 'distance_to_bone', 'Phenotype', 'image_ID', 'disease', 'patient_ID', 'ROI', 'disease2', 'distance_to_bone_corrected', 'Phenotype2', 'Phenotype3', 'cellcharter_CN11'
    var: 'name', 'channel', 'deepcell', 'mean', 'std'
    uns: 'spatial', 'spatial_neighbors', 'image_ID_colors', '_cellcharter', 'cellcharter_CN11_Phenotype3_enrichment', 'cellcharter_CN11_disease2_enrichment', 'cellcharter_CN11_colors'
    obsm: 'X_trvae', 'spatial', 'X_cellcharter'
    layers: 'arcsinh', 'zscore'
    obsp: 'spatial_connectivities', 'spatial_distances'

adata.obs['disease2'].value_counts()

disease2
MM_BD      733976
MM_noBD    152872
SMM         66094
MGUS        65638
Name: count, dtype: int64

adata.uns['spatial'] 

{'TS-373_IMC77_B_001.csv': {},
 'TS-373_IMC03_MGUS_002.csv': {},
 'TS-373_IMC04_MGUS_002.csv': {},
 ...
  'TS-373_IMC83_B_001.csv': {}}

Does it accept 4 conditions in the first place? Or did I use the wrong library key?

Thanks for the help again!

Version information

No response

[marcovarrone]

You are right! I found a memory leak in the implementation.
Today I will publish a new version of CellCharter with the fixed (and much faster!) version.

Thank you for spotting it :) I'll keep you updated

[marcovarrone]

You can now download CellCharter v0.3.2 and it should work.
If not, feel free to reopen the issue.