Multiomic tutorial with cellular predictor donwtask.

[Ragagnin]

Hello, first I would like to congratulate you on the incredible work!
I'm currently using Multiomic tutorial to try to predict cell types, like the Annotation tutorial.

Does anyone have any ideas on how I can perform this task using multiomics data?

At first I'm trying to "merge" the two available tutorials: Multi_omic Tutorial with the Annotation Tutorial. But I'm having difficulty at some points.

If anyone has a tip on how to carry out this process, I would greatly appreciate it!

Thank you very much in advance!

[subercui]

Hi, thank you for the question! That would be interesting, one straight-forward idea I have is to use the classification training pipeline in the annotation tutorial while using the multi-omics data processing and model settings. Please note the current annotation tutorial assumes you have some "reference" data that has been annotated, and will train a classifier on these data and apply it to the actual query data. Is this what you need?

[Ragagnin]

Exactly! I have a database as a reference.

So I just use the training part of the annotation tutorial, importing the essential libraries to do the training?

It would be something like this:

1. Specify hyper-parameter setup for integration task (Multiomics)
2. Load and pre-process data (Multiomics)
3. Load the pre-trained scGPT model (Annotation)
4. Finetune scGPT with task-specific objectives (Annotation)
5. Evaluate fine-tuned scGPT (Annotation)

Maybe step 1 has to be a mix of the two tutorials to get the essential parameters for step 3. Right?

[subercui]

Yes, that makes sense to me. Only in step 3, you likely also want to load the multi-omics model as in the multi-omic tutorial. The important setting to borrow is enabling the "CLS" option, and you can search for this in the annotation tutorial. Basically, you will need to enable this line

scGPT/scgpt/model/multiomic_model.py

Line 393 in 418b0f6

output["cls_output"] = self.cls_decoder(cell_emb) # (batch, n_cls)

to let the model output the predictions.

Let me know how it works.

[Ragagnin]

Hello, Thanks for the tips! Unfortunately I am still unable to complete this task. But I can get to phase 4: "Step 4: Finetune scGPT with task-specific objectives". When I try to do the training it returns an error that I cannot resolve.

In summary, I made these changes:
1 -Specify hyper-parameter setup for integration task (Multiomics/Annotation)
2 -Load and pre-process data (Multiomics)
- Import the reference data
- Process multiomics data

3 -Load the pre-trained scGPT model (Multiomics/Annotation)
4 -Finetune scGPT with task-specific objectives (Annotation)
5 -Evaluate fine-tuned scGPT (Annotation)

The error occurs during training:
"if config.do_train:
train(
model,
loader=train_loader,
)"

This is the error message:
random masking at epoch 1, ratio of masked values in train: 0.3988

---

AssertionError Traceback (most recent call last)
in <cell line: 6>()
23
24 if config.do_train:
---> 25 train(
26 models,
27 loader=train_loader,

4 frames
in train(model, loader)
30 src_key_padding_mask = input_gene_ids.eq(vocab[pad_token])
31 with torch.cuda.amp.autocast(enabled=config.amp):
---> 32 output_dict = model(
33 input_gene_ids,
34 input_values,

/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
1188 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
1189 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1190 return forward_call(*input, **kwargs)
1191 # Do not call functions when jit is used
1192 full_backward_hooks, non_full_backward_hooks = [], []

/content/scGPT/scgpt/model/multiomic_model.py in forward(self, src, values, src_key_padding_mask, batch_labels, CLS, CCE, MVC, ECS, do_sample, mod_types)
348 dict of output Tensors.
349 """
--> 350 transformer_output = self._encode(
351 src, values, src_key_padding_mask, batch_labels
352)

/content/scGPT/scgpt/model/multiomic_model.py in _encode(self, src, values, src_key_padding_mask, batch_labels)
172 batch_labels: Optional[Tensor] = None, # (batch,)
173 ) -> Tensor:
--> 174 self._check_batch_labels(batch_labels)
175
176 src = self.encoder(src) # (batch, seq_len, embsize)

/content/scGPT/scgpt/model/multiomic_model.py in _check_batch_labels(self, batch_labels)
225 def _check_batch_labels(self, batch_labels: Tensor) -> None:
226 if self.use_batch_labels or self.domain_spec_batchnorm:
--> 227 assert batch_labels is not None
228 elif batch_labels is not None:
229 raise ValueError(

AssertionError:

---

From what I understand, the error indicates that batch_labels is None, but I defined this variable previously since my dataset has two different batches. I'm thinking that the error may be related to a parameter that I may be assigning wrong.

Here is Setp1 -Specify hyper-parameter setup for integration task (Multiomics/Annotation):

hyperparameter_defaults = dict(
task = 'multiomic',
seed=42,
dataset_name="Met_Expression", # Dataset name
do_train=True, # Flag to indicate whether to do update model parameters during training
load_model="../save/scGPT_human", # Path to pre-trained model
freeze = False, #freeze
GEP=True, # Gene expression modeling
GEPC=True, # Gene expression modeling for cell objectives
CLS=True, #Change to True
ESC=False,
DAR = True, # DAR objective weight for batch correction
DSBN = False, # Domain-spec batchnorm,
mask_ratio=0.4, # Default mask ratio
explicit_zero_prob = False, # whether explicit bernoulli for zeros
ecs_thres=0, # Elastic cell similarity objective, 0.0 to 1.0, 0.0 to disable
dab_weight=1.0,
use_batch_labels = True,
use_mod = True,
per_seq_batch_sample = False,
epochs=25, # Default number of epochs for fine-tuning
input_layer_key = "X_binned", # Default expression value binning in data pre-processing
n_bins=51, # Default number of bins for value binning in data pre-processing
n_hvg = 1200, # Default number of highly variable genes
n_hvp = 4000,
max_seq_len = 4001, # # Default n_hvg+1
lr=1e-3, # Default learning rate for fine-tuning
batch_size=16, # Default batch size for fine-tuning
layer_size=512,
nlayers=4,
nhead=8, # if load model, batch_size, layer_size, nlayers, nhead will be ignored
dropout=0.2, # Default dropout rate during model fine-tuning
schedule_ratio=0.95, # Default rate for learning rate decay
save_eval_interval=5, # Default model evaluation interval
log_interval=100, # Default log interval
fast_transformer=True, # Default setting
pre_norm=False, # Default setting
amp=True, # Default setting: Automatic Mixed Precision
pad_token = "",
mask_value = -1,
pad_value = -2,
include_zero_gene = False,
MVC=False,
)

the hyper parameters above were taken from the multiomics tutorial and the parameters below were taken from the annotation tutorial:

settings for input and preprocessing
pad_token = ""
special_tokens = [pad_token, "", ""]
mask_ratio = config.mask_ratio
mask_value = "auto" # for masked values, now it should always be auto

include_zero_gene = config.include_zero_gene # if True, include zero genes among hvgs in the training
max_seq_len = 3001
n_bins = config.n_bins

input/output representation
input_style = "binned" # "normed_raw", "log1p", or "binned"
output_style = "binned" # "normed_raw", "log1p", or "binned"

settings for training
MLM = False # whether to use masked language modeling, currently it is always on.
CLS = True # celltype classification objective
ADV = False # Adversarial training for batch correction
CCE = False # Contrastive cell embedding objective
MVC = config.MVC # Masked value prediction for cell embedding
ECS = config.ecs_thres > 0 # Elastic cell similarity objective
DAB = False # Domain adaptation by reverse backpropagation, set to 2 for separate optimizer
INPUT_BATCH_LABELS = False # TODO: have these help MLM and MVC, while not to classifier
input_emb_style = "continuous" # "category" or "continuous" or "scaling"
cell_emb_style = "cls" # "avg-pool" or "w-pool" or "cls"
adv_E_delay_epochs = 0 # delay adversarial training on encoder for a few epochs
adv_D_delay_epochs = 0
mvc_decoder_style = "inner product"
ecs_threshold = config.ecs_thres
dab_weight = config.dab_weight

explicit_zero_prob = MLM and include_zero_gene # whether explicit bernoulli for zeros
do_sample_in_train = False and explicit_zero_prob # sample the bernoulli in training

per_seq_batch_sample = False

settings for optimizer
lr = config.lr # TODO: test learning rate ratio between two tasks
lr_ADV = 1e-3 # learning rate for discriminator, used when ADV is True
batch_size = config.batch_size
eval_batch_size = config.batch_size
epochs = config.epochs
schedule_interval = 1

settings for the model
fast_transformer = config.fast_transformer
fast_transformer_backend = "flash" # "linear" or "flash"
embsize = config.layer_size # embedding dimension
d_hid = config.layer_size # dimension of the feedforward network in TransformerEncoder
nlayers = config.nlayers # number of TransformerEncoderLayer in TransformerEncoder
nhead = config.nhead # number of heads in nn.MultiheadAttention
dropout = config.dropout # dropout probability

logging
log_interval = 100 # iterations
save_eval_interval = config.save_eval_interval # epochs
do_eval_scib_metrics = True

I changed the CLS parameter to true in hyper_parameters.

I don't know if I'm doing something wrong.
Could you help me with this issue?
I tried to include all the information you could need, but if is missing any info, just tell and I'll let you know as soon as possible.

Thank you in advance for all your help.