train process problem

[CS123n]

Hi, I used your code to train SD+T5 on my own.
However, the results deteriorated rapidly after only 500 steps.

Here's what the training loss looks like:

Do you have any advice? I tried changing the learning rate to 1e-5, but it didn't solve the problem.

[ShihaoZhaoZSH]

Thank you for your interest in our LaVi-Bridge! We haven't encountered such a situation in our experiment, and the released training and inference code has undergone thorough testing to ensure its correctness. We suggest checking the following points: 1. Adjust the learning rate appropriately. 2. Train using full precision. 3. Double-check the inference process to ensure the correct loading of LoRA and proper input of (un)conditional text embeddings into the adapter.