Correct Learning Rate

[codestuff001]

In Dreambooth-GUI, the default Learning Rate (LR) is set to 1e-5, but when I check Shivam's Dreambooth notebook, it's set to 5e-6.

In this article they give a range of high to low, from 5e-6 to 2e-6:
https://wandb.ai/psuraj/dreambooth/reports/Dreambooth-training-analysis--VmlldzoyNzk0NDc3

I couldn't find anything online that explains what the LR value does.

[futurevessel]

This is likely a question of trial and error unless you want to get deep into the math. Hopefully someone will do a deep dive tutorial into this subject eventually, the ideal learning rate value seems dependent on the number of steps and probably the amount of images you train on, perhaps even the subject (person, style etc).

[codestuff001]

What exactly is the learning rate, and how does it affect everything? I had to write it out because the scientific notation kept messing with my head:
1e-4 (0.0001)______Overtrained
1e-5 (0.00001)_____Dreambooth-GUI default
5e-6 (0.000005)____High-LR (Shavim's default old)
2e-6 (0.000002)____Low-LR
1e-6 (0.000001)____Undertrained (Shavim's default current)

I understand higher LR leads to overtraining, but how does it affect things like processing time?

Is there an equation with LR, steps, and number of images, like: LR * steps / images = factor (time, fit level, quality)

Or something like that?

[futurevessel]

Wish I had some good answers, I'm just experimenting myself and trying to find information from those who produce good results.