training time RAVE vs PRIOR

[ghost]

when training both models in order to combine them - should their training be about the same amount in steps / epochs - or is one model to be trained longer?

[ghost]

so while going through older issues i found a comment (by Antoine i think) where one would train the prior for 1M steps, i assume that was after training the rave with 2M steps..?

[adrianwedd]

I hope this is correct. Seems the prior doesn't resume like the rave training? I'm at ~970000 steps on prior after the first run timed out and hoping it completes before my credits run out. I wrongly assumed prior would run for the same number of epochs as the rave and was/am scared it'd try to run for 2 million steps.

[adrianwedd]

nope, still going past 1M :-(

[ghost]

for checkpoint resumption insert this into the prior command, similiar as it is in the rave one:
--ckpt /content/drive/MyDrive/RAVE_COLLAB/runs/$training_name/prior/
see also #122

as i understand it there is no completion, only approximation, so around +/- 1M steps sounds fine. maybe you can try the prior out and judge it with your ears and if necessary resume training

[jreus]

+1 for evaluating by ear. I've had useful priors train in less than a day.. sometimes 100k steps was even enough to have something nice.