Is it possible to continue fine-tuning after the training has been interrupted?

[hschaeufler]

Every now and then I had to interrupt the fine-tuning. I then tried to continue the training by adjusting the resume_adapter_file field in my config.yaml. However, the training then starts from iteration 1 and not at Iteration 3000 where the last Adapter-File was written (0003000_adapters.safetensors). Am I doing something wrong? Or do I have to adjust the number of iterations manually to remaining iterations?

Which saftensor file do I have to specify in the field? The adapters.saftensors or the 0003000_adapters.safetensors file?

It tried this

# Load path to resume training with the given adapter weights.
resume_adapter_file: "adapters/adapters.safetensors"
# Save/load path for the trained adapter weights.
adapter_path: null

and i also tried it like this.

# Load path to resume training with the given adapter weights.
resume_adapter_file: "adapters/0003000_adapters.safetensors"
# Save/load path for the trained adapter weights.
adapter_path: null

[awni]

You can resume trianing by specifying --resume-adapter-file as you are doing. Specifying the latest numbered adapter file is the same as specifying adapters.safetensors (which is always the latest).

We don't keep the iteration state which is why it resumes at iteration 1. But it is actually continuing from the partially trained adapters.

Does that work ok for you?

[hschaeufler]

You can resume trianing by specifying --resume-adapter-file as you are doing. Specifying the latest numbered adapter file is the same as specifying adapters.safetensors (which is always the latest).

We don't keep the iteration state which is why it resumes at iteration 1. But it is actually continuing from the partially trained adapters.

Does that work ok for you?

I can reduce the steps accordingly. The problem is that I use cosine_decay as learning-rate scheduler with certain values for warmup (309 steps), warmup-init (0) etc. This means that I start again at the initial values and do not follow the intended curve? It would be cool as a feature if you could take the number of steps already performed from the adapter file name and then calculate the continuation point for the learning rate.

[awni]

We should add support for this. If it's still useful to you, could you file an issue about it?