Trainer._load_from_checkpoint - support loading multiple Peft adapters

[claralp]

What does this PR do?

Since it it possible to have multiple Peft adapters in the same model, it should also be possible to resume a training of such models from checkpoint with transformers.Trainer.train(resume_from_checkpoint=True|"path").
No documentation changes because this is a fix of sth already existing, tested with DPO/KTO

Fixes #30478

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

@lewtun @kashif @younesbelkada

[kashif]

nice @claralp would you mind also adding some tests?

[claralp]

@kashif will try. Guess I can use a similar model setup as here test_trainer.py#L933

Another question: How is the checkpoint loading with DeepSpeed supposed to work with this trainer.py#L1847?

[younesbelkada]

Thanks for adding the support for loading multiple adapters with load_from_checkpoint!

[lewtun]

Very clean implementation @claralp !

Guess I can use a similar model setup as here test_trainer.py#L933

Yeah this makes sense and I think you should be able to adapt the logic used for full training here:

transformers/tests/trainer/test_trainer.py

Line 1787 in 1e05671

def test_can_resume_training(self):

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[claralp]

@kashif @lewtun please check if this works now for you.

Note: Discovered another bug while implementing this.
While the PeftAdapterMixin.set_adapter function supports multiple active adapters, its internal logic for setting multiple adapters active is broken.
But this needs to be fixed in the Peft library first

[kashif]

ok @claralp let's link the peft issue here too then?

[younesbelkada]

Thanks again ! seconded what @kashif said ! if you could have a small reproducer to repro the bug and file it on PEFT it would be really great 🙏

[claralp]

@kashif, @younesbelkada the issue cannot be reproduced anymore.

The background:
Since multiple adapters can be active at the same time I would have expected the same in src/peft/utils/other.py#L272.
I had an error previously when calling this with a list of adapters in src/transformers/integrations/peft.py#L311.
This was before commits/16bd0ef66f642ef4b69f4f21cfc368d548318672 when I accessed the active_adapters property instead of active_adapter.
But when I now run a test case with two active adapters, it works.
So I would treat this as solved

[claralp]

@kashif @younesbelkada @lewtun is there any open point for me to fix now or is this ready?

[younesbelkada]

On my end looks great, just waiting for a final review cc @ArthurZucker @LysandreJik

[LysandreJik]

This looks good to me! cc @muellerzr if you're fine with the changes in Trainer, feel free to merge.

[muellerzr]

LG2M, thanks for also adding a test!