Synchronize lora's merge, unmerge, etc. modifications to lora's tp_layer.

[zhangsheng377]

There was a previous commit that moved the merge and other functions in LoraLayer to Linear, etc., but the LoraParallelLinear in tp_layer was missed.

[zhangsheng377]

@BenjaminBossan

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[BenjaminBossan]

Thanks for this update. Could you please run make style?

[zhangsheng377]

Thanks for this update. Could you please run make style?

Ok. Sorry, I forgot it.

[BenjaminBossan]

The PR is failing on Python 3.8 because of type annotation syntax. Adding from __future__ import annotations should fix that. I assume that you tried out that the changes you made to the megatron layer work?

[BenjaminBossan]

This should not be necessary, right?

[zhangsheng377]

The PR is failing on Python 3.8 because of type annotation syntax. Adding from __future__ import annotations should fix that. I assume that you tried out that the changes you made to the megatron layer work?

The merge func is that I copyed from Linear.merge

But I forgot to add the from __future__ import annotations

[BenjaminBossan]

I don't really have experience with megatron, so I'm not sure if these methods would just work when copied 1:1. Just to be sure, did you test with your setup that nothing breaks with these changes? If yes, I think we can merge and in the future try to be more careful to keep tp_layer.py in sync.

[zhangsheng377]

I don't really have experience with megatron, so I'm not sure if these methods would just work when copied 1:1. Just to be sure, did you test with your setup that nothing breaks with these changes? If yes, I think we can merge and in the future try to be more careful to keep tp_layer.py in sync.

Ha, yes, in our code we added the patch like this:

setattr(peft.tuners.lora.LoraLayer, 'merge', peft.tuners.lora.Linear.merge)
setattr(peft.tuners.lora.LoraLayer, 'unmerge', peft.tuners.lora.Linear.unmerge)
setattr(peft.tuners.lora.LoraLayer, 'get_delta_weight', peft.tuners.lora.Linear.get_delta_weight)

It works good.

[BenjaminBossan]

It works good.

Okay, great that you tested it. LGTM.