Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Plans to add DoRA? #29153

Closed
RonanKMcGovern opened this issue Feb 20, 2024 · 4 comments
Closed

Plans to add DoRA? #29153

RonanKMcGovern opened this issue Feb 20, 2024 · 4 comments
Labels
Feature request Request for a new feature

Comments

@RonanKMcGovern
Copy link

Feature request

Improves on LoRA by allowing magnitude fine-tuning.

Motivation

Improved perplexity.

Your contribution

Sebastien Bubeck has published demo code. https://github.com/rasbt/dora-from-scratch
@amyeroberts amyeroberts added the Feature request Request for a new feature label Feb 20, 2024
@amyeroberts
Copy link
Collaborator

cc @younesbelkada @pacman100

@younesbelkada
Copy link
Contributor

Hi @RonanKMcGovern !
Thanks for the feature request! There is already an ongoing work from @BenjaminBossan to add DoRA in PEFT: huggingface/peft#1474

@RonanKMcGovern
Copy link
Author

Closing as there is a PR underway.

@younesbelkada
Copy link
Contributor

OK thank you @RonanKMcGovern !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Feature request Request for a new feature
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@amyeroberts @younesbelkada @RonanKMcGovern