Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ENH: Enable OFT adapter for mixed adapter models #1204

Merged
merged 5 commits into from
Dec 4, 2023
Merged

ENH: Enable OFT adapter for mixed adapter models #1204

merged 5 commits into from
Dec 4, 2023

Commits on Dec 1, 2023

  1. ENH Enable OFT adapter for mixed models 

    This PR makes it possible to use the newly added OFT adapter in mixed
adapter type models, similar to LoRA, LoHa, etc.

Notes

Adding the integration was pretty straightforward, which is a good sign.

The difficult part was actually about the tests. This stems from the
fact that OFT is (if my understanding is correct) never commutative.
What I mean is that even if the adapters are applied to the last layer
of a model, it makes a difference whether we apply, say, first LoRA,
then OFT vs first OFT, then LoRA.

This is different for the other adapters that were added so far for
mixed models, as they basically do:

- Xa = X + dXa
- Xab = Xa + dXb = X + dXa + dXb = X + dXb + dXa = Xb + dXa = Xba

IIUC, this is not true for OFT, so when OFT is used, I had to ensure
that no test was applied that (implicitly) assumes commutativity.

Furthermore, I had to increase the model size, see this comment:

huggingface#1160 (comment)
    @BenjaminBossan
    BenjaminBossan committed Dec 1, 2023
    Configuration menu
    Copy the full SHA
    4c55e52 View commit details
    Browse the repository at this point in the history

  2. Make test name shorter 

    No point in including "Config".
    @BenjaminBossan
    BenjaminBossan committed Dec 1, 2023
    Configuration menu
    Copy the full SHA
    7e9c2f0 View commit details
    Browse the repository at this point in the history

  3. Increase tolerance a bit more 

    On CPU, apparently higher tolerance is required than on GPU.
    @BenjaminBossan
    BenjaminBossan committed Dec 1, 2023
    Configuration menu
    Copy the full SHA
    ab08cb4 View commit details
    Browse the repository at this point in the history

  4. Make style

    @BenjaminBossan
    BenjaminBossan committed Dec 1, 2023
    Configuration menu
    Copy the full SHA
    59b27ae View commit details
    Browse the repository at this point in the history

  5. Increase tolerance a bit more...

    @BenjaminBossan
    BenjaminBossan committed Dec 1, 2023
    Configuration menu
    Copy the full SHA
    cbfe937 View commit details
    Browse the repository at this point in the history