FIX: More VeRA tests, fix tests, more checks #1900

BenjaminBossan · 2024-07-02T12:43:38Z

Fixes incorrect config for VeRA in a test
Add VeRA to multi-adapter tests
Add more checks on the VeRA A/B shapes

The latter becomes necessary when we add more than one VeRA adapter. The shapes for VeRA A and B are only determined once, when the first VeRA adapter is created. After that, they are fixed. However, users may add a second VeRA adapter. As long as that adapter targets the same layers and has the same rank, we're good. But if it targets other, bigger layers, or if it has increased rank, the shapes of VeRA A and/or VeRA B will be too small, resulting in an error during the forward pass. To prevent this, we already check the shapes during initialization of the new adapter and raise an error right away.

- Fixes incorrect config for VeRA in a test - Add VeRA to multi-adapter tests - Add more checks on the VeRA A/B shapes The latter becomes necessary when we add more than one VeRA adapter. The shapes for VeRA A and B are only determined once, when the first VeRA adapter is created. After that, they are fixed. However, users may add a second VeRA adapter. As long as that adapter targets the same layers and has the same rank, we're good. But if it targets other, bigger layers, or if it has increased rank, the shapes of VeRA A and/or VeRA B will be too small, resulting in an error during the forward pass. To prevent this, we already check the shapes during initialization of the new adapter and raise an error right away.

HuggingFaceDocBuilderDev · 2024-07-02T12:47:28Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

BenjaminBossan · 2024-07-02T15:49:52Z

Could you please review @dkopi?

dkopi

looks good, I've only added few comments regarding the raised message;

to easily fix this problem in vera I think the simplest solution would be to add optional config arguments that allow to specify the shared A/B shapes manually;
the other way would be to create separate "shared A/B" matrices per each adapter

dkopi · 2024-07-03T17:16:16Z

src/peft/tuners/vera/layer.py

@@ -100,6 +100,23 @@ def update_layer(
            # we can take any of the existing adapter's parameters, as they should all be identical
            vera_A_param = list(self.vera_A.values())[0]
            vera_B_param = list(self.vera_B.values())[0]
+
+            error_tmpl = (
+                "{} has a size of {} but {} is or greater required; this probably happened because an additional VeRA "


a typo - should be "{} has a size of {} but {} or greater is required [...]", right?

we could also have separate messages for incompatible r dimension or input/output dimensions;
can also give a hint that adding adapters in different order may resolve the problem

dkopi · 2024-07-03T17:18:27Z

tests/test_initialization.py

+
+        model = get_peft_model(self.get_model(), config0)
+        # not full message but enough to identify the error
+        msg = "vera_A has a size of 10 but 20 is or greater required"


dkopi · 2024-07-03T17:24:26Z

tests/test_initialization.py

+
+        model = get_peft_model(self.get_model(), config0)
+        # not full message but enough to identify the error
+        msg = "vera_A has a size of 123 but 456 is or greater required"


BenjaminBossan · 2024-07-04T10:39:47Z

@dkopi Thanks for the helpful comments, please check if I have addressed them sufficiently. CI is failing for unrelated reasons.

Also a thought that I had:

With your recent fix to the shape issue, we should be able to allow some models that formerly didn't work, right?

peft/src/peft/utils/constants.py

Lines 200 to 207 in 09358aa

    
           # "btlm": ["c_proj", "c_attn"],  # tested, does not work because of different shapes 
        
           "codegen": ["qkv_proj"], 
        
           # "mistral": ["q_proj", "v_proj"],  # tested, does not work because of different shapes 
        
           # "mixtral": ["q_proj", "v_proj"],  # tested, does not work because of different shapes 
        
           "stablelm": ["q_proj", "v_proj"], 
        
           # "phi": ["q_proj", "v_proj", "fc1", "fc2"],  # tested, does not work because of different shapes 
        
           "phi": ["q_proj", "v_proj"], 
        
           # "gemma": ["q_proj", "v_proj"],  # tested, does not work because of different shapes

I cannot test at the moment, but I think we can uncomment them.

dkopi · 2024-07-04T10:52:21Z

yeah, some/all of them should work now;

I've already tested adapting all linear layers of gemma and phi and it worked

BenjaminBossan · 2024-07-04T11:05:18Z

@matthewdouglas LMK if you feel ready for reviewing this PR.

sayakpaul

Thanks. Very minor comments.

sayakpaul · 2024-07-22T10:24:43Z

tests/test_custom_models.py

+        "VeRA Same",
+        "vera",
+        VeraConfig,
+        {"target_modules": ["lin0"], "init_weights": False},
+        {"target_modules": ["lin0"], "init_weights": False},


So essentially, these two adapters ("VeRA Same" and "vera") are the same. Right?

"VeRA Same" is just the name of the test, "vera" is the name of the method being used. I added a comment at the top that describes the items of this tuple for clarification.

The two adapters here have the same config, but they are two different adapters.

sayakpaul · 2024-07-22T10:31:59Z

tests/test_initialization.py

+
+        model = get_peft_model(self.get_model(), config0)
+        # not full message but enough to identify the error
+        msg = "vera_A has a size of 123 but 456 or greater is required"


Can we programmatically determine 123 and 456?

Changed the code to make it obvious where the numbers come from.

sayakpaul · 2024-07-22T10:32:16Z

tests/test_initialization.py

+
+        model = get_peft_model(self.get_model(), config0)
+        # not full message but enough to identify the error
+        msg = "vera_A has a size of 10 but 20 or greater is required"


Same. Can we programmatically determine the magic numbers here?

Changed the code to make it obvious where the numbers come from.

BenjaminBossan · 2024-07-22T13:41:36Z

@sayakpaul Your points should be addressed, LMK if you want to do another review or if the PR can be merged.

sayakpaul · 2024-07-22T13:42:30Z

Thanks for clarifying the comments.

Merge branch 'main' into fix-vera-tests-and-additional-shape-checks

160eda1

dkopi suggested changes Jul 3, 2024

View reviewed changes

Revier feedback: wording, better error message

06168c9

dkopi approved these changes Jul 4, 2024

View reviewed changes

BenjaminBossan added 3 commits July 15, 2024 11:58

Merge branch 'main' into fix-vera-tests-and-additional-shape-checks

3e8cc2b

Merge branch 'main' into fix-vera-tests-and-additional-shape-checks

d9489c1

Merge branch 'main' into fix-vera-tests-and-additional-shape-checks

d8422ea

BenjaminBossan requested a review from sayakpaul July 22, 2024 09:50

sayakpaul approved these changes Jul 22, 2024

View reviewed changes

Reviewer feedback: Clarify tests

f9c4b83

sayakpaul merged commit ba75bb1 into huggingface:main Jul 22, 2024
14 checks passed

BenjaminBossan deleted the fix-vera-tests-and-additional-shape-checks branch July 22, 2024 14:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

FIX: More VeRA tests, fix tests, more checks #1900

FIX: More VeRA tests, fix tests, more checks #1900

BenjaminBossan commented Jul 2, 2024

HuggingFaceDocBuilderDev commented Jul 2, 2024

BenjaminBossan commented Jul 2, 2024

dkopi left a comment

dkopi Jul 3, 2024

dkopi Jul 3, 2024

dkopi Jul 3, 2024

dkopi Jul 3, 2024

BenjaminBossan commented Jul 4, 2024 •

edited

Loading

dkopi commented Jul 4, 2024 •

edited

Loading

BenjaminBossan commented Jul 4, 2024

sayakpaul left a comment

sayakpaul Jul 22, 2024

BenjaminBossan Jul 22, 2024

sayakpaul Jul 22, 2024

BenjaminBossan Jul 22, 2024

sayakpaul Jul 22, 2024

BenjaminBossan Jul 22, 2024

BenjaminBossan commented Jul 22, 2024

sayakpaul commented Jul 22, 2024

FIX: More VeRA tests, fix tests, more checks #1900

FIX: More VeRA tests, fix tests, more checks #1900

Conversation

BenjaminBossan commented Jul 2, 2024

HuggingFaceDocBuilderDev commented Jul 2, 2024

BenjaminBossan commented Jul 2, 2024

dkopi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BenjaminBossan commented Jul 4, 2024 • edited Loading

dkopi commented Jul 4, 2024 • edited Loading

BenjaminBossan commented Jul 4, 2024

sayakpaul left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BenjaminBossan commented Jul 22, 2024

sayakpaul commented Jul 22, 2024

BenjaminBossan commented Jul 4, 2024 •

edited

Loading

dkopi commented Jul 4, 2024 •

edited

Loading