tied weights for adapters #6928

arendu · 2023-06-27T21:36:47Z

What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

Collection: [Note which collection this PR will affect]

Changelog

Add specific line by line info of high level changes in this PR.

Usage

You can potentially add a usage example below

# Add a code snippet demonstrating how to use this

Before your PR is "Ready for review"

Pre checks:

Make sure you read and followed Contributor guidelines
Did you write any new necessary tests?
Did you add or update any necessary documentation?
Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
- Reviewer: Does the PR have correct import guards for all optional libraries?

PR Type:

New Feature
Bugfix
Documentation

If you haven't finished some of the above items you can still open "Draft" PR.

Who can review?

Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.

Additional Information

Related to # (issue)

Signed-off-by: arendu <[email protected]>

for more information, see https://pre-commit.ci

Signed-off-by: arendu <[email protected]>

for more information, see https://pre-commit.ci

…into adithyare/tied_lora

Signed-off-by: arendu <[email protected]>

…into adithyare/tied_lora

Signed-off-by: arendu <[email protected]>

for more information, see https://pre-commit.ci

Signed-off-by: arendu <[email protected]>

…into adithyare/tied_lora

Signed-off-by: arendu <[email protected]>

for more information, see https://pre-commit.ci

Signed-off-by: arendu <[email protected]>

for more information, see https://pre-commit.ci

Signed-off-by: arendu <[email protected]>

for more information, see https://pre-commit.ci

Signed-off-by: arendu <[email protected]>

…into adithyare/tied_lora

github-actions · 2023-07-25T02:00:39Z

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

Signed-off-by: arendu <[email protected]>

for more information, see https://pre-commit.ci

nemo/collections/nlp/models/language_modeling/megatron_gpt_peft_models.py

+                adapter_l = layer.get_adapter_module(adapter_name)
+                adapter_0 = layer0.get_adapter_module(adapter_name)
+                if hasattr(adapter_0, "layer_norm"):
+                    lnorm = adapter_0.layer_norm


nemo/collections/nlp/models/language_modeling/megatron_gpt_peft_models.py

+                adapter_0 = layer0.self_attention.get_adapter_module(adapter_name)
+                position_embeddings_0 = None
+                if adapter_0.position_embedding_strategy:
+                    position_embeddings_0 = adapter_0.position_embeddings


Signed-off-by: arendu <[email protected]>

for more information, see https://pre-commit.ci

…into adithyare/tied_lora

Signed-off-by: arendu <[email protected]>

aklife97

LGTM, thank you!

* wip Signed-off-by: arendu <[email protected]> * wip Signed-off-by: arendu <[email protected]> * tied lora weights Signed-off-by: arendu <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * lora and adapter tying Signed-off-by: arendu <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * layer selection wip Signed-off-by: arendu <[email protected]> * added layer selection Signed-off-by: arendu <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * make ln optional Signed-off-by: arendu <[email protected]> * layer section Signed-off-by: arendu <[email protected]> * small dim pos emb Signed-off-by: arendu <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * adapter w/o layer norm and weight tying Signed-off-by: arendu <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * eval works with all pos embeddings strategy Signed-off-by: arendu <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * update Signed-off-by: arendu <[email protected]> * mlp transform of pos embeddings Signed-off-by: arendu <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * zero init position bias Signed-off-by: arendu <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * merge Signed-off-by: arendu <[email protected]> * minor fix Signed-off-by: arendu <[email protected]> --------- Signed-off-by: arendu <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: dorotat <[email protected]>

* wip Signed-off-by: arendu <[email protected]> * wip Signed-off-by: arendu <[email protected]> * tied lora weights Signed-off-by: arendu <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * lora and adapter tying Signed-off-by: arendu <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * layer selection wip Signed-off-by: arendu <[email protected]> * added layer selection Signed-off-by: arendu <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * make ln optional Signed-off-by: arendu <[email protected]> * layer section Signed-off-by: arendu <[email protected]> * small dim pos emb Signed-off-by: arendu <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * adapter w/o layer norm and weight tying Signed-off-by: arendu <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * eval works with all pos embeddings strategy Signed-off-by: arendu <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * update Signed-off-by: arendu <[email protected]> * mlp transform of pos embeddings Signed-off-by: arendu <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * zero init position bias Signed-off-by: arendu <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * merge Signed-off-by: arendu <[email protected]> * minor fix Signed-off-by: arendu <[email protected]> --------- Signed-off-by: arendu <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

arendu added 3 commits June 23, 2023 17:50

wip

78366ff

Signed-off-by: arendu <[email protected]>

wip

02e824a

Signed-off-by: arendu <[email protected]>

tied lora weights

67f35ce

Signed-off-by: arendu <[email protected]>

github-actions bot added the NLP label Jun 27, 2023

arendu changed the title ~~Adithyare/tied lora~~ tied weights for adapters Jun 27, 2023

pre-commit-ci bot and others added 20 commits June 27, 2023 21:37

[pre-commit.ci] auto fixes from pre-commit.com hooks

eebe3da

for more information, see https://pre-commit.ci

lora and adapter tying

fef1763

Signed-off-by: arendu <[email protected]>

[pre-commit.ci] auto fixes from pre-commit.com hooks

7676dc9

for more information, see https://pre-commit.ci

Merge branch 'main' into adithyare/tied_lora

b38de10

Merge branch 'adithyare/tied_lora' of https://github.com/NVIDIA/NeMo …

5adebba

…into adithyare/tied_lora

layer selection wip

db949f3

Signed-off-by: arendu <[email protected]>

Merge branch 'adithyare/tied_lora' of https://github.com/NVIDIA/NeMo …

e4cd1dc

…into adithyare/tied_lora

added layer selection

356ec63

Signed-off-by: arendu <[email protected]>

[pre-commit.ci] auto fixes from pre-commit.com hooks

96c936b

for more information, see https://pre-commit.ci

make ln optional

f2c0017

Signed-off-by: arendu <[email protected]>

Merge branch 'adithyare/tied_lora' of https://github.com/NVIDIA/NeMo …

81d081f

…into adithyare/tied_lora

layer section

1188a16

Signed-off-by: arendu <[email protected]>

small dim pos emb

f5dcf33

Signed-off-by: arendu <[email protected]>

[pre-commit.ci] auto fixes from pre-commit.com hooks

ab1e295

for more information, see https://pre-commit.ci

adapter w/o layer norm and weight tying

7430d3e

Signed-off-by: arendu <[email protected]>

[pre-commit.ci] auto fixes from pre-commit.com hooks

fc8278c

for more information, see https://pre-commit.ci

eval works with all pos embeddings strategy

620aabd

Signed-off-by: arendu <[email protected]>

[pre-commit.ci] auto fixes from pre-commit.com hooks

889d475

for more information, see https://pre-commit.ci

update

18e2f37

Signed-off-by: arendu <[email protected]>

Merge branch 'adithyare/tied_lora' of https://github.com/NVIDIA/NeMo …

123f89c

…into adithyare/tied_lora

github-actions bot added the stale label Jul 25, 2023

arendu and others added 2 commits July 27, 2023 13:44

mlp transform of pos embeddings

8c18bc4

Signed-off-by: arendu <[email protected]>

[pre-commit.ci] auto fixes from pre-commit.com hooks

be78026

for more information, see https://pre-commit.ci

github-advanced-security bot found potential problems Jul 27, 2023

View reviewed changes

github-actions bot removed the stale label Jul 28, 2023

arendu and others added 10 commits July 28, 2023 11:48

zero init position bias

368312a

Signed-off-by: arendu <[email protected]>

merged

2d95ff1

Signed-off-by: arendu <[email protected]>

[pre-commit.ci] auto fixes from pre-commit.com hooks

202ca1d

for more information, see https://pre-commit.ci

Merge branch 'main' into adithyare/tied_lora

dd0c9ee

Merge branch 'adithyare/tied_lora' of https://github.com/NVIDIA/NeMo …

82e3a1d

…into adithyare/tied_lora

merge

bea09bb

Signed-off-by: arendu <[email protected]>

Merge branch 'main' into adithyare/tied_lora

e0977c9

Merge branch 'main' into adithyare/tied_lora

0d60d72

Merge branch 'main' into adithyare/tied_lora

86812b5

Merge branch 'main' into adithyare/tied_lora

df26ad7

arendu marked this pull request as ready for review August 15, 2023 04:10

arendu requested a review from aklife97 August 15, 2023 04:10

arendu and others added 5 commits August 16, 2023 10:47

Merge branch 'main' into adithyare/tied_lora

dcc5817

Merge branch 'main' into adithyare/tied_lora

6fc7722

minor fix

90aa911

Signed-off-by: arendu <[email protected]>

Merge branch 'main' into adithyare/tied_lora

631d520

Merge branch 'main' into adithyare/tied_lora

87bd338

aklife97 approved these changes Aug 18, 2023

View reviewed changes

aklife97 merged commit f26b2f1 into main Aug 18, 2023

aklife97 deleted the adithyare/tied_lora branch August 18, 2023 21:13

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

tied weights for adapters #6928

tied weights for adapters #6928

arendu commented Jun 27, 2023

github-actions bot commented Jul 25, 2023

aklife97 left a comment

tied weights for adapters #6928

tied weights for adapters #6928

Conversation

arendu commented Jun 27, 2023

What does this PR do ?

Changelog

Usage

Before your PR is "Ready for review"

Who can review?

Additional Information

github-actions bot commented Jul 25, 2023

aklife97 left a comment

Choose a reason for hiding this comment