Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix] some param will be set to optimizer twice when using MLM transformer heads #965

Open
wants to merge 9 commits into
base: main
Choose a base branch
from

Conversation

butterluo
Copy link

summary:
When using MMFTransformer with MLM head, an warning will occur which will be an error in future: "UserWarning: optimizer contains a parameter group with duplicate parameters; in future, this will cause an error; see github.com/pytorch/pytorch/issues/40967 for more information"
This is cuase by get_optimizer_parameters() function will get some parameters twice if these parameters which belong to backbone were tied to head in head's tie_weights() method

Tested locally

Thanks for your contribution!

If you're sending a large PR (e.g., >50 lines), please open an issue first about
the feature/bug, and indicate how you want to contribute.

Use contributing guidelines before opening up the PR to follow MMF style guidelines.

…ch will be an error in future:

    UserWarning: optimizer contains a parameter group with duplicate parameters; in future, this will cause an error; see github.com/pytorch/pytorch/issues/40967 for more information

This is cuase by get_optimizer_parameters() function will get some parameters twice if these parameters which belong to backbone were tied to some heads in head's tie_weights() method (eg. the head called MLM)
@facebook-github-bot facebook-github-bot added the CLA Signed Do not delete this pull request or issue due to inactivity. label Jun 10, 2021
Copy link
Contributor

@apsdehal apsdehal left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this change and helping make MMF better. I have left some comments and then this PR should be ready to land.

mmf/models/transformers/base.py Outdated Show resolved Hide resolved
@@ -103,25 +103,13 @@ def build(self):

def get_optimizer_parameters(self, config):
lr = config.optimizer.params.lr

backbone_param_set = set()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's name this trunk_params_set.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your reviewing. I will change my code according to ur suggestion.

mmf/models/transformers/base.py Outdated Show resolved Hide resolved
):
lr_multiplier = config.get("lr_multiplier", 1.0)
if backbone_param_set is None:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this is None, make it an empty list, [].

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your reviewing. I will change my code according to ur suggestion.

):
lr_multiplier = config.get("lr_multiplier", 1.0)
if backbone_param_set is None:
module_param = list(module.named_parameters())
else:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now, you can remove this else condition.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your reviewing. I will change my code according to ur suggestion.

@facebook-github-bot
Copy link
Contributor

@apsdehal has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

@facebook-github-bot
Copy link
Contributor

@butterluo has updated the pull request. You must reimport the pull request before landing.

@facebook-github-bot
Copy link
Contributor

@butterluo has updated the pull request. You must reimport the pull request before landing.

@butterluo
Copy link
Author

Hi @apsdehal can this bug fix code pass your review? And the auto-merging of ci seems was blocked by the issue https://github.com/facebookresearch/mmf/issues/966 ? If the codes have pass your review how can i merge it into master?

Copy link
Contributor

@apsdehal apsdehal left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks good to me. Thanks for making the updates. Can you fix lint issues? I will then work on landing it.

@facebook-github-bot
Copy link
Contributor

@butterluo has updated the pull request. You must reimport the pull request before landing.

@facebook-github-bot
Copy link
Contributor

@apsdehal has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

@facebook-github-bot
Copy link
Contributor

@butterluo has updated the pull request. You must reimport the pull request before landing.

@butterluo
Copy link
Author

butterluo commented Jun 23, 2021

It looks good to me. Thanks for making the updates. Can you fix lint issues? I will then work on landing it.

@apsdehal I've reformated the the bug fix codes using pycharm. Can it fix the lint issues?

@facebook-github-bot
Copy link
Contributor

@butterluo has updated the pull request. You must reimport the pull request before landing.

@facebook-github-bot
Copy link
Contributor

@butterluo has updated the pull request. You must reimport the pull request before landing.

@butterluo
Copy link
Author

It looks good to me. Thanks for making the updates. Can you fix lint issues? I will then work on landing it.
@apsdehal I formated it using 'black',could u approve the ci workflow?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CLA Signed Do not delete this pull request or issue due to inactivity.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants