Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bug/sync optimization logger across ranks #1970

Merged
merged 37 commits into from
Feb 25, 2023
Merged

Bug/sync optimization logger across ranks #1970

merged 37 commits into from
Feb 25, 2023

Conversation

bmosaicml
Copy link
Contributor

@bmosaicml bmosaicml commented Feb 15, 2023 •
edited
Loading

What does this PR do?

image

Changing the optimization logger logic so that it correctly aggregates l2_norms, and cosines across ranks.

What issue(s) does this change relate to?

Related to this bug.
https://mosaicml.atlassian.net/browse/CO-1787

Before submitting

  • Have you read the contributor guidelines?
  • Is this change a documentation change or typo fix? If so, skip the rest of this checklist.
  • Was this change discussed/approved in a GitHub issue first? It is much more likely to be merged if so.
  • Did you update any related docs and document your change?
  • Did you update any related tests and add any new tests related to your change? (see testing)
  • Did you run the tests locally to make sure they pass?
  • Did you run pre-commit on your change? (see the pre-commit section of prerequisites)

@bmosaicml bmosaicml force-pushed the bug/sync_optimization_logger_across_ranks branch 5 times, most recently from 243a490 to 0a3067e Compare February 18, 2023 02:35
@bmosaicml bmosaicml force-pushed the bug/sync_optimization_logger_across_ranks branch 8 times, most recently from f89fb3e to 4a24898 Compare February 21, 2023 04:43
@bmosaicml bmosaicml force-pushed the bug/sync_optimization_logger_across_ranks branch from 4a24898 to e49741e Compare February 21, 2023 05:03
@bmosaicml bmosaicml requested a review from a team as a code owner February 24, 2023 22:23
mvpatel2000
mvpatel2000 reviewed Feb 24, 2023
View reviewed changes
Copy link
Contributor

@mvpatel2000 mvpatel2000 left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: can we please start comments with a capital letter to be consistent with style?

composer/optim/decoupled_weight_decay.py Outdated Show resolved Hide resolved
composer/optim/decoupled_weight_decay.py Outdated Show resolved Hide resolved
composer/optim/decoupled_weight_decay.py Outdated Show resolved Hide resolved
mvpatel2000
mvpatel2000 reviewed Feb 24, 2023
View reviewed changes
Copy link
Contributor

@mvpatel2000 mvpatel2000 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

more nits sorry for being annoying lol

tests/callbacks/test_optimizer_monitor.py Outdated Show resolved Hide resolved
tests/callbacks/test_optimizer_monitor.py Outdated Show resolved Hide resolved
mvpatel2000
mvpatel2000 reviewed Feb 24, 2023
View reviewed changes
composer/callbacks/optimizer_monitor.py Show resolved Hide resolved
composer/optim/decoupled_weight_decay.py Outdated Show resolved Hide resolved
composer/optim/decoupled_weight_decay.py Outdated Show resolved Hide resolved
@bmosaicml bmosaicml force-pushed the bug/sync_optimization_logger_across_ranks branch 6 times, most recently from c2118ec to 3f9b979 Compare February 25, 2023 00:03
@bmosaicml bmosaicml force-pushed the bug/sync_optimization_logger_across_ranks branch from f0d386a to e8fb131 Compare February 25, 2023 00:28
@bmosaicml bmosaicml merged commit a3189d1 into mosaicml:dev Feb 25, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mvpatel2000 mvpatel2000 mvpatel2000 approved these changes

@dakinggg dakinggg Awaiting requested review from dakinggg

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@bmosaicml @mvpatel2000 @eracah