Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ESM2 Infer partial batches using predict method #304

Closed
wants to merge 21 commits into from

Conversation

farhadrgh
Copy link
Collaborator

@farhadrgh farhadrgh commented Oct 11, 2024

This depends on changes in NVIDIA/NeMo#10837 to expose drop_last in MegatronDataSampler and allow inference of partial batches.

Note:

The NVIDIA/NeMo#10934 has overridden the changes to expose drop_last. We should now wrap dataloaders with nemo.lightning.data.WrappedDataLoader that can store the mode attribute when creating the dataloader in datamodules. Then drop_last=False if dataloader is in test or predict mode.

Copy link
Collaborator

@jstjohn jstjohn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes! Doesn't this also depend on the latest nemo though? Should you bump nemo to top of tree?

@jstjohn jstjohn mentioned this pull request Oct 11, 2024
@jstjohn
Copy link
Collaborator

jstjohn commented Oct 11, 2024

Also see #302 which bumps the nemo version, is it new enough for your needs?

@yzhang123
Copy link
Collaborator

/build-ci

@farhadrgh
Copy link
Collaborator Author

The NVIDIA/NeMo#10934 has overridden the changes to expose drop_last. We should now wrap dataloaders with nemo.lightning.data.WrappedDataLoader that can store the mode attribute when creating the dataloader in datamodules. Then drop_last=False if dataloader is in test or predict mode.

CC @jstjohn

@farhadrgh
Copy link
Collaborator Author

/build-ci

@farhadrgh farhadrgh changed the title Infer using exposed drop_last in MegatronDataSampler ESM2 Infer partial batches using predict method Oct 30, 2024
Copy link
Collaborator

@pstjohn pstjohn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, but where does this use the new drop_last argument?

@farhadrgh
Copy link
Collaborator Author

LGTM, but where does this use the new drop_last argument?

drop_last is no longer exposed. Megatron Sampler set it to false when dataloader mode is test/predict: https://github.com/NVIDIA/NeMo/pull/10934/files#diff-c423cc7981d5621754bb3e5e509fceaa9adb8537b269b7def042995cfa21d529R81

@farhadrgh farhadrgh self-assigned this Oct 30, 2024
@farhadrgh
Copy link
Collaborator Author

/build-ci

@farhadrgh
Copy link
Collaborator Author

/build-ci

@farhadrgh
Copy link
Collaborator Author

/build-ci

@farhadrgh
Copy link
Collaborator Author

/build-ci

1 similar comment
@farhadrgh
Copy link
Collaborator Author

/build-ci

@farhadrgh
Copy link
Collaborator Author

/build-ci

1 similar comment
@farhadrgh
Copy link
Collaborator Author

/build-ci

@farhadrgh
Copy link
Collaborator Author

/build-ci

@farhadrgh farhadrgh mentioned this pull request Nov 19, 2024
@farhadrgh farhadrgh closed this Nov 19, 2024
auto-merge was automatically disabled November 19, 2024 17:33

Pull request was closed

@pstjohn pstjohn deleted the farhadr/infer_partial_batch branch January 17, 2025 18:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants