Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix tokenizer file caching where torch.distributed may not be initialized yet #7061

Merged
merged 2 commits into from
Aug 1, 2023
Merged

Fix tokenizer file caching where torch.distributed may not be initialized yet #7061

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
e851657
Fix rank where torch.distributed may not be initialized yet and would…
findkim Jul 18, 2023
092507d
Merge branch 'main' into findkim/fix-multinode-tokenizer-cache
Davood-M Jul 27, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions nemo/collections/nlp/modules/common/megatron/megatron_utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
import wget
from torch.hub import _get_torch_home

from nemo.utils import get_rank, logging
from nemo.utils import logging

__all__ = [
"get_megatron_lm_model",
Expand Down Expand Up @@ -203,7 +203,7 @@ def _download(path: str, url: str):
if url is None:
return None

if get_rank.is_global_rank_zero() and not os.path.exists(path):
if (not torch.distributed.is_initialized() or torch.distributed.get_rank() == 0) and not os.path.exists(path):
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a reason you are not using get_rank from nemo_utils?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found that is_global_rank_zero() requires the cache dir to be accessible to all nodes on a shared volume and needs TORCH_HOME env to be set or defaults to /root/.cache/torch which errors since that's local to Global rank 0.

This change reverts it closer to the original intent before PR #7016

os.makedirs(MEGATRON_CACHE, exist_ok=True)
logging.info(f"Downloading from {url} to {path}")
downloaded_path = wget.download(url)
Expand Down