Download checkpoints #133
Merged
Download checkpoints #133
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Update engine.py Normalize target for visualization. * Update README.md * Create recurrentvarnet.py * Create config.py * Create __init__.py * Update recurrentvarnet.py * Create recurrentvarnet_engine.py * Update recurrentvarnet.py * Update recurrentvarnet.py * Update recurrentvarnet_engine.py * Update documentation * Create base_recurrentvarnet.yaml * Update README.md * Update model_zoo.md * Option to choose between max/min * Update parse_metrics_log.py Co-authored-by: George Yiasemis <[email protected]>
- formatting updates
tools/parse_metrics_log.py
Outdated
Comment on lines
27
to
38
| print([i.name for i in args.metrics_path.glob("*.pt")]) | ||
| with open(args.metrics_path / "metrics.json", "r") as f: | ||
| data = f.readlines() | ||
| data = [json.loads(_) for _ in data] | ||
|
|
||
| x = np.asarray([(int(_["iteration"]), -_[args.key]) for _ in data if args.key in _]) | ||
| out = x[np.where(x[:, 1] == x[:, 1].max())][0] | ||
|
|
||
| x = np.asarray([(int(_["iteration"]), _[args.key]) for _ in data if args.key in _]) | ||
| if args.max: | ||
| out = x[np.where(x[:, 1] == x[:, 1].max())][0] | ||
| else: | ||
| out = x[np.where(x[:, 1] == x[:, 1].min())][0] | ||
| print(f"{args.key} - {int(out[0])}: {out[1]}") | ||
| print(x[np.where(x[:, 1] == 148520)][0]) |
There was a problem hiding this comment.
Need to rebase again from main
| @@ -281,6 +283,7 @@ def mask_func(self, shape, return_acs=False, seed=None): | |||
|
|
|||
|
|
|||
| class CalgaryCampinasMaskFunc(BaseMaskFunc): | |||
| BASE_URL = "https://s3.aiforoncology.nl/direct-project/calgary_campinas_masks/" | |||
There was a problem hiding this comment.
I feel like https://s3.aiforoncology.nl/direct-project/ could be a global variable for direct given that we store everything here.
| @@ -319,12 +322,16 @@ def mask_func(self, shape, return_acs=False): | |||
| return torch.from_numpy(mask[choice][np.newaxis, ..., np.newaxis]) | |||
|
|
|||
| def __load_masks(self, acceleration): | |||
| masks_path = pathlib.Path(pathlib.Path(__file__).resolve().parent / "calgary_campinas_masks") | |||
| masks_path = DIRECT_CACHE_DIR / "calgary_campinas_masks" | |||
There was a problem hiding this comment.
So you download everything in cache memory. Does that mean that you have to download them everytime?
Comment on lines
+24
to
+30
| # Environmental variables | ||
| DIRECT_ROOT_DIR = pathlib.Path(pathlib.Path(__file__).resolve().parent.parent) | ||
| DIRECT_CACHE_DIR = pathlib.Path(os.environ.get("DIRECT_CACHE_DIR", str(DIRECT_ROOT_DIR))) | ||
| DIRECT_MODEL_DOWNLOAD_DIR = ( | ||
| pathlib.Path(os.environ.get("DIRECT_MODEL_DOWNLOAD_DIR", str(DIRECT_ROOT_DIR))) / "downloaded_models" | ||
| ) | ||
|
|
There was a problem hiding this comment.
Where are these stored?
- Fix tests - Allow path or URL in cfg settings
- Download Calgary Campinas masks when needed, remove from repo - Allow parsing of config from url - Allow loading checkpoint from url
jonasteuwen
force-pushed
the
download-checkpoints
branch
from
bbdc4ca to
99c93bf
Compare
- Download Calgary Campinas masks when needed, remove from repo - Allow parsing of config from url - Allow loading checkpoint from url
- Download Calgary Campinas masks when needed, remove from repo - Allow parsing of config from url - Allow loading checkpoint from url
- Download Calgary Campinas masks when needed, remove from repo - Allow parsing of config from url - Allow loading checkpoint from url
…into download-checkpoints
|
Failure of the tox test is slightly random. Can you review @georgeyiasemis @jonatanferm |
Comment on lines
+301
to
304
| type=file_or_url, | ||
| help="If this value is set to a proper checkpoint when training starts, " | ||
| "the model will be initialized with the weights given. " | ||
| "No other keys in the checkpoint will be loaded. " |
There was a problem hiding this comment.
I would add a bit more elaborate description about loading from directory or downloading from a url. (Maybe in the README?)
There was a problem hiding this comment.
I have adjusted it.
georgeyiasemis
approved these changes
Dec 10, 2021
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a draft for a pull request to download models from URLs. We need some way to also initialize a model easily. Right now loading from the checkpoint as an URL should work.
Needs to check