Testing Varnet by splitting knee validation dataset

[Sarah-2021-scu]

Hi Mmuckley,

As suggested in 293 I am trying to test the varnet demo version for knee data by splitting the validation dataset but the test mode does not work. Is there some pre-processing that needs to be done on the validation data to run it in test mode?

[mmuckley]

Hello @Sarah-2021-scu, could you share a stack trace of the error?

One funky thing about the code is the data transform keys on whether to run in train mode or val mode based on whether there is a mask in the .h5 file. Since you're running on the validation set, there should be no mask and the transform will assume in validation mode. Is this the type of error you're seeing?

[Sarah-2021-scu]

It keeps on looking for the old test files. Full stack trace below.

Global seed set to 42
/home/sarah/anaconda3/envs/fastmri/lib/python3.9/site-packages/torchmetrics/utilities/prints.py:36: UserWarning: Torchmetrics v0.9 introduced a new argument class property called full_state_update that has
not been set for this class (DistributedMetricSum). The property determines if update by
default needs access to the full metric state. If this is not the case, significant speedups can be
achieved and we recommend setting this to False.
We provide an checking function
from torchmetrics.utilities import check_forward_no_full_state
that can be used to check if the full_state_update=True (old and potential slower behaviour,
default for now) or if full_state_update=False can be used safely.

warnings.warn(*args, **kwargs)
/home/sarah/anaconda3/envs/fastmri/lib/python3.9/site-packages/pytorch_lightning/trainer/connectors/checkpoint_connector.py:51: LightningDeprecationWarning: Setting Trainer(resume_from_checkpoint=) is deprecated in v1.5 and will be removed in v1.7. Please pass Trainer.fit(ckpt_path=) directly instead.
rank_zero_deprecation(
GPU available: True, used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
Global seed set to 42
Global seed set to 42
Initializing distributed: GLOBAL_RANK: 1, MEMBER: 2/2
Global seed set to 42
Initializing distributed: GLOBAL_RANK: 0, MEMBER: 1/2

distributed_backend=nccl
All distributed processes registered. Starting with 2 processes

LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]
LOCAL_RANK: 1 - CUDA_VISIBLE_DEVICES: [0,1]
Testing: 0it [00:00, ?it/s]Traceback (most recent call last):
File "/data/sarah/fastMRI/fastmri_examples/varnet/train_varnet_demo.py", line 197, in
run_cli()
File "/data/sarah/fastMRI/fastmri_examples/varnet/train_varnet_demo.py", line 193, in run_cli
cli_main(args)
File "/data/sarah/fastMRI/fastmri_examples/varnet/train_varnet_demo.py", line 75, in cli_main
trainer.test(model, datamodule=data_module)
File "/home/sarah/anaconda3/envs/fastmri/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 938, in test
return self._call_and_handle_interrupt(self._test_impl, model, dataloaders, ckpt_path, verbose, datamodule)
File "/home/sarah/anaconda3/envs/fastmri/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 723, in _call_and_handle_interrupt
return trainer_fn(*args, **kwargs)
File "/home/sarah/anaconda3/envs/fastmri/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 985, in _test_impl
results = self._run(model, ckpt_path=self.ckpt_path)
File "/home/sarah/anaconda3/envs/fastmri/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1236, in _run
results = self._run_stage()
File "/home/sarah/anaconda3/envs/fastmri/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1320, in _run_stage
return self._run_evaluate()
File "/home/sarah/anaconda3/envs/fastmri/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1365, in _run_evaluate
eval_loop_results = self._evaluation_loop.run()
File "/home/sarah/anaconda3/envs/fastmri/lib/python3.9/site-packages/pytorch_lightning/loops/base.py", line 204, in run
self.advance(*args, **kwargs)
File "/home/sarah/anaconda3/envs/fastmri/lib/python3.9/site-packages/pytorch_lightning/loops/dataloader/evaluation_loop.py", line 154, in advance
dl_outputs = self.epoch_loop.run(self._data_fetcher, dl_max_batches, kwargs)
File "/home/sarah/anaconda3/envs/fastmri/lib/python3.9/site-packages/pytorch_lightning/loops/base.py", line 204, in run
self.advance(*args, **kwargs)
File "/home/sarah/anaconda3/envs/fastmri/lib/python3.9/site-packages/pytorch_lightning/loops/epoch/evaluation_epoch_loop.py", line 112, in advance
batch = next(data_fetcher)
File "/home/sarah/anaconda3/envs/fastmri/lib/python3.9/site-packages/pytorch_lightning/utilities/fetching.py", line 184, in next
return self.fetching_function()
File "/home/sarah/anaconda3/envs/fastmri/lib/python3.9/site-packages/pytorch_lightning/utilities/fetching.py", line 259, in fetching_function
self._fetch_next_batch(self.dataloader_iter)
File "/home/sarah/anaconda3/envs/fastmri/lib/python3.9/site-packages/pytorch_lightning/utilities/fetching.py", line 273, in _fetch_next_batch
batch = next(iterator)
File "/home/sarah/anaconda3/envs/fastmri/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 530, in next
data = self._next_data()
File "/home/sarah/anaconda3/envs/fastmri/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 1224, in _next_data
return self._process_data(data)
File "/home/sarah/anaconda3/envs/fastmri/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 1250, in _process_data
data.reraise()
File "/home/sarah/anaconda3/envs/fastmri/lib/python3.9/site-packages/torch/_utils.py", line 457, in reraise
raise exception
FileNotFoundError: Caught FileNotFoundError in DataLoader worker process 0.
Original Traceback (most recent call last):
File "/home/sarah/anaconda3/envs/fastmri/lib/python3.9/site-packages/torch/utils/data/_utils/worker.py", line 287, in _worker_loop
data = fetcher.fetch(index)
File "/home/sarah/anaconda3/envs/fastmri/lib/python3.9/site-packages/torch/utils/data/_utils/fetch.py", line 49, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/sarah/anaconda3/envs/fastmri/lib/python3.9/site-packages/torch/utils/data/_utils/fetch.py", line 49, in
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/data/sarah/fastMRI/fastmri/data/mri_data.py", line 383, in getitem
with h5py.File(fname, "r") as hf:
File "/home/sarah/anaconda3/envs/fastmri/lib/python3.9/site-packages/h5py/_hl/files.py", line 533, in init
fid = make_fid(name, mode, userblock_size, fapl, fcpl, swmr=swmr)
File "/home/sarah/anaconda3/envs/fastmri/lib/python3.9/site-packages/h5py/_hl/files.py", line 226, in make_fid
fid = h5f.open(name, flags, fapl=fapl)
File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
File "h5py/h5f.pyx", line 106, in h5py.h5f.open
FileNotFoundError: [Errno 2] Unable to open file (unable to open file: name = '/data/sarah/fastmri_knee_data/multicoil_test/file1000082.h5', errno = 2, error message = 'No such file or directory', flags = 0, o_flags = 0)

Traceback (most recent call last):
File "/data/sarah/fastMRI/fastmri_examples/varnet/train_varnet_demo.py", line 197, in
run_cli()
File "/data/sarah/fastMRI/fastmri_examples/varnet/train_varnet_demo.py", line 193, in run_cli
cli_main(args)
File "/data/sarah/fastMRI/fastmri_examples/varnet/train_varnet_demo.py", line 75, in cli_main
trainer.test(model, datamodule=data_module)
File "/home/sarah/anaconda3/envs/fastmri/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 938, in test
return self._call_and_handle_interrupt(self._test_impl, model, dataloaders, ckpt_path, verbose, datamodule)
File "/home/sarah/anaconda3/envs/fastmri/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 721, in _call_and_handle_interrupt
return self.strategy.launcher.launch(trainer_fn, *args, trainer=self, **kwargs)
File "/home/sarah/anaconda3/envs/fastmri/lib/python3.9/site-packages/pytorch_lightning/strategies/launchers/subprocess_script.py", line 93, in launch
return function(*args, **kwargs)
File "/home/sarah/anaconda3/envs/fastmri/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 985, in _test_impl
results = self._run(model, ckpt_path=self.ckpt_path)
File "/home/sarah/anaconda3/envs/fastmri/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1236, in _run
results = self._run_stage()
File "/home/sarah/anaconda3/envs/fastmri/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1320, in _run_stage
return self._run_evaluate()
File "/home/sarah/anaconda3/envs/fastmri/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1365, in _run_evaluate
eval_loop_results = self._evaluation_loop.run()
File "/home/sarah/anaconda3/envs/fastmri/lib/python3.9/site-packages/pytorch_lightning/loops/base.py", line 204, in run
self.advance(*args, **kwargs)
File "/home/sarah/anaconda3/envs/fastmri/lib/python3.9/site-packages/pytorch_lightning/loops/dataloader/evaluation_loop.py", line 154, in advance
dl_outputs = self.epoch_loop.run(self._data_fetcher, dl_max_batches, kwargs)
File "/home/sarah/anaconda3/envs/fastmri/lib/python3.9/site-packages/pytorch_lightning/loops/base.py", line 204, in run
self.advance(*args, **kwargs)
File "/home/sarah/anaconda3/envs/fastmri/lib/python3.9/site-packages/pytorch_lightning/loops/epoch/evaluation_epoch_loop.py", line 112, in advance
batch = next(data_fetcher)
File "/home/sarah/anaconda3/envs/fastmri/lib/python3.9/site-packages/pytorch_lightning/utilities/fetching.py", line 184, in next
return self.fetching_function()
File "/home/sarah/anaconda3/envs/fastmri/lib/python3.9/site-packages/pytorch_lightning/utilities/fetching.py", line 259, in fetching_function
self._fetch_next_batch(self.dataloader_iter)
File "/home/sarah/anaconda3/envs/fastmri/lib/python3.9/site-packages/pytorch_lightning/utilities/fetching.py", line 273, in _fetch_next_batch
batch = next(iterator)
File "/home/sarah/anaconda3/envs/fastmri/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 530, in next
data = self._next_data()
File "/home/sarah/anaconda3/envs/fastmri/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 1224, in _next_data
return self._process_data(data)
File "/home/sarah/anaconda3/envs/fastmri/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 1250, in _process_data
data.reraise()
File "/home/sarah/anaconda3/envs/fastmri/lib/python3.9/site-packages/torch/_utils.py", line 457, in reraise
raise exception
FileNotFoundError: Caught FileNotFoundError in DataLoader worker process 0.
Original Traceback (most recent call last):
File "/home/sarah/anaconda3/envs/fastmri/lib/python3.9/site-packages/torch/utils/data/_utils/worker.py", line 287, in _worker_loop
data = fetcher.fetch(index)
File "/home/sarah/anaconda3/envs/fastmri/lib/python3.9/site-packages/torch/utils/data/_utils/fetch.py", line 49, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/sarah/anaconda3/envs/fastmri/lib/python3.9/site-packages/torch/utils/data/_utils/fetch.py", line 49, in
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/data/sarah/fastMRI/fastmri/data/mri_data.py", line 383, in getitem
with h5py.File(fname, "r") as hf:
File "/home/sarah/anaconda3/envs/fastmri/lib/python3.9/site-packages/h5py/_hl/files.py", line 533, in init
fid = make_fid(name, mode, userblock_size, fapl, fcpl, swmr=swmr)
File "/home/sarah/anaconda3/envs/fastmri/lib/python3.9/site-packages/h5py/_hl/files.py", line 226, in make_fid
fid = h5f.open(name, flags, fapl=fapl)
File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
File "h5py/h5f.pyx", line 106, in h5py.h5f.open
FileNotFoundError: [Errno 2] Unable to open file (unable to open file: name = '/data/sarah/fastmri_knee_data/multicoil_test/file1000082.h5', errno = 2, error message = 'No such file or directory', flags = 0, o_flags = 0)

Testing: 0it [00:03, ?it/s]

[mmuckley]

Hello @Sarah-2021-scu, this is the default configuration. If you want to test on a different directory, you'll need to use the --test_path argument here.

This alone may not be enough, since those files will not have masks, so you should also considering modifying the VarNetDataTransform to fit your use case.

[Sarah-2021-scu]

I am testing on the default directory, only copying new files into it, and still getting that error.

[mmuckley]

@Sarah-2021-scu you are putting the default directory in your code but you haven't put any files there - that's what the error message says "No such file or directory". Those files don't exist.