Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Better message when DataLoader is wrong #1131

Closed
stathius opened this issue Mar 12, 2020 · 2 comments · Fixed by #1280
Closed

Better message when DataLoader is wrong #1131

stathius opened this issue Mar 12, 2020 · 2 comments · Fixed by #1280
Assignees
Labels
bug Something isn't working let's do it! approved to implement
Milestone

Comments

@stathius
Copy link

stathius commented Mar 12, 2020

On the verge between bug and improvement.

There was a bug in my Validation DataLoader and was returning irrelevant staff. Accidentally the length was 0. Probably an edge case combination. The error I was getting during the validation sanity check was quite cryptic:

Traceback (most recent call last):
  File "UNet_WaveProp.py", line 174, in <module>
    trainer.fit(model)
  File "/mnt/RDS/home/code/pytorch-lightning/pytorch_lightning/trainer/trainer.py", line 629, in fit
    self.run_pretrain_routine(model)
  File "/mnt/RDS/home/code/pytorch-lightning/pytorch_lightning/trainer/trainer.py", line 809, in run_pretrain_routine
    False)
  File "/mnt/RDS/home/code/pytorch-lightning/pytorch_lightning/trainer/evaluation_loop.py", line 300, in evaluate
    eval_results = model.validation_epoch_end(outputs)
  File "UNet_WaveProp.py", line 138, in validation_epoch_end
    avg_loss = torch.stack([x['val_loss'] for x in outputs]).mean()
RuntimeError: stack expects a non-empty TensorList

I had to go through the code of pytorch-lightning for few hours to understand what was happening.
Maybe a more informative message would make more sense?

One thing would be to check if the DataLoader's size is 0.
What do you think? I could take a stab at a PR.

@stathius stathius added bug Something isn't working help wanted Open to be worked on labels Mar 12, 2020
@github-actions
Copy link
Contributor

Hi! thanks for your contribution!, great first issue!

@Borda
Copy link
Member

Borda commented Mar 14, 2020

yeah, better error messing always helps, thx and PR is welcome! :]

@Borda Borda added let's do it! approved to implement and removed help wanted Open to be worked on labels Mar 14, 2020
@Borda Borda added this to the 0.7.2 milestone Mar 14, 2020
@Borda Borda modified the milestones: v0.7., v0.7.x Apr 18, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working let's do it! approved to implement
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants