Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

conf_mat semaphore leak #61

Closed
lvoegtlin opened this issue Oct 8, 2021 · 0 comments · Fixed by #68
Closed

conf_mat semaphore leak #61

lvoegtlin opened this issue Oct 8, 2021 · 0 comments · Fixed by #68
Assignees
@lvoegtlin
Labels
bug Something isn't working Config Adding or manipulating a existing config Important Important issue
Milestone
0.1.1

Comments

@lvoegtlin
Copy link
Contributor

Describe the bug
When training big models with a lot of data the conf_mats and f1 heatmap create concurrency problems. This is due to the shared lists they all have which are not synchronized. PL already has the solution for this in the changelog as well as in the master so we will fix it after release of the new version.

Fixed in Lightning-AI/pytorch-lightning#6886

To Reproduce
Run with a conf_mat_*_wandb.yaml callback with the cb55_full_unet.yaml experiment
@lvoegtlin lvoegtlin added bug Something isn't working Important Important issue Config Adding or manipulating a existing config labels Oct 8, 2021
@lvoegtlin lvoegtlin added this to the 0.1.1 milestone Oct 8, 2021
@lvoegtlin lvoegtlin modified the milestones: 0.1.5, 0.1.1 Oct 11, 2021
@lvoegtlin lvoegtlin self-assigned this Oct 12, 2021
lvoegtlin added a commit that referenced this issue Oct 19, 2021
@lvoegtlin
❇️ 🔧 ♻️ conf mat is now a variable in the base task. it works now 🎉 #61
3965a3a
@lvoegtlin lvoegtlin linked a pull request Oct 19, 2021 that will close this issue
Dev fixing conf and f1 heatmap #68
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@lvoegtlin lvoegtlin

Labels
bug Something isn't working Config Adding or manipulating a existing config Important Important issue
Projects
None yet
Milestone
0.1.1
Development

Successfully merging a pull request may close this issue.

Dev fixing conf and f1 heatmap DIVA-DIA/DIVA-DAF
1 participant
@lvoegtlin