don't save config in trainer.log_dir or override it?

[tshu-w]

after first dev_run, config.yaml was save to trainer.log_dir. If I run it again, I got the following error:

ValueError: Refusing to overwrite existing file: /trainer.log_dir/config.yaml

[awaelchli]

Hey!

I think you get this because you use the LightnigCLI is it correct?
It's because config.yaml goes to the current working dir (default_root_dir) because during fast_dev_run we don't save logs and there is no logger.

@mauvilsa @carmocca what's the workflow if we want to force overwrite the config? rm config.yaml and run script again, is that the recommended use?

[mauvilsa]

Maybe it could make sense that we change SaveConfigCallback so that the config file is not saved when doing a fast_dev_run.

If someone does two proper runs I would vote to preserve the current behavior. If someone is reusing the same directory, it means mixing results from different runs which in my opinion is not a good idea. Better that it fails and it is up to the user to remove files or use a different directory, hinting that this is what is recommended. If someone does not like this behavior, they can subclass SaveConfigCallback so that the config is removed beforehand, or disable the config save by setting save_config_callback=None.

The current behavior could be improved by catching that exception so that a better error message is given.

[awaelchli]

Yes, special mode for fast_dev_run could be an option, I like the idea.

For regular runs I agree, it's a safety feature, though I was also a bit confused when I hit that problem as a new user. I didn't know if I should delete the config myself or if I ran it wrong the first time :)

[carmocca]

Yeah, let's do what Mauricio proposes!

[tshu-w]

Yes, I also agree add a special mode for fast_dev_run.

[mauvilsa]

I can create a pull request for this.