Multiple dvc exp run --temp runs

[ciaochiaociao]

Dear DVC, I am very happy to join this community. I have been working with it and I really love this tool to manage my ML experiments. However, currently (2.0.11 as I post this issue), running multiple dvc exp run --temp will have

ERROR: Unable to acquire lock. Most likely another DVC process is running or was terminated abruptly. Check the page <https://dvc.org/doc/user-guide/troubleshooting#lock-issue> for other possible reasons and to learn how to resolve this.

Is it possible to do that? Or are you including this feature in your schedule?

The reason why I want this is

I want to run some other experiments when I was running one. And I know I can use dvc exp run --run-all. But sometimes it is hard for me to plan ahead all experiments.

[pmrowla]

Unfortunately it's not currently possible to do this, but we do have plans to eventually improve the queue/run-all functionality so that you can modify the queue in place as needed.

Closing this as a duplicate of #5615 (this issue is the same as the suggested queue --add).

[dberenbaum]

@pmrowla It makes sense to put this under the umbrella of #5615, but the description for queue --add is the same as run --queue, which would be insufficient to address this issue. Seems like this is an additional feature that might make sense as part of queue --add?

[pmrowla]

The semantics of how we want to handle queuing an experiment can be discussed in the other ticket, but the underlying issue here is that doing any dvc exp run currently locks the repo workspace until that full command finishes (whether it is a single dvc exp run --temp call or a batch of experiments via dvc exp run --run-all).

Making queueing and tempdir execution work so that it does not lock the repo is essentially a prerequisite for everything in #5615, so I don't think this is a separate issue. Once we have proper queue/executor management, exp run --temp will end up being deprecated, as the preferred method to "stage" a tempdir run would be via the hypothetical queue --add command instead.