Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Short flag for persisted outputs #1884

Closed
guysmoilov opened this issue Apr 14, 2019 · 4 comments
Closed

Short flag for persisted outputs #1884

guysmoilov opened this issue Apr 14, 2019 · 4 comments
Labels
enhancement Enhances DVC p2-medium Medium priority, should be done, but less important

Comments

@guysmoilov
Copy link
Contributor

From: #1214
Since this is will probably be an often-used feature, I believe it warrants its own single-letter command line flag. Maybe -p -P?

@efiop
Copy link
Contributor

efiop commented Apr 14, 2019

@guysmoilov We are still debating whether or not persist should be an allowed hack or not. Here is another question that we are thinking about right now while working on #1877 with @MrOutis . Imagine you have persist outputs for some stage in your pipeline. Should dvc repro consider them as changed every time and re-run them or should it not? If we are talking about continuing from a checkpoint, it feels like persist should only be used until you are happy with the results, after which you should unset that and maybe even re-run your pipeline to ensure that it was indeed produced by your script, and so dvc repro should always consider such stages as changed and re-run them. On the other hand, if we are using such persistent output as a kind of a log file, then it shouldn't trigger dvc repro. What are your thoughts about that as a user?

@guysmoilov
Copy link
Contributor Author

I think it definitely shouldn't be "always changed".
Changed implies that the dependencies are different, not that I want to resume from a checkpoint.
When I want to resume from a checkpoint, I just force the re-running of the stage.
tbh I also am not a huge fan of the "stage with command and no deps is always considered changed" approach, it's annoying when I want a stage to import raw data using wget from a site that has no etag. I think such a "callback stage" should be a user-controlled decision. But maybe I'm outdated here? I remember there were feature requests regarding callback stages.

@efiop
Copy link
Contributor

efiop commented Apr 14, 2019

@guysmoilov Great point! I agree with that.

We have an issue for callback stages already https://github.com/iterative/dvc/issues?utf8=%E2%9C%93&q=is%3Aissue+is%3Aopen+callback . We'll try to permanently fix it in 1.0.

@efiop efiop added c1-quick-fix enhancement Enhances DVC p2-medium Medium priority, should be done, but less important labels Apr 14, 2019
@efiop
Copy link
Contributor

efiop commented Dec 7, 2020

Closing due to a lack of interest from the community.

@efiop efiop closed this as completed Dec 7, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Enhances DVC p2-medium Medium priority, should be done, but less important
Projects
None yet
Development

No branches or pull requests

2 participants