Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

external outs: document pushing from remote cache will have no effect #1865

Closed
pared opened this issue Oct 14, 2020 · 6 comments
Closed

external outs: document pushing from remote cache will have no effect #1865

pared opened this issue Oct 14, 2020 · 6 comments
Labels
A: docs Area: user documentation (gatsby-theme-iterative) p2-nice-to-have Less of a priority at the moment. We don't usually deal with this immediately. type: enhancement Something is not clear, small updates, improvement suggestions

Comments

@pared
Copy link
Contributor

pared commented Oct 14, 2020

In iterative/dvc#4686 user got confused, when push for repository with external cache had no effect and did not copy the data to the remote.

We already mention that we need an external cache for external outputs. We should probably include the information that this cache is taking care of versioning those outputs, and we assume that the user will not want to move external outputs from the already configured cache.

@shcheklein shcheklein added A: docs Area: user documentation (gatsby-theme-iterative) type: enhancement Something is not clear, small updates, improvement suggestions p2-nice-to-have Less of a priority at the moment. We don't usually deal with this immediately. good first issue Good for newcomers hacktoberfest labels Oct 14, 2020
@jorgeorpinel
Copy link
Contributor

jorgeorpinel commented Oct 17, 2020

We need an external cache for external dependencies? This cache is taking care of versioning those dependencies? We mean outputs, right? 🙂

assume that the user will not want to move external outputs from the already configured cache.

Indeed thit is not expressed in docs. I'm not sure I knew this before actually... So you can't push external outputs? Will --store work to change this behavior? (See iterative/dvc/pull/4581)

Thanks @pared

@pared
Copy link
Contributor Author

pared commented Oct 19, 2020

@jorgeorpinel Yeah, I meant outputs, sorry for the confusion.
As to iterative/dvc/pull/4581 : I don't think it is something related to current issue: OP's problem was inability of pushing external outputs from external cache to remote - while iterative/dvc/pull/4581 is about checkout and import.

@jorgeorpinel
Copy link
Contributor

jorgeorpinel commented Oct 20, 2020

@pared Just to confirm, so there's no way to push external outputs, right?

I think iterative/dvc/pull/4581 is definitely related: it introduces a store: true option to outs in dvc.yaml that means that they can be pushed to remote storage. The value would be true by default for all outputs except if they are imports according to that issue. But you are bringing up another case where it seems the default value would always be false...

@pared
Copy link
Contributor Author

pared commented Oct 22, 2020

@jorgeorpinel

Just to confirm, so there's no way to push external outputs, right?

Yes, that is true.

So as to iterative/dvc#4581: This is about data that is stored in local cache. Essentialy this change lets you decide whether you want to synchronize your import -ed data with your remote. The problem from iterative/dvc#4686 is about external output, that has been saved in external cache. This is a special situation (pushing data from external cache to remote), which, as mentioned in discussion is not being supported by DVC. So even after --store, this scenario will not be supported.

@jorgeorpinel
Copy link
Contributor

So even after --store, this scenario will not be supported.

Great, thanks. I just wanted to know the answer, so we don't give users false expectations when we introduce store: false. (To a user pushing is pushing, whether it's from a local or external cache.)

@shcheklein shcheklein changed the title Document pushing from remote cache will have no effect external outs: document pushing from remote cache will have no effect Feb 5, 2021
@jorgeorpinel
Copy link
Contributor

So iterative/dvc/pull/4581 isn't progressing but a prominent disclaimer was added in #2104, which addresses this issue, I think.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A: docs Area: user documentation (gatsby-theme-iterative) p2-nice-to-have Less of a priority at the moment. We don't usually deal with this immediately. type: enhancement Something is not clear, small updates, improvement suggestions
Projects
None yet
Development

No branches or pull requests

3 participants