CLI make_local creates local data source even if the command fails

[forman]

CLI make_local creates a local data source even if the command is cancelled or otherwise fails, e.g. no internet connection, or no source files found. After this, it cannot be corrected.

Reproduce:

1. Call cate ds make_local <id-1> <id-2> <time>
2. Cancel running command, then run again.

Workaround:

Delete concerned local data source manually.

Fix:

If make_local() fails, it should keep all "good" files, but remember that it is incomplete so we can update calling the command again with same data source ID.

[forman]

It is fully legal that the download of local data sources is being interrupted by means of

• networking problems
• disk full
• files not found / renamed / moved
• internal errors / Python exceptions

And it is crucial that the downloading can be continued later where all completely downloaded files are being preserved and incomplete ones are being deleted or overwritten.

An incomplete local data source shall be indicated as such so that we can display this fact in the CLI and GUI.

[JanisGailis]

In addition to this, I'm seeing the following pattern: I include make_local in a notebook, or a script, but then, if I run the script more than once, make_local will raise an error because the given local_name already exists, so I then ignore the error and open the dataset which I know exists.

I'm not sure what would be the most elegant interface to solve this.

[kbernat]

perhaps we need some 'override' option

[JanisGailis]

Yeah, something like that. I thought of 'overwrite', but then, I wouldn't want to download the dataset again and again.

Maybe 'update_local' could also do what 'make_local' does, if the dataset doesn't exist. I'm not really sure how to solve this best.

[mzuehlke]

If make_local is called with the same arguments it should be checked if the local copy is complete:

• if yes return it
• if no continue copying

If make_local is called with different arguments raise an exception like now.
No new parameter needed.

How about this ?

For the GUI / CLI we may need the information that a dataset has only been downloaded partially.

[forman]

I full agree with @mzuehlke

[forman]

Cancellation works fine, but incomplete files remain. Cancelled data source should dissapear without a trace from the user's file system. Reopening because it is incomplete.

[forman]

@kbernat any news here?

[kbernat]

@forman I will review and complete fix yet today..

[kbernat]

fixed in #375