File Upload: Detection of duplicate files via md5 found in an existing file is confusing to users.

[posixeleni]

User reported an issue uploading files where no duplicates appeared in the UI but they get an error saying "This file already exists in this dataset. Please upload another file."

See RT https://help.hmdc.harvard.edu/Ticket/Display.html?id=232612

[pdurbin]

https://help.hmdc.harvard.edu/Ticket/Display.html?id=232612 was marked resolved with "Our system looks at the MD5 checksum rather than the file name to determine duplicates."

I'm not sure what development work should be done here (or that this is actually a bug). Should the error explain how the check is performed? Passing back to @posixeleni for feedback.

[bencomp]

One of our partners encountered this issue in our test environment. The error occurs when content (as determined by the checksum apparently) and timestamp are a match to an existing file, but to him it was unclear whether it should be an error to allow multiple files with the same content when the timestamps are different.

I would say that some clarity about "acceptible duplication" is welcome.

[bencomp]

MD5 checksums are not considered sufficient for certain use cases, because it is too easy to create different files with the same checksum. Researchers might not be trying to do this, but you might want to know about it.

If Dataverse doesn't accept 'duplicate' files for storage reasons (e.g. because files on the file system are named using their checksum and you would get a name clash), you could try to link many DataFiles to a single object in storage instead.

[shlake]

This issue is exactly what I was describing and wondering about ("why") in the Dataverse User Community google group (on March 16th):

Restating my concern over duplicate file flagging (actually not just "flagging" but preventing upload).

I am thinking that I am not in the business of questioning why a researcher has "duplicate" files with different file names in their dataset. So is there any work around for dataverse to accept these files?

If a researcher has two files that just happen to contain the same information (the same checksum), I don't think that should stop that file from being uploaded, maybe flagged??. There may be a reason for different filenames w/ same content (such as: used in a script as part of analysis - where the title of the file is important to the script and thus would be important for transparency and understanding of the methodology).

Thanks for listening and welcome feedback and comments.

Edit: Here's the link to the post above: https://groups.google.com/d/msg/dataverse-community/FLnm8-60sOs/gNByf3c4CAAJ

[pdurbin]

Duplicate file detection is a feature that was developed in #357 and if it was designed in a suboptimal way it might be worth looking at that issue for the reasoning behind the feature.

[kcondon]

Made this a suggestion since it works as expected but confusing to users. We may revisit how we detect and communicate this.

[pdurbin]

See also #3571.

[pdurbin]

I agree with @landreev 's comment at #3571 (comment) that we should move toward closing this issue so I'm parking it in Development until the pull request for #2290 is made so we can associated this issue with it.

[kcondon]

Much improved, closing.