Display deposited (rather than ingested) copy of tabular files

[adam3smith]

This feature request comes out of the discussion on data curation at the 2021 DV community meeting:

Current behavior
When an ingestable tabular file is deposited (.xlsx, .sav, .dta), the default download format (and the displayed file extension) is the ingested .tab version of the file. The original file format is available from the File access menu together with file-level metadata and the explorer tools

Suggested behavior
I suggest that the deposited file format is better suited as the default download format, with .tab (or .tsv as it should be called ;)) being available through the File access menu

Rationale
There are several reasons deposited file formats are preferable:

1. The default display of the ingested file is confusing for depositors, as @amberleahey noted during the discussion.
2. Frequently, deposit format are richer than the extracted .tab. E.g., Excel files may have additional rich text formatting, which makes them easier to ready than their plain text counterparts
3. In some cases, ingest can cause data loss (e.g. for Excel files with multiple tabs, undesirable as those may be). Defaulting to the deposited format somewhat mitigates this, even though it is still problematic.

On a more theoretical level, in the terminology of the OAIS reference mode, we clearly have the SIP (the deposited file) and the AIP (the archived/preservation copy) defined and the question is which of the two is the better DIP. I would argue that is the more commonly useable and often richer data format -- that not just the case for Excel, but also for things like .sav files which include rich metadata that reads nicely not just into SPSS but also into tools like R with appropriate packages.

cc @sbarbosadataverse who was also part of this dicussion

[pdurbin]

There was a similar conversation about the "download all" button in #4000.

Since many repositories include code that expects data files to be in a particular format, it's frustrating that dataverse defaults to downloading data files as .tab.

IMHO, the default should be the original file format, with options for all the others.

This was fixed for "download all" in pull request #4979.

[amberleahey]

I would support renaming this to .tsv if it is in fact not distinct (.tab vs. .tsv), but does the Dataverse software extract metadata and structure the .tab differently than a .tsv?

Regarding the display of the .tab for end users, it's interesting, some users do get confused and they think their original data is gone or less promoted, and the improvement @pdurbin mentioned above is great! So perhaps it's just a matter of improving the labelling of the .tab file in the file listing? Could it be moved to the bottom of the list? Add a label 'Preservation copy'?

[TaniaSchlatter]

"original format" is listed first now:

[adam3smith]

(the .tsv vs. .tab discussion is in #6006, but just to re-iterate, these are literally the same format. If you want Excel to open a .tab file, you change the extension to .tsv)

[amberleahey]

@TaniaSchlatter could the .tab also appear lower in the main file listing window (or have something to differentiate it from other original files?) not just in the download file access window?

[TaniaSchlatter]

@amberleahey, my quick response is that I see this as an opportunity for an automatic file tag, "Original Format" which users could filter on to change the order in the file table.

[BPeuch]

Sounds like this issue can be closed now, doesn't it?

The distinction between original format and .tab in the dropdown menu is very handy!

[adam3smith]

@BPeuch - no, the main request here has not been implemented, although I think there's agreement for it.

While, as @TaniaSchlatter notes, the original format is (now? not sure this is new) listed above the tab/archival format in the dropdown, the default download/display format continues to be .tab. See e.g. on demo.dataverse here: https://demo.dataverse.org/file.xhtml?persistentId=doi:10.70122/FK2/OXQWCP/DU8NM2&version=1.0

[sbarbosadataverse]

Sorry that I didn't chime in earlier (I was @ with the incorrect username). I'll read through the correspondence and see if there's anything I can add...

…

On Mon, Aug 9, 2021, 9:06 AM Sebastian Karcher ***@***.***> wrote: @BPeuch <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_BPeuch&d=DwMCaQ&c=WO-RGvefibhHBZq3fL85hQ&r=8R6PzVqt1PEocauQgZMGXsGz29-nb19M7eqlo1d8EVs&m=yWG4jvxv00NK1dmYAJ_1MZtZDajlNzZpeHBHr1HiKk0&s=tOUCrxbU0M5yUWVXaXo3igXKsKSQM0z6kymVKN058jI&e=> - no, the main request has here has not been implemented, although I think there's agreement for it. While, as @TaniaSchlatter <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_TaniaSchlatter&d=DwMCaQ&c=WO-RGvefibhHBZq3fL85hQ&r=8R6PzVqt1PEocauQgZMGXsGz29-nb19M7eqlo1d8EVs&m=yWG4jvxv00NK1dmYAJ_1MZtZDajlNzZpeHBHr1HiKk0&s=ag5zUpdoV2mIPUx6IEqbSLMhCglLGKpDrNduIQKI1qU&e=> notes, the original format is (now? not sure this is new) listed above the tab/archival format in the dropdown, the default download/display format continues to be .tab. See e.g. on demo.dataverse here: https://demo.dataverse.org/file.xhtml?persistentId=doi:10.70122/FK2/OXQWCP/DU8NM2&version=1.0 <https://urldefense.proofpoint.com/v2/url?u=https-3A__demo.dataverse.org_file.xhtml-3FpersistentId-3Ddoi-3A10.70122_FK2_OXQWCP_DU8NM2-26version-3D1.0&d=DwMCaQ&c=WO-RGvefibhHBZq3fL85hQ&r=8R6PzVqt1PEocauQgZMGXsGz29-nb19M7eqlo1d8EVs&m=yWG4jvxv00NK1dmYAJ_1MZtZDajlNzZpeHBHr1HiKk0&s=IQiu6D3PSYXXfkuu_7wKnoRXiyKpv-RG9fNTmOC-KQo&e=> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_IQSS_dataverse_issues_7956-23issuecomment-2D895206692&d=DwMCaQ&c=WO-RGvefibhHBZq3fL85hQ&r=8R6PzVqt1PEocauQgZMGXsGz29-nb19M7eqlo1d8EVs&m=yWG4jvxv00NK1dmYAJ_1MZtZDajlNzZpeHBHr1HiKk0&s=oi2zrj2biGXXlbnVJg68Bn3BwcH-xyKmwLokBwy5IzY&e=>, or unsubscribe <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AB7P2KUO3VJX7X6MIK3UGM3T37HGNANCNFSM467S33NQ&d=DwMCaQ&c=WO-RGvefibhHBZq3fL85hQ&r=8R6PzVqt1PEocauQgZMGXsGz29-nb19M7eqlo1d8EVs&m=yWG4jvxv00NK1dmYAJ_1MZtZDajlNzZpeHBHr1HiKk0&s=1SlotG94wmj2-OqyEpA1mqBU2NvI2MFUaq9qmJ6wMVI&e=> . Triage notifications on the go with GitHub Mobile for iOS <https://urldefense.proofpoint.com/v2/url?u=https-3A__apps.apple.com_app_apple-2Dstore_id1477376905-3Fct-3Dnotification-2Demail-26mt-3D8-26pt-3D524675&d=DwMCaQ&c=WO-RGvefibhHBZq3fL85hQ&r=8R6PzVqt1PEocauQgZMGXsGz29-nb19M7eqlo1d8EVs&m=yWG4jvxv00NK1dmYAJ_1MZtZDajlNzZpeHBHr1HiKk0&s=xwUeeV1TdvOGxBFrLoEspSgZukS7plRRmFabWqqp1dY&e=> or Android <https://urldefense.proofpoint.com/v2/url?u=https-3A__play.google.com_store_apps_details-3Fid-3Dcom.github.android-26utm-5Fcampaign-3Dnotification-2Demail&d=DwMCaQ&c=WO-RGvefibhHBZq3fL85hQ&r=8R6PzVqt1PEocauQgZMGXsGz29-nb19M7eqlo1d8EVs&m=yWG4jvxv00NK1dmYAJ_1MZtZDajlNzZpeHBHr1HiKk0&s=jL6ZD5Yr5MHbrfOwgRoO6SKE5Epml0u53GM_7dVRqWI&e=> .

[BPeuch]

Oh my bad @adam3smith I did not realize this format was still prominent specifically in file webpages.

[adam3smith]

Same on the dataset page -- .tab is still the default display&download format in the list of files as well

[BPeuch]

That is true. I thought the problem was only about the 'Download' button but I can see the arguments for highlighting the original format. Users who cannot reuse, say, SPSS files will know to look for alternatives (such as a .tsv output) either way.

[sbarbosadataverse]

Have we made any decisions on this? ᐧ

…

On Tue, Aug 10, 2021 at 2:50 AM Benjamin Peuch ***@***.***> wrote: That is true. I thought the problem was only about the 'Download' button but I can see the arguments for highlighting the original format. Users who cannot reuse, say, SPSS files will know to look for alternatives (such as a .tsv output) either way. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_IQSS_dataverse_issues_7956-23issuecomment-2D895774243&d=DwMCaQ&c=WO-RGvefibhHBZq3fL85hQ&r=8R6PzVqt1PEocauQgZMGXsGz29-nb19M7eqlo1d8EVs&m=42YEtY85YyjV7bsILXQjCUGjLq86DSXoRwT5fMa5FJw&s=vMvK_UwK4GWGvC3EB4igxx_Zxqg0LBQp1NbxRqqVITM&e=>, or unsubscribe <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AB7P2KW4Q5SJYGAWTUBL6XLT4DD45ANCNFSM467S33NQ&d=DwMCaQ&c=WO-RGvefibhHBZq3fL85hQ&r=8R6PzVqt1PEocauQgZMGXsGz29-nb19M7eqlo1d8EVs&m=42YEtY85YyjV7bsILXQjCUGjLq86DSXoRwT5fMa5FJw&s=wfGxzp3YdFCc2dS2azqCeYTl7W7s5yY5_ZDBrg6oNas&e=> . Triage notifications on the go with GitHub Mobile for iOS <https://urldefense.proofpoint.com/v2/url?u=https-3A__apps.apple.com_app_apple-2Dstore_id1477376905-3Fct-3Dnotification-2Demail-26mt-3D8-26pt-3D524675&d=DwMCaQ&c=WO-RGvefibhHBZq3fL85hQ&r=8R6PzVqt1PEocauQgZMGXsGz29-nb19M7eqlo1d8EVs&m=42YEtY85YyjV7bsILXQjCUGjLq86DSXoRwT5fMa5FJw&s=EjVlxIkUewqEPgerh-5qV6bVGgw8DTRlgwrg-C9L5-0&e=> or Android <https://urldefense.proofpoint.com/v2/url?u=https-3A__play.google.com_store_apps_details-3Fid-3Dcom.github.android-26utm-5Fcampaign-3Dnotification-2Demail&d=DwMCaQ&c=WO-RGvefibhHBZq3fL85hQ&r=8R6PzVqt1PEocauQgZMGXsGz29-nb19M7eqlo1d8EVs&m=42YEtY85YyjV7bsILXQjCUGjLq86DSXoRwT5fMa5FJw&s=4AGhQcAwaWaXwmHMuBQ1j-QuOVRRTuY2DsOSXgOmyTo&e=> .

-- Sonia Barbosa Manager of Data Curation, The Harvard Dataverse Repository Manager of the Murray Research Archive, IQSS The Dataverse Project Data Science Harvard University Harvard Library RDM services: https://hlrdm.library.harvard.edu/network Need to deposit data? Visit http://dataverse.harvard.edu All Harvard Dataverse Repository inquiries should be sent to: ***@***.*** All software inquiries should be sent to: ***@***.*** Interested in sharing sensitive data? Coming soon to Harvard Dataverse: http://datatags.org/ All test dataverses should be created in our demo environment: https://demo.dataverse.org/ Join our Dataverse Community! https://groups.google.com/forum/#!forum/dataverse-communit <https://groups.google.com/forum/#!forum/dataverse-community>y

[qqmyers]

I've been looking into this at QDR and it looks relatively straight forward to change the display in the file table, on the file page, and in the file citations - i.e. giving the name, size, checksum, content type of the original. And the download menu has already been changed to show the original as the first option.

The one aspect of functionality I've run into so far where this change is somewhat problematic is in allowing the filename to be edited (for a given dataset version). Currently, what you change is the name of the ingested file (i.e. the *.tab version) and if you change the file extension, you are changing the extension of the tab file. The original file gets <newname>.<original extension>. The mimetype itself is not updated (and can't otherwise be changed) if you do update the extension.

I think this could be changed so you edit the name of the original instead - but that would involve dealing with any existing files where the tab version has had the extension changed which would be more work.

Alternately, things could be changed so you can't edit extensions. That seems like it could work for QDR, but may not be acceptable generally.

So - to move this forward - any thoughts on whether being able to change the extensions on the original and/or tab versions are needed, or how to address legacy data if we just flip to allowing the original file's extension to be changed, etc. are welcome.

[pdurbin]

What options do we have for changing the filename and extension via API? If we forbid changing the extension via UI, can it be changed via API? Is there a workaround, I mean.

(By the way, it would be kind of cool if you could change the extension and have Dataverse redetect the mimetype.)

[DS-INRAE]

I created #10067 to dissociate the specific aspect of editing extensions apart and make it more granular and have the discussion on this aspect there 😃

[qqmyers]

FWIW: QDR did implement this, making the extension non-editable.

[vkush]

Same on the dataset page -- .tab is still the default display&download format in the list of files as well

As download behaviour was already extended, do we also have some updates for display format to show the originally deposited file format (e.g. .csv) instead of .tab?

[pdurbin]

@vkush this issue is still open so, no, the .tab is still always shown. Pull requests are welcome! ❤️

[qqmyers]

If the way QDR implemented is acceptable, I can make a PR of that (or if that helps as a starting point). Presumably this could/should have an SPA issue as well?

[pdurbin]

@qqmyers please! ❤️