Skip to content

Commit

Permalink
add highlights and upgrade steps #10853
Browse files Browse the repository at this point in the history
  • Loading branch information
@pdurbin
pdurbin committed Sep 19, 2024
1 parent c69e783 commit c706372
Showing 1 changed file with 178 additions and 20 deletions.
198 changes: 178 additions & 20 deletions doc/release-notes/6.4-release-notes.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,24 @@ This release brings new features, enhancements, and bug fixes to Dataverse. Than

## Release Highlights

- Users will have DOIs/PIDs reserved for their files as part of file upload instead of at publication time. (Issue #7068, PR #7334)
Dataverse 6.4 fixes a couple important bugs described below and in [a post](https://groups.google.com/g/dataverse-community/c/evn5C-pyrS8/m/JrH9vp47DwAJ) on the mailing list:

## Features
- "Update Current Version" can cause metadata loss
- Publishing breaks designated dataset thumbnail, messes up collection page

New features include:

- All ISO 639-3 languages are now supported
- There is now a button for "Unlink Dataset"
- Users will have DOIs/PIDs reserved for their files as part of file upload instead of at publication time
- Datasets can now have types such as "software" or "workflow"
- Croissant support
- RO-Crate support
- A new Rust library

All this and more can be found below. Read on!

## Features Added

### Full List of ISO 639-3 Languages Now Supported

Expand All @@ -32,10 +47,6 @@ Out of the box, all datasets now have the type "dataset" but superusers can add

For details see [the guides](https://guides.dataverse.org/en/6.4/user/dataset-management.html#dataset-types), #10517 and #10694. Please note that this feature is highly experimental and is expected to [evolve](https://github.com/IQSS/dataverse-pm/issues/307).

### Collection Thumbnail Logo for Featured Collections

Collections can now have a thumbnail logo that is displayed when the collection is configured as a featured collection. If present, this thumbnail logo is shown. Otherwise, the collection logo is shown. Configuration is done under the "Theme" for a collection as explained in [the guides](https://guides.dataverse.org/en/6.4/user/dataverse-management.html#theme). See also #10291 and #10433.

### Croissant Support (Metadata Export)

A new metadata export format called [Croissant](https://github.com/mlcommons/croissant) is now available as an external metadata exporter. It is oriented toward making datasets consumable by machine learning.
Expand All @@ -52,6 +63,10 @@ Dataverse now supports [RO-Crate](https://www.researchobject.org/ro-crate/) as a

An Dataverse API client library for the Rust programming language is now available at https://github.com/gdcc/rust-dataverse and has been added to the [list of client libraries](https://guides.dataverse.org/en/6.4/api/client-libraries.html) in the API Guide. See also #10758.

### Collection Thumbnail Logo for Featured Collections

Collections can now have a thumbnail logo that is displayed when the collection is configured as a featured collection. If present, this thumbnail logo is shown. Otherwise, the collection logo is shown. Configuration is done under the "Theme" for a collection as explained in [the guides](https://guides.dataverse.org/en/6.4/user/dataverse-management.html#theme). See also #10291 and #10433.

### Saved Searches Can Be Deleted

Saved searches can now be deleted via API. See the [Saved Search](https://guides.dataverse.org/en/6.4/api/native-api.html#saved-search) section of the API Guide, #9317 and #10198.
Expand Down Expand Up @@ -84,7 +99,7 @@ JSON Schema validation has been enhanced with checks for required and allowed ch

Counter Processor 1.05 is now supported for use with Make Data Count. If you are running Counter Processor, you should reinstall/reconfigure it as described in the latest guides. Note that Counter Processor 1.05 requires Python 3, so you will need to follow the full Counter Processor install. Also note that if you configure the new version the same way, it will reprocess the days in the current month when it is first run. This is normal and will not affect the metrics in Dataverse. See also #10479.

## Bug Fixes
## Bugs Fixed

### Update Current Version

Expand Down Expand Up @@ -161,7 +176,7 @@ When any `ApiBlockingFilter` policy applies to a request, the JSON in the body o

See also #10508, #10672 and #10722.

## API
## API Updates

### Search API: affiliation, parentDataverseName, image_url, etc.

Expand Down Expand Up @@ -260,7 +275,7 @@ The addDataverse (`/api/dataverses/{identifier}`) API endpoint has been extended
The `/api/dataverses/{identifier}/metadatablocks` endpoint has been fixed to not return fields marked as displayOnCreate=true if there is an input level with include=false, when query parameters returnDatasetFieldTypes=true and onlyDisplayedOnCreate=true are set. See also #10741 and #10767.
## Settings
## Settings Added
The following settings have been added:
Expand All @@ -272,33 +287,176 @@ The following settings have been added:
- dataverse.pid.*.crossref.depositor
- dataverse.pid.*.crossref.depositor-email
### Backward incompatible changes
## Backward Incompatible Changes
The oai_dc export format has changed. See the "Remap oai_dc" section above.
## Complete List of Changes
For the complete list of code changes in this release, see the [6.4 milestone](https://github.com/IQSS/dataverse/issues?q=milestone%3A6.4+is%3Aclosed) in GitHub.
## Getting Help
For help with upgrading, installing, or general questions please post to the [Dataverse Community Google Group](https://groups.google.com/g/dataverse-community) or email [email protected].
## Installation
If this is a new installation, please follow our [Installation Guide](https://guides.dataverse.org/en/latest/installation/). Please don't be shy about [asking for help](https://guides.dataverse.org/en/latest/installation/intro.html#getting-help) if you need it!
Once you are in production, we would be delighted to update our [map of Dataverse installations](https://dataverse.org/installations) around the world to include yours! Please [create an issue](https://github.com/IQSS/dataverse-installations/issues) or email us at [email protected] to join the club!
You are also very welcome to join the [Global Dataverse Community Consortium](https://www.gdcc.io/) (GDCC).
## Upgrade Instructions
In order for changes to the `oai_dc` metadata export format to be reflected in existing datasets, a [reexport all](https://guides.dataverse.org/en/6.4/admin/metadataexport.html#batch-exports-through-the-api) should be run. See also "remap oai_dc" above.
Upgrading requires a maintenance window and downtime. Please plan accordingly, create backups of your database, etc.
Update your Solr schema.xml file to pick up the "datasetType" and "dvParentAlias" additions described above and do a full reindex.
These instructions assume that you've already upgraded through all the 5.x releases and are now running Dataverse 6.2.
### Additional Upgrade Steps
0\. These instructions assume that you are upgrading from the immediate previous version. If you are running an earlier version, the only supported way to upgrade is to progress through the upgrades to all the releases in between before attempting the upgrade to this version.
- If you currently use a PermaLink provider with a configured `base-url`: You must manually append
"/citation?persistentId=" to the existing base URL to maintain functionality.
- If you use a PermaLink provider without a configured `base-url`: No changes are required.
If you are running Payara as a non-root user (and you should be!), **remember not to execute the commands below as root**. Use `sudo` to change to that user first. For example, `sudo -i -u dataverse` if `dataverse` is your dedicated application user.
To restore any broken thumbnails caused by this bug, you can call the http://localhost:8080/api/admin/clearThumbnailFailureFlag API, which will attempt to clear the flag on all files (regardless of whether caused by this bug or some other problem with the file) or the http://localhost:8080/api/admin/clearThumbnailFailureFlag/id to clear the flag for individual files. Calling the former, batch API is recommended.
In the following commands, we assume that Payara 6 is installed in `/usr/local/payara6`. If not, adjust as needed.
```shell
export PAYARA=/usr/local/payara6`
```

(or `setenv PAYARA /usr/local/payara6` if you are using a `csh`-like shell)

6\. Update the Citation metadata block:
1\. Undeploy the previous version

```shell
$PAYARA/bin/asadmin undeploy dataverse-6.3
```
- `wget https://github.com/IQSS/dataverse/releases/download/v6.4/citation.tsv`
- `curl http://localhost:8080/api/admin/datasetfield/load -X POST --data-binary @citation.tsv -H "Content-type: text/tab-separated-values"`

2\. Stop and start Payara

```shell
service payara stop
sudo service payara start
```

3\. Deploy this version

```shell
$PAYARA/bin/asadmin deploy dataverse-6.4.war
```

Note: if you have any trouble deploying, stop Payara, remove the following directories, start Payara, and try to deploy again.

```shell
service payara stop
rm -rf $PAYARA/glassfish/domains/domain1/generated
rm -rf $PAYARA/glassfish/domains/domain1/osgi-cache
rm -rf $PAYARA/glassfish/domains/domain1/lib/databases
```

4\. For installations with internationalization:

Please remember to update translations via [Dataverse language packs](https://github.com/GlobalDataverseCommunityConsortium/dataverse-language-packs).

5\. Restart Payara

```shell
service payara stop
service payara start
```

6\. Update metadata blocks

These changes reflect incremental improvements made to the handling of core metadata fields.

```shell
wget https://raw.githubusercontent.com/IQSS/dataverse/v6.4/scripts/api/data/metadatablocks/citation.tsv
curl http://localhost:8080/api/admin/datasetfield/load -H "Content-type: text/tab-separated-values" -X POST --upload-file citation.tsv
```

7\. Update Solr schema.xml file. See specific instructions below for those installations without custom metadata blocks (7a) and those with custom metadata blocks (7b).

7a\. For installations without custom or experimental metadata blocks:

Stop Solr (usually `service solr stop`, depending on Solr installation/OS, see the [Installation Guide](https://guides.dataverse.org/en/6.4/installation/prerequisites.html#solr-init-script)).

```shell
service solr stop
```

Replace schema.xml

```shell
wget https://raw.githubusercontent.com/IQSS/dataverse/v6.4/conf/solr/schema.xml
cp schema.xml /usr/local/solr/solr-9.4.1/server/solr/collection1/conf
```

Start Solr.

```shell
service solr start
```

7b\. For installations with custom or experimental metadata blocks:

Stop Solr.

```shell
service solr stop
```

There are two ways to regenerate the schema, either by collecting the output of the Dataverse schema API and feeding it to the `update-fields.sh` script that we supply, as in the example below (modify the command lines as needed)...

```shell
wget https://raw.githubusercontent.com/IQSS/dataverse/master/conf/solr/9.4.1/update-fields.sh
chmod +x update-fields.sh
curl "http://localhost:8080/api/admin/index/solr/schema" | ./update-fields.sh /usr/local/solr/solr-9.4.1/server/solr/collection1/conf/schema.xml
```

...OR, alternatively, you edit your schema.xml by hand and add the following lines from the [6.4 tagged version](https://github.com/IQSS/dataverse/blob/v6.4/conf/solr/schema.xml) of schema.xml.

```
<field name="dvParentAlias" type="text_en" stored="true" indexed="true" multiValued="false"/>
<field name="datasetType" type="string" stored="true" indexed="true" multiValued="false"/>
<copyField source="datasetType" dest="_text_" maxChars="3000"/>
```

Start Solr.

```shell
service solr start
```

8\. Reindex Solr

Below is the simple way to reindex Solr. If you have a large installation of Dataverse, you might want to reindex in place, as described in [the guides](https://guides.dataverse.org/en/latest/admin/solr-search-index.html).

```shell
curl http://localhost:8080/api/admin/index
```

9\. Run reExportAll to update dataset metadata exports

This step is necessary because of changes described above for the `oai_dc` export format.

Below is the simple way to reexport all dataset metadata. For more advanced usage, please see [the guides](http://guides.dataverse.org/en/6.4/admin/metadataexport.html#batch-exports-through-the-api).

```shell
curl http://localhost:8080/api/admin/metadata/reExportAll
```

### Additional Upgrade Steps

10\. If there are broken thumbnails

To restore any broken thumbnails caused by the bug described above, you can call the `http://localhost:8080/api/admin/clearThumbnailFailureFlag` API, which will attempt to clear the flag on all files (regardless of whether caused by this bug or some other problem with the file) or the `http://localhost:8080/api/admin/clearThumbnailFailureFlag/$FILE_ID` to clear the flag for individual files. Calling the former, batch API is recommended.

```shell
http://localhost:8080/api/admin/clearThumbnailFailureFlag/$FILE_ID
```

11\. PermaLinks with custom base-url

If you currently use PermaLinks with a custom `base-url`: You must manually append `/citation?persistentId=` to the base URL to maintain functionality.

If you use a PermaLinks without a configured `base-url`, no changes are required.

0 comments on commit c706372

Please sign in to comment.