create 6.4 release notes and add about half the updates #10853

IQSS · Sep 18, 2024 · 8d92ebc · 8d92ebc
1 parent 1fffec4
commit 8d92ebc
Show file tree

Hide file tree

Showing 21 changed files with 169 additions and 126 deletions.
diff --git a/doc/release-notes/10169-JSON-schema-validation.md b/doc/release-notes/10169-JSON-schema-validation.md
diff --git a/doc/release-notes/10287-use-support-address-in-system-email-text.md b/doc/release-notes/10287-use-support-address-in-system-email-text.md
diff --git a/doc/release-notes/10341-croissant.md b/doc/release-notes/10341-croissant.md
diff --git a/doc/release-notes/10433-add-thumbnail-for-featured-dataverses.md b/doc/release-notes/10433-add-thumbnail-for-featured-dataverses.md
diff --git a/doc/release-notes/10508-base-image-fixes.md b/doc/release-notes/10508-base-image-fixes.md
diff --git a/doc/release-notes/10517-datasetType.md b/doc/release-notes/10517-datasetType.md
diff --git a/doc/release-notes/10583-dataset-unlink-functionality-same-permission-as-link.md b/doc/release-notes/10583-dataset-unlink-functionality-same-permission-as-link.md
diff --git a/doc/release-notes/10606-dataverse-in-windows-wsl.md b/doc/release-notes/10606-dataverse-in-windows-wsl.md
diff --git a/doc/release-notes/10633-add-dataverse-api-extension.md b/doc/release-notes/10633-add-dataverse-api-extension.md
diff --git a/doc/release-notes/6.4-release-notes.md b/doc/release-notes/6.4-release-notes.md
@@ -0,0 +1,169 @@
+# Dataverse 6.4
+
+Please note: To read these instructions in full, please go to https://github.com/IQSS/dataverse/releases/tag/v6.4 rather than the list of releases, which will cut them off.
+
+This release brings new features, enhancements, and bug fixes to Dataverse. Thank you to all of the community members who contributed code, suggestions, bug reports, and other assistance across the project.
+
+## Release Highlights
+
+## Features
+
+### Full List of ISO 639-3 Languages Now Supported
+
+The controlled vocabulary values list for the metadata field "Language" in the citation block has now been extended to include roughly 7920 ISO 639-3 values.
+
+Some of the language entries in the pre-6.4 list correspond to "macro languages" in ISO-639-3 and admins/users may wish to update to use the corresponding individual language entries from ISO-639-3. As these cases are expected to be rare (they do not involve major world languages), finding them is not covered in the release notes. Anyone who desires help in this area is encouraged to reach out to the Dataverse community via any of the standard communication channels.
+
+ISO 639-3 codes were downloaded from [sil.org](https://iso639-3.sil.org/code_tables/download_tables#Complete%20Code%20Tables:~:text=iso%2D639%2D3_Code_Tables_20240415.zip) and the file used for merging with the existing citation.tsv was "iso-639-3.tab". See also #8578 and #10762.
+
+### Unlink Dataset Button
+
+New "Unlink Dataset" button has been added to the dataset page to allow a user to unlink a dataset from a collection. The user must possess the same permissions needed to unlink the dataset as they would to link the Dataset. Additionally, the [existing API](https://guides.dataverse.org/en/6.4/admin/dataverses-datasets.html#unlink-a-dataset) for unlinking datasets has been updated to no longer require superuser access as the "Publish Dataset" permission is now enough. See also #10583 and #10689.
+
+### Pre-Publish File DOI Reservation
+
+Dataverse installations using DataCite as a persistent identifier (PID) provider (or other providers that support reserving PIDs) will be able to reserve PIDs for files when they are uploaded (rather than at publication time). Note that reserving file DOIs can slow uploads with large numbers of files so administrators may need to adjust timeouts (specifically any Apache "``ProxyPass / ajp://localhost:8009/ timeout=``" setting in the recommended Dataverse configuration).
+
+### Initial Support for Dataset Types
+
+Out of the box, all datasets now have the type "dataset" but superusers can add additional types. At this time the type of a dataset can only be set at creation time via API. The types "dataset", "software", and "workflow" (just those three, for now) will be sent to DataCite when the dataset is published.
+
+For details see [the guides](https://guides.dataverse.org/en/6.4/user/dataset-management.html#dataset-types), #10517 and #10694. Please note that this feature is highly experimental and is expected to [evolve](https://github.com/IQSS/dataverse-pm/issues/307).
+
+### Collection Thumbnail Logo for Featured Collections
+
+Collections can now have a thumbnail logo that is displayed when the collection is configured as a featured collection. If present, this thumbnail logo is shown. Otherwise, the collection logo is shown. Configuration is done under the "Theme" for a collection as explained in [the guides](https://guides.dataverse.org/en/6.4/user/dataverse-management.html#theme). See also #10291 and #10433.
+
+### Croissant
+
+A new metadata export format called Croissant is now available as an external metadata exporter. It is oriented toward making datasets consumable by machine learning.
+
+When enabled, Croissant replaces the Schema.org JSON-LD format in the `<head>` of dataset landing pages. For details, see the [Schema.org JSON-LD/Croissant Metadata](https://guides.dataverse.org/en/6.4/admin/discoverability.html#schema-org-head) under the discoverability section of the Admin Guide and #10382.
+
+For more about the Croissant exporter, including installation instructions, see <https://github.com/gdcc/exporter-croissant>. See also #10341, #10533, and [discussion](https://groups.google.com/g/dataverse-community/c/JI8HPgGarr8/m/DqEIkiwlAgAJ) on the mailing list.
+
+### Saved Searches Can Be Deleted
+
+Saved searches can now be deleted via API. See the [Saved Search](https://guides.dataverse.org/en/6.4/api/native-api.html#saved-search) section of the API Guide, #9317 and #10198.
+
+### Notification Email Improvement
+
+When notification emails are sent the part of the closing that says "contact us for support at" will now show the support email address (`dataverse.mail.support-email`), when configured, instead of the default system email address. Using the system email address here was particularly problematic when it was a "noreply" address. See also #10287 and #10504.
+
+### More Flexible PermaLinks
+
+The configuration setting `dataverse.pid.*.permalink.base-url`, which is used for PermaLinks, has been updated to support greater flexibility. Previously, the string `/citation?persistentId=` was automatically appended to the configured base URL. With this update, the base URL will now be used exactly as configured, without any automatic additions. See also #10775.
+
+### Development on Windows
+
+New instructions have been added for developers on Windows trying to run a Dataverse development environment using Windows Subsystem for Linux (WSL). See [the guides](https://guides.dataverse.org/en/6.4/developers/windows.html), #10606, and #10608.
+
+### Experimental Crossref PID (DOI) Provider
+
+Crossref can now be used as a PID (DOI) provider, but this feature is experimental. Please provide feedback through the usual channels. See also the [guides](https://guides.dataverse.org/en/6.4/installation/config.html#crossref-specific-settings), #8581, and #10806.
+
+### Improved JSON Schema Validation for Datasets
+
+JSON Schema validation has been enhanced with checks for required and allowed child objects as well as type checking for field types including `primitive`, `compound` and `controlledVocabulary`. More user-friendly error messages help pinpoint the issues in the dataset JSON. See [Retrieve a Dataset JSON Schema for a Collection](https://guides.dataverse.org/en/6.4/api/native-api.html#retrieve-a-dataset-json-schema-for-a-collection) in the API Guide, #10169, and #10543.
+
+### Counter Processor 1.05 Support (Make Data Count)
+
+Counter Processor 1.05 is now supported for use with Make Data Count. If you are running Counter Processor, you should reinstall/reconfigure it as described in the latest guides. Note that Counter Processor 1.05 requires Python 3, so you will need to follow the full Counter Processor install. Also note that if you configure the new version the same way, it will reprocess the days in the current month when it is first run. This is normal and will not affect the metrics in Dataverse. See also #10479.
+
+## Bug Fixes
+
+### No License Nor Terms of Use
+
+When datasets have neither a license nor custom terms of use, the dataset page will indicate this. Also, these datasets will no longer be indexed as having custom terms. See also #8796, #10513, and #10614.
+
+### CC0 License Bug Fix
+
+At a high level, some datasets have been mislabeled as "Custom License" when they should have been "CC0 1.0". This has been corrected.
+
+In Dataverse 5.10, datasets with only "CC0 Waiver" in the "termsofuse" field were converted to "Custom License" (instead of the CC0 1.0 license) through a SQL migration script (see #10634). On deployment of Dataverse 6.4, a new SQL migration script will be run automatically to correct this, changing these datasets to CC0. You can review the script in #10634, which only affect the following datasets:
+
+- The existing "Terms of Use" must be equal to "This dataset is made available under a Creative Commons CC0 license with the following additional/modified terms and conditions: CC0 Waiver" (this was set in #10634).
+- The following terms fields must be empty: Confidentiality Declaration, Special Permissions, Restrictions, Citation Requirements, Depositor Requirements, Conditions, and Disclaimer.
+- The license ID must not be assigned.
+
+The script will set the license ID to that of the CC0 1.0 license and remove the contents of "termsofuse" field. See also #9081 and #10634.
+
+### Remap oai_dc Export and Harvesting Format Fields: dc:type and dc:date
+
+The `oai_dc` export and harvesting format has had the following fields remapped:
+
+- dc:type was mapped to the field "Kind of Data". Now it is hard-coded to the word "Dataset".
+- dc:date was mapped to the field "Production Date" when available and otherwise to "Publication Date". Now it is mapped the field "Publication Date" or the field used for the citation date, if set (see [Set Citation Date Field Type for a Dataset](https://guides.dataverse.org/en/6.4/api/native-api.html#set-citation-date-field-type-for-a-dataset)).
+
+In order for these changes to be reflected in existing datasets, a [reexport all](https://guides.dataverse.org/en/6.4/admin/metadataexport.html#batch-exports-through-the-api) should be run (mentioned below). See #8129 and #10737.
+
+### Zip File No Longer Misdetected as Shapefile (Hidden Directories)
+
+When detecting files types, Dataverse would previously detect a zip file as a shapefile if it contained [markers of a shapefile](https://guides.dataverse.org/en/6.4/developers/geospatial.html) in hidden directories. These hidden directories are now ignored when deciding if a zip file is a shapefile or not. See also #8945 and #10627.
+
+### Valid JSON in Error Response
+
+When any `ApiBlockingFilter` policy applies to a request, the JSON in the body of the error response is now valid JSON. See also #10085.
+
+### Docker Container Base Image Security and Compatibility
+
+- Switch "wait-for" to "wait4x", aligned with the Configbaker Image
+- Update "jattach" to v2.2
+- Install AMD64 / ARM64 versions of tools as necessary
+- Run base image as unprivileged user by default instead of `root` - this was an oversight from OpenShift changes
+- Linux User, Payara Admin and Domain Master passwords:
+  - Print hints about default, public knowledge passwords in place for
+  - Enable replacing these passwords at container boot time
+- Enable building with updates Temurin JRE image based on Ubuntu 24.04 LTS
+- Fix entrypoint script troubles with pre- and postboot script files
+- Unify location of files at CONFIG_DIR=/opt/payara/config, avoid writing to other places
+
+See also #10508, #10672 and #10722.
+
+## API
+
+### addDataverse
+
+The addDataverse (`/api/dataverses/{identifier}`) API endpoint has been extended to allow adding metadata blocks, input levels and facet IDs at creation time, as the Dataverse page in create mode does in JSF. See also [the guides](https://guides.dataverse.org/en/6.4/api/native-api.html#create-a-dataverse-collection) #10633 and #10644.
+
+## Major Use Cases
+
+- Users will have DOIs/PIDs reserved for their files as part of file upload instead of at publication time. (Issue #7068, PR #7334)
+
+## Settings
+
+The following settings have been added:
+
+- dataverse.pid.*.crossref.url
+- dataverse.pid.*.crossref.rest-api-url
+- dataverse.pid.*.crossref.username
+- dataverse.pid.*.crossref.password
+- dataverse.pid.*.crossref.depositor
+- dataverse.pid.*.crossref.depositor-email
+
+### Backward incompatible changes
+
+The oai_dc export format has changed. See the "Remap oai_dc" section above.
+
+## Complete List of Changes
+
+## Getting Help
+
+## Upgrade Instructions
+
+In order for changes to the `oai_dc` metadata export format to be reflected in existing datasets, a [reexport all](https://guides.dataverse.org/en/6.4/admin/metadataexport.html#batch-exports-through-the-api) should be run. See also "remap oai_dc" above.
+
+Update your Solr schema.xml file to pick up the "dataset type" additions described above and do a full reindex.
+
+### Additional Upgrade Steps
+
+- If you currently use a PermaLink provider with a configured `base-url`: You must manually append
+   "/citation?persistentId=" to the existing base URL to maintain functionality.
+- If you use a PermaLink provider without a configured `base-url`: No changes are required.
+
+6\. Update the Citation metadata block:
+
+```
+- `wget https://github.com/IQSS/dataverse/releases/download/v6.4/citation.tsv`
+- `curl http://localhost:8080/api/admin/datasetfield/load -X POST --data-binary @citation.tsv -H "Content-type: text/tab-separated-values"`
+```
diff --git a/doc/release-notes/7068-reserve-file-pids.md b/doc/release-notes/7068-reserve-file-pids.md
diff --git a/doc/release-notes/8129-harvesting.md b/doc/release-notes/8129-harvesting.md