From bd890a5925a19df0f7e190e248cd3bd6ab612b99 Mon Sep 17 00:00:00 2001 From: Danny Brooke Date: Fri, 28 Jan 2022 13:56:37 -0500 Subject: [PATCH] potential process updates --- doc/release-notes/4.16-release-notes.md | 101 ----- doc/release-notes/4.17-release-notes.md | 110 ------ doc/release-notes/4.18-release-notes.md | 122 ------ doc/release-notes/4.18.1-release-notes.md | 45 --- doc/release-notes/4.19-release-notes.md | 125 ------- doc/release-notes/4.20-release-notes.md | 224 ----------- doc/release-notes/5.0-release-notes.md | 353 ------------------ doc/release-notes/5.1-release-notes.md | 99 ----- doc/release-notes/5.1.1-release-notes.md | 63 ---- doc/release-notes/5.2-release-notes.md | 105 ------ doc/release-notes/5.3-release-notes.md | 164 -------- doc/release-notes/5.4-release-notes.md | 344 ----------------- doc/release-notes/5.4.1-release-notes.md | 46 --- doc/release-notes/5.5-release-notes.md | 154 -------- doc/release-notes/5.6-release-notes.md | 138 ------- doc/release-notes/5.7-release-notes.md | 155 -------- doc/release-notes/5.8-release-notes.md | 112 ------ doc/release-notes/5.9-release-notes.md | 171 --------- doc/release-notes/README.md | 8 +- .../source/developers/making-releases.rst | 2 +- 20 files changed, 6 insertions(+), 2635 deletions(-) delete mode 100644 doc/release-notes/4.16-release-notes.md delete mode 100644 doc/release-notes/4.17-release-notes.md delete mode 100644 doc/release-notes/4.18-release-notes.md delete mode 100644 doc/release-notes/4.18.1-release-notes.md delete mode 100644 doc/release-notes/4.19-release-notes.md delete mode 100644 doc/release-notes/4.20-release-notes.md delete mode 100644 doc/release-notes/5.0-release-notes.md delete mode 100644 doc/release-notes/5.1-release-notes.md delete mode 100644 doc/release-notes/5.1.1-release-notes.md delete mode 100644 doc/release-notes/5.2-release-notes.md delete mode 100644 doc/release-notes/5.3-release-notes.md delete mode 100644 doc/release-notes/5.4-release-notes.md delete mode 100644 doc/release-notes/5.4.1-release-notes.md delete mode 100644 doc/release-notes/5.5-release-notes.md delete mode 100644 doc/release-notes/5.6-release-notes.md delete mode 100644 doc/release-notes/5.7-release-notes.md delete mode 100644 doc/release-notes/5.8-release-notes.md delete mode 100644 doc/release-notes/5.9-release-notes.md diff --git a/doc/release-notes/4.16-release-notes.md b/doc/release-notes/4.16-release-notes.md deleted file mode 100644 index 8feb263d2ab..00000000000 --- a/doc/release-notes/4.16-release-notes.md +++ /dev/null @@ -1,101 +0,0 @@ -# Dataverse 4.16 - -This release brings new features, enhancements, and bug fixes to Dataverse. Thank you to all of the community members who contributed code, suggestions, bug reports, and other assistance across the project. - -## Release Highlights - -### Metrics Redesign - -The metrics view at both the Dataset and File level has been redesigned. The main driver of this redesign has been the expanded metrics (citations and views) provided through an integration with Make Data Count, but installations that do not adopt Make Data Count will also be able to take advantage of the new metrics panel. - -### HTML Codebook Export - -Users will now be able to download HTML Codebooks as an additional Dataset Export type. This codebook is a more human-readable version of the DDI Codebook 2.5 metadata export and provides valuable information about the contents and structure of a dataset and will increase reusability of the datasets in Dataverse. - -### Harvesting Improvements - -The Harvesting code will now better handle problematic records during incremental harvests. Fixing this will mean not only fewer manual interventions by installation administrators to keep harvesting running, but it will also mean users can more easily find and access data that is important to their research. - -## Major Use Cases - -Newly-supported use cases in this release include: - -- As a user, I can view the works that have cited a dataset. -- As a user, I can view the downloads and views for a dataset, based on the Make Data Count standard. -- As a user, I can export an HTML codebook for a dataset. -- As a user, I can expect harvested datasets to be made available more regularly. -- As a user, I'll encounter fewer locks as I go through the publishing process. -- As an installation administrator, I no longer need to destroy a PID in another system after destroying a dataset in Dataverse. - -## Notes for Dataverse Installation Administrators - -### Run ReExportall - -We made changes to the citation block in this release that will require installations to run ReExportall as part of the upgrade process. We've included this in the detailed instructions below. - -### Custom Analytics Code Changes - -You should update your custom analytics code to include CDATA sections, inside the `script` tags, around the javascript code. We have updated the documentation and sample analytics code snippet provided in [Installation Guide > Configuration > Web Analytics Code](http://guides.dataverse.org/en/latest/installation/config.html#web-analytics-code) to fix a bug that broke the rendering of the 403 and 500 custom error pgs (#5967). - -### Destroy Updates - -Destroying Datasets in Dataverse will now unregister/delete the PID with the PID provider. This eliminates the need for an extra step to "clean up" a PID registration after destroying a Dataset. - -### Deleting Notifications - -In making the fix for #5687 we discovered that notifications created prior to 2018 may have been invalidated. With this release we advise that these older notifications are deleted from the database. The following query can be used for this purpose: - -`delete from usernotification where date_part('year', senddate) < 2018;` - -### Lock Improvements - -In 4.15 a new lock was added to prevent parallel edits. After seeing that the lock was not being released as expected, which required administrator intervention, we've adjusted this code to release the lock as expected. - -### New Database Settings - -:AllowCors - Allows Cross-Origin Resource sharing(CORS). By default this setting is absent and Dataverse assumes it to be true. - -## Notes for Tool Developers and Integrators - -### OpenAIRE Export Changes - -The OpenAIRE metadata export now correctly expresses information about a dataset's Production Place and GeoSpatial Bounding Box. When users add metadata to Dataverse's Production Place and GeoSpatial Bounding Box fields, those fields are now mapped to separate DataCite geoLocation properties. - -Metadata about the software name and version used to create a dataset, Software Name and Software Version, are re-mapped from DataCite's more general descriptionType="Methods" property to descriptionType="TechnicalInfo", which was added in a recent version of the DataCite schema in order to improve discoverability of metadata about the software used to create datasets. - -## Complete List of Changes - -For the complete list of code changes in this release, see the 4.16 milestone in Github. - -For help with upgrading, installing, or general questions please post to the Dataverse Google Group or email support@dataverse.org. - -## Installation - -If this is a new installation, please see our Installation Guide. - -## Upgrade - -1. Undeploy the previous version. - -- <glassfish install path>/glassfish4/bin/asadmin list-applications -- <glassfish install path>/glassfish4/bin/asadmin undeploy dataverse - -2. Stop glassfish and remove the generated directory, start - -- service glassfish stop -- remove the generated directory: rm -rf <glassfish install path>glassfish4/glassfish/domains/domain1/generated -- service glassfish start - -3. Deploy this version. - -- <glassfish install path>/glassfish4/bin/asadmin deploy <path>dataverse-4.16.war - -4. Restart glassfish - -5. Update Citation Metadata Block - - `curl http://localhost:8080/api/admin/datasetfield/load -X POST --data-binary @citation.tsv -H "Content-type: text/tab-separated-values"` - -6. Run ReExportall to update the citations - - diff --git a/doc/release-notes/4.17-release-notes.md b/doc/release-notes/4.17-release-notes.md deleted file mode 100644 index f723268d93f..00000000000 --- a/doc/release-notes/4.17-release-notes.md +++ /dev/null @@ -1,110 +0,0 @@ -# Dataverse 4.17 - -This release brings new features, enhancements, and bug fixes to Dataverse. Thank you to all of the community members who contributed code, suggestions, bug reports, and other assistance across the project. - -## Release Highlights - -### Dataset Level Explore Tools - -Tools that integrate with Dataverse can now be launched from the dataset page! This makes it possible to develop and add tools that work across the entire dataset instead of single files. Tools to verify reproducibility and allow researchers to compute on an entire dataset will take advantage of this new infrastructure. - -### Performance Enhancements - -Dataverse now allows installation administrators to configure the session timeout for logged in users using the new `:LoginSessionTimeout` setting. (Session length for anonymous users has been reduced from 24 hours to 10 minutes.) Setting this lower will release system resources as configured and will result in better performance (less memory use) throughout a Dataverse installation. - -Dataverse and Dataset pages have also been optimized to discard more of the objects they allocate immediately after the page load. Thus keeping less memory permanently tied up for the duration of the user's login session. These savings are especially significant in the Dataverse page. - -## Major Use Cases - -Newly-supported use cases in this release include: - -- As a user, I can launch and utilize external tools that allow me to work across the code, data, and other files in a dataset. -- As a user, I can add a footer to my dataverse to show the logo for a funder or other entity. -- As a developer, I can build external tools to verify reproducibility or allow computation. -- As a developer, I can check to see the impact of my proposed changes on memory utilization. -- As an installation administrator, I can make a quick configuration change to provide a better experience for my installation's users. - -## Notes for Dataverse Installation Administrators - -### Configurable User Session Timeout - -Idle session timeout for logged-in users has been made configurable in this release. -The default is now set to 8 hours (this is a change from the previous default value of 24 hours). -If you want to change it, set the setting :LoginSessionTimeout to the new value *in minutes*. -For example, to reduce the timeout to 4 hours: - - curl -X PUT -d 240 http://localhost:8080/api/admin/settings/:LoginSessionTimeout - -Once again, this is the session timeout for *logged-in* users only. For the anonymous sessions the sessions are set to time out after the default ``session-timeout`` value (also in minutes) in the web.xml of the Dataverse application, which is set to 10 minutes. You will most likely not ever need to change this, but if you do, configure it by editing the web.xml file. - -### Flexible Solr Schema, optionally reconfigure Solr - -With this release, we moved all fields in Solr search index that relate to the default metadata schemas from `schema.xml` to separate files. Custom metadata block configuration of the search index can be more easily automated that way. For details, see admin/metadatacustomization.html#updating-the-solr-schema. - -This is optional, but all future changes will go to these files. It might be a good idea to reconfigure Solr now or be aware to look for changes to these files in the future, too. Here's how: - -1. You will need to replace or modify your `schema.xml` with the recent one (containing XML includes) -2. Copy `schema_dv_mdb_fields.xml` and `schema_dv_mdb_copies.xml` to the same location as the `schema.xml` -3. A re-index is not necessary as long no other changes happened, as this is only a reorganization of Solr fields from a single schema.xml file into multiple files. - -In case you use custom metadata blocks, you might find the new `updateSchemaMDB.sh` script beneficial. Again, -see http://guides.dataverse.org/en/4.17/admin/metadatacustomization.html#updating-the-solr-schema - -### Memory Benchmark Test - -Developers and installation administrators can take advantage of new scripts to produce graphs of memory usage and garbage collection events. This is helpful for developers to investigate the implications of changes on memory usage and it is helpful for installation administrators to compare graphs across releases or time periods. For details see the `scripts/tests/ec2-memory-benchmark` directory. - -### New Database Settings - -[:LoginSessionTimeout](http://guides.dataverse.org/en/4.17/installation/config.html#loginsessiontimeout) controls the session timeout (in minutes) for logged-in users. - -## Notes for Tool Developers and Integrators - -### New Features and Breaking Changes for External Tool Developers - -The good news is that external tools can now be defined at the dataset level and there is new and improved documentation for external tool developers, linked below. - -Additionally, the reserved words `{datasetPid}`, `{{filePid}`, and `{localeCode}` were added. Please consider making it possible to translate your tool into various languages! The reserved word `{datasetVersion}` has been made more flexible. - -The bad news is that there are two breaking changes. First, tools must now define a "scope" of either "file" or "dataset" for the manifest to be successfully loaded into Dataverse. Existing tools in a Dataverse installations will be assigned a scope of "file" automatically by a SQL migration script but new installations of Dataverse will need to load an updated manifest file with this new "scope" variable. - -Second, file level tools that did not previously define a "contentType" are now required to do so. In previously releases, file level tools that did not define a contentType were automatically given a contentType of "text/tab-separated-values" but now Dataverse will refuse to load the manifest file if contentType is not specified. - -The Dataverse team has been reaching out to tool makers about these breaking changes and getting various tools working in the https://github.com/IQSS/dataverse-ansible repo. Thank you for your patience as the dust settles around the external tool framework. - -For more information, check out new Building External Tools section of the API Guide. - -## Complete List of Changes - -For the complete list of code changes in this release, see the 4.17 milestone in Github. - -For help with upgrading, installing, or general questions please post to the Dataverse Google Group or email support@dataverse.org. - -## Installation - -If this is a new installation, please see our Installation Guide. - -## Upgrade - -1. Undeploy the previous version. - -- <glassfish install path>/glassfish4/bin/asadmin list-applications -- <glassfish install path>/glassfish4/bin/asadmin undeploy dataverse - -2. Stop glassfish and remove the generated directory, start - -- service glassfish stop -- remove the generated directory: rm -rf <glassfish install path>glassfish4/glassfish/domains/domain1/generated -- service glassfish start - -3. Deploy this version. - -- <glassfish install path>/glassfish4/bin/asadmin deploy <path>dataverse-4.17.war - -4. Restart glassfish - -5. Update Citation Metadata Block - -- `wget https://github.com/IQSS/dataverse/releases/download/v4.17/citation.tsv` -- `curl http://localhost:8080/api/admin/datasetfield/load -X POST --data-binary @citation.tsv -H "Content-type: text/tab-separated-values"` - diff --git a/doc/release-notes/4.18-release-notes.md b/doc/release-notes/4.18-release-notes.md deleted file mode 100644 index 97ab25aa0fc..00000000000 --- a/doc/release-notes/4.18-release-notes.md +++ /dev/null @@ -1,122 +0,0 @@ -# Dataverse 4.18 - -**Note: There is an issue in 4.18 with the display of validation messages on the dataset page (#6380) and we recommend using 4.18.1 for any production environments.** - -This release brings new features, enhancements, and bug fixes to Dataverse. Thank you to all of the community members who contributed code, suggestions, bug reports, and other assistance across the project. - -## Release Highlights - -### File Page Previews and Previewers - -File-level External Tools can now be configured to display in a "Preview Mode" designed for embedding within the file landing page. - -While not technically part of this release, previewers have been made available for several common file types. The previewers support for spreadsheet, image, text, document, audio, video, html files and more. These previewers can be found in the Qualitative Data Repository Github Repository. The spreadsheet viewer was contributed by the [Dataverse SSHOC][] project. - -[Dataverse SSHOC]: https://www.sshopencloud.eu/news/developing-sshoc-dataverse - -### Microsoft Login - -Users can now create Dataverse accounts and login using self-provisioned Microsoft accounts such as live.com and outlook.com. Users can also use Microsoft accounts managed by their institutions. This new feature not only makes it easier to log in to Dataverse but will also streamline the interaction between any external tools that utilize Azure services that require login. - -### Add Data and Host Dataverse - -More workflows to add data have been added across the UI, including a new button on the My Data tab of the Account page, as well as a link in the Dataverse navbar, which will display on every page. This will provider users much easier access to start depositing data. By default, the Host Dataverse will be the installation root dataverse for these new Add Data workflows, but there is now a dropdown component allowing creators to select a dataverse you have proper permissions to create a new dataverse or dataset in. - -### Primefaces 7 - -Primefaces, the open source UI framework upon which the Dataverse front end is built, has been updated to the most recent version. This provides security updates and bug fixes and will also allow Dataverse developers to take advantage of new features and enhancements. - -### Integration Test Pipeline and Test Health Reporting - -As part of the Dataverse Community's ongoing efforts to provide more robust automated testing infrastructure, and in support of the project's desire to have the develop branch constantly in a "release ready" state, API-based integration tests are now run every time a branch is merged to develop. The status of the last test run is available as a badge at the bottom of the README.md file that serves as the homepage of Dataverse Github Repository. - -### Make Data Count Metrics Updates - -A new configuration option has been added that allows Make Data Count metrics to be collected, but not reflected in the front end. This option was designed to allow installations to collect and verify metrics for a period before turning on the display to users. - -### Search API Enhancements - -The Dataverse Search API will now display unpublished content when an API token is passed (and appropriate permissions exist). - -### Additional Dataset Author Identifiers - -The following dataset author identifiers are now supported: - -- DAI: https://en.wikipedia.org/wiki/Digital_Author_Identifier -- ResearcherID: http://researcherid.com -- ScopusID: https://www.scopus.com - -## Major Use Cases - -Newly-supported use cases in this release include: - -- Users can view previews of several common file types, eliminating the need to download or explore a file just to get a quick look. -- Users can log in using self-provisioned Microsoft accounts and also can log in using Microsoft accounts managed by an organization. -- Dataverse administrators can now revoke and regenerate API tokens with an API call. -- Users will receive notifications when their ingests complete, and will be informed if the ingest was a success or failure. -- Dataverse developers will receive feedback about the health of the develop branch after their pull request was merged. -- Dataverse tool developers will be able to query the Dataverse API for unpublished data as well as published data. -- Dataverse administrators will be able to collect Make Data Count metrics without turning on the display for users. -- Users with a DAI, ResearcherID, or ScopusID and use these author identifiers in their datasets. - -## Notes for Dataverse Installation Administrators - -### API Token Management - -- You can now delete a user's API token, recreate a user's API token, and find a token's expiration date. See the Native API guide for more information. - -### New JVM Options - -[:mdcbaseurlstring](http://guides.dataverse.org/en/4.18/installation/config.html#mdcbaseurlstring) allows dataverse administrators to use a test base URL for Make Data Count. - -### New Database Settings - -[:DisplayMDCMetrics](http://guides.dataverse.org/en/4.18/installation/config.html#DisplayMDCMetrics) can be set to false to disable display of MDC metrics. - -## Notes for Tool Developers and Integrators - -### Preview Mode - -Tool Developers can now add the `hasPreviewMode` parameter to their file level external tools. This setting provides an embedded, simplified view of the tool on the file pages for any installation that installs the tool. See Building External Tools for more information. - -### API Token Management - -If your tool writes content back to Dataverse, you can now take advantage of administrative endpoints that delete and re-create API tokens. You can also use an endpoint that provides the expiration date of a specific API token. See the Native API guide for more information. - -### View Unpublished Data Using Search API - -If you pass a token, the search API output will include unpublished content. - -## Complete List of Changes - -For the complete list of code changes in this release, see the 4.18 milestone in Github. - -For help with upgrading, installing, or general questions please post to the Dataverse Google Group or email support@dataverse.org. - -## Installation - -If this is a new installation, please see our Installation Guide. - -## Upgrade - -1. Undeploy the previous version. - -- <glassfish install path>/glassfish4/bin/asadmin list-applications -- <glassfish install path>/glassfish4/bin/asadmin undeploy dataverse - -2. Stop glassfish and remove the generated directory, start. - -- service glassfish stop -- remove the generated directory: rm -rf <glassfish install path>glassfish4/glassfish/domains/domain1/generated -- service glassfish start - -3. Deploy this version. - -- <glassfish install path>/glassfish4/bin/asadmin deploy <path>dataverse-4.18.war - -4. Restart glassfish. - -5. Update Citation Metadata Block - -- `wget https://github.com/IQSS/dataverse/releases/download/v4.18/citation.tsv` -- `curl http://localhost:8080/api/admin/datasetfield/load -X POST --data-binary @citation.tsv -H "Content-type: text/tab-separated-values"` diff --git a/doc/release-notes/4.18.1-release-notes.md b/doc/release-notes/4.18.1-release-notes.md deleted file mode 100644 index 99db66464a8..00000000000 --- a/doc/release-notes/4.18.1-release-notes.md +++ /dev/null @@ -1,45 +0,0 @@ -# Dataverse 4.18.1 - -This release provides a fix for a regression introduced in 4.18 and implements a few other small changes. - -## Release Highlights - -### Proper Validation Messages - -When creating or editing dataset metadata, users were not receiving field-level indications about what entries failed validation and were only receiving a message at the top of the page. This fix restores field-level indications. - -## Major Use Cases - -Use cases in this release include: - -- Users will receive the proper messaging when dataset metadata entries are not valid. -- Users can now view the expiration date of an API token and revoke a token on the API Token tab of the account page. - -## Complete List of Changes - -For the complete list of code changes in this release, see the 4.18.1 milestone in Github. - -For help with upgrading, installing, or general questions please post to the Dataverse Google Group or email support@dataverse.org. - -## Installation - -If this is a new installation, please see our Installation Guide. - -## Upgrade - -1. Undeploy the previous version. - -- <glassfish install path>/glassfish4/bin/asadmin list-applications -- <glassfish install path>/glassfish4/bin/asadmin undeploy dataverse - -2. Stop glassfish and remove the generated directory, start. - -- service glassfish stop -- remove the generated directory: rm -rf <glassfish install path>glassfish4/glassfish/domains/domain1/generated -- service glassfish start - -3. Deploy this version. - -- <glassfish install path>/glassfish4/bin/asadmin deploy <path>dataverse-4.18.1.war - -4. Restart glassfish. diff --git a/doc/release-notes/4.19-release-notes.md b/doc/release-notes/4.19-release-notes.md deleted file mode 100644 index 70c8711582c..00000000000 --- a/doc/release-notes/4.19-release-notes.md +++ /dev/null @@ -1,125 +0,0 @@ -# Dataverse 4.19 - -This release brings new features, enhancements, and bug fixes to Dataverse. Thank you to all of the community members who contributed code, suggestions, bug reports, and other assistance across the project. - -## Release Highlights - -### Open ID Connect Support - -Dataverse now provides basic support for any OpenID Connect (OIDC) compliant authentication provider. - -Prior to supporting this standard, new authentication methods needed to be added by pull request. OIDC support provides a standardized way for authentication, sharing user information, and more. You are able to use any compliant provider just by loading a configuration file, without touching the codebase. While the usual prominent providers like Google and others feature OIDC support there are plenty of other options to easily attach your installation to a custom authentication provider, using enterprise grade software. - -See the [OpenID Connect Login Options documentation](http://guides.dataverse.org/en/4.19/installation/oidc.html) in the Installation Guide for more details. - -This is to be extended with support for attribute mapping, group syncing and more in future versions of the code. - -### Python Installer - -We are introducing a new installer script, written in Python. It is intended to eventually replace the old installer (written in Perl). For now it is being offered as an (experimental) alternative. - -See [README_python.txt](https://github.com/IQSS/dataverse/blob/v4.19/scripts/installer/README_python.txt) in scripts/installer and/or in the installer bundle for more information. - -## Major Use Cases - -Newly-supported use cases in this release include: - -- Dataverse installation administrators will be able to experiment with a Python Installer (Issue #3937, PR #6484) -- Dataverse installation administrators will be able to set up an OIDC-compliant login options by editing a configuration file and with no need for a code change (Issue #6432, PR #6433) -- Following setup by a Dataverse administration, users will be able to log in using OIDC-compliant methods (Issue #6432, PR #6433) -- Users of the Search API will see additional fields in the JSON output (Issues #6300, #6396, PR #6441) -- Users loading the support form will now be presented with the math challenge as expected and will be able to successfully send an email to support (Issue #6307, PR #6462) -- Users of https://mybinder.org can now spin up Jupyter Notebooks and other computational environments from Dataverse DOIs (Issue #4714, PR #6453) - -## Notes for Dataverse Installation Administrators - -### Security vulnerability in Solr - -A serious security issue has recently been identified in multiple versions of Solr search engine, including v.7.3 that Dataverse is currently using. Follow the instructions below to verify that your installation is safe from a potential attack. You can also consult the following link for a detailed description of the issue: - -RCE in Solr via Velocity Template. - -The vulnerability allows an intruder to execute arbitrary code on the system running Solr. Fortunately, it can only be exploited if Solr API access point is open to direct access from public networks (aka, "the outside world"), which is NOT needed in a Dataverse installation. - -We have always recommended having Solr (port 8983) firewalled off from public access in our installation guides. But we recommend that you double-check your firewall settings and verify that the port is not accessible from outside networks. The simplest quick test is to try the following URL in your browser: - - `http://:8983` - -and confirm that you get "access denied" or that it times out, etc. - -In most cases, when Solr runs on the same server as the Dataverse web application, you will only want the port accessible from localhost. We also recommend that you add the following arguments to the Solr startup command: `-j jetty.host=127.0.0.1`. This will make Solr accept connections from localhost only; adding redundancy, in case of the firewall failure. - -In a case where Solr needs to run on a different host, make sure that the firewall limits access to the port only to the Dataverse web host(s), by specific ip address(es). - -We would also like to reiterate that it is simply never a good idea to run Solr as root! Running the process as a non-privileged user would substantially minimize any potential damage even in the event that the instance is compromised. - -### Citation and Geospatial Metadata Block Updates - -We updated two metadata blocks in this release. Updating these metadata blocks is mentioned in the step-by-step upgrade instructions below. - -### Run ReExportall - -We made changes to the JSON Export in this release (#6246). If you'd like these changes to reflected in your JSON exports, you should run ReExportall as part of the upgrade process. We've included this in the step-by-step instructions below. - -### BinderHub - -https://mybinder.org now supports spinning up Jupyter Notebooks and other computational environments from Dataverse DOIs. - -### Widgets update for OpenScholar - -We updated the code for widgets so that they will keep working in OpenScholar sites after the upcoming upgrade OpenScholar upgrade to Drupal 8. If users of your dataverse have embedded widgets on an Openscholar site that upgrades to Drupal 8, you will need to run this Dataverse version (or later) for the widgets to keep working. - -### Payara tech preview - -Dataverse 4 has always run on Glassfish 4.1 but changes in this release (PR #6523) should open the door to upgrading to Payara 5 eventually. Production installations of Dataverse should remain on Glassfish 4.1 but feedback from any experiments running Dataverse on Payara 5 is welcome via the [usual channels](https://dataverse.org/contact). - -## Notes for Tool Developers and Integrators - -### Search API - -The boolean parameter `query_entities` has been removed from the Search API. The former "true" behavior of "whether entities are queried via direct database calls (for developer use)" is now always true. - -Additional fields are now available via the Search API, mostly related to information about specific dataset versions. - -## Complete List of Changes - -For the complete list of code changes in this release, see the 4.19 milestone in Github. - -For help with upgrading, installing, or general questions please post to the Dataverse Google Group or email support@dataverse.org. - -## Installation - -If this is a new installation, please see our Installation Guide. - -## Upgrade - -1. Undeploy the previous version. - -- <glassfish install path>/glassfish4/bin/asadmin list-applications -- <glassfish install path>/glassfish4/bin/asadmin undeploy dataverse - -2. Stop glassfish and remove the generated directory, start. - -- service glassfish stop -- remove the generated directory: rm -rf <glassfish install path>glassfish4/glassfish/domains/domain1/generated -- service glassfish start - -3. Deploy this version. - -- <glassfish install path>/glassfish4/bin/asadmin deploy <path>dataverse-4.19.war - -4. Restart glassfish. - -5. Update Citation Metadata Block - -- `wget https://github.com/IQSS/dataverse/releases/download/v4.19/citation.tsv` -- `curl http://localhost:8080/api/admin/datasetfield/load -X POST --data-binary @citation.tsv -H "Content-type: text/tab-separated-values"` - -6. Update Geospatial Metadata Block - -- `wget https://github.com/IQSS/dataverse/releases/download/v4.19/geospatial.tsv` -- `curl http://localhost:8080/api/admin/datasetfield/load -X POST --data-binary @geospatial.tsv -H "Content-type: text/tab-separated-values"` - -7. (Optional) Run ReExportall to update JSON Exports - - diff --git a/doc/release-notes/4.20-release-notes.md b/doc/release-notes/4.20-release-notes.md deleted file mode 100644 index e29953db101..00000000000 --- a/doc/release-notes/4.20-release-notes.md +++ /dev/null @@ -1,224 +0,0 @@ -# Dataverse 4.20 - -This release brings new features, enhancements, and bug fixes to Dataverse. Thank you to all of the community members who contributed code, suggestions, bug reports, and other assistance across the project. - -## Release Highlights - -### Multiple Store Support - -Dataverse can now be configured to store files in more than one place at the same time (multiple file, s3, and/or swift stores). - -General information about this capability can be found below and in the Configuration Guide - File Storage section. - -### S3 Direct Upload support - -S3 stores can now optionally be configured to support direct upload of files, as one option for supporting upload of larger files. In the current implementation, each file is uploaded in a single HTTP call. For AWS, this limits file size to 5 GB. With Minio the theoretical limit should be 5 TB and 50+ GB file uploads have been tested successfully. (In practice other factors such as network timeouts may prevent a successful upload a multi-TB file and minio instances may be configured with a < 5 TB single HTTP call limit.) No other S3 service providers have been tested yet. Their limits should be the lower of the maximum object size allowed and any single HTTP call upload limit. - -General information about this capability can be found in the Big Data Support Guide with specific information about how to enable it in the Configuration Guide - File Storage section. - -To support large data uploads, installations can now configure direct upload to S3, bypassing the application server. This will allow for larger uploads over a more resilient transfer method. - -General information about this capability can be found below and in the Configuration Guide. - -### Integration Test Coverage Reporting - -The percentage of code covered by the API-based integration tests is now shown on a badge at the bottom of the README.md file that serves as the homepage of Dataverse Github Repository. - -### New APIs - -New APIs for Role Management and Dataset Size have been added. Previously, managing roles at the dataset and file level was only possible through the UI. API users can now also retrieve the size of a dataset through an API call, with specific parameters depending on the type of information needed. - -More information can be found in the API Guide. - -## Major Use Cases - -Newly-supported use cases in this release include: - -- Users will now be able to see the number of linked datasets and dataverses accurately reflected in the facet counts on the Dataverse search page. (Issue #6564, PR #6262) -- Users will be able to upload large files directly to S3. (Issue #6489, PR #6490) -- Users will be able to see the PIDs of datasets and files in the Guestbook export. (Issue #6534, PR #6628) -- Administrators will be able to configure multiple stores per Dataverse installation, which allow dataverse-level setting of storage location, upload size limits, and supported data transfer methods (Issue #6485, PR #6488) -- Administrators and integrators will be able to manage roles using a new API. (Issue #6290, PR #6622) -- Administrators and integrators will be able to determine a dataset's size. (Issue #6524, PR #6609) -- Integrators will now be able to retrieve the number of files in a dataset as part of a single API call instead of needing to count the number of files in the response. (Issue #6601, PR #6623) - -## Notes for Dataverse Installation Administrators - -### Potential Data Integrity Issue - -We recently discovered a *potential* data integrity issue in Dataverse databases. One manifests itself as duplicate DataFile objects created for the same uploaded file (https://github.com/IQSS/dataverse/issues/6522); the other as duplicate DataTable (tabular metadata) objects linked to the same DataFile (https://github.com/IQSS/dataverse/issues/6510). This issue impacted approximately .03% of datasets in Harvard's Dataverse. - -To see if any datasets in your installation have been impacted by this data integrity issue, we've provided a diagnostic script here: - -https://github.com/IQSS/dataverse/raw/develop/scripts/issues/6510/check_datafiles_6522_6510.sh - -The script relies on the PostgreSQL utility psql to access the database. You will need to edit the credentials at the top of the script to match your database configuration. - -If neither of the two issues is present in your database, you will see a message "... no duplicate DataFile objects in your database" and "no tabular files affected by this issue in your database". - -If either, or both kinds of duplicates are detected, the script will provide further instructions. We will need you to send us the produced output. We will then assist you in resolving the issues in your database. - -### Multiple Store Support Changes - -**Existing installations will need to make configuration changes to adopt this version, regardless of whether additional stores are to be added or not.** - -Multistore support requires that each store be assigned a label, id, and type - see the Configuration Guide for a more complete explanation. For an existing store, the recommended upgrade path is to assign the store id based on it's type, i.e. a 'file' store would get id 'file', an 's3' store would have the id 's3'. - -With this choice, no manual changes to datafile 'storageidentifier' entries are needed in the database. If you do not name your existing store using this convention, you will need to edit the database to maintain access to existing files. - -The following set of commands to change the Glassfish JVM options will adapt an existing file or s3 store for this upgrade: -For a file store: - - ./asadmin create-jvm-options "\-Ddataverse.files.file.type=file" - ./asadmin create-jvm-options "\-Ddataverse.files.file.label=file" - ./asadmin create-jvm-options "\-Ddataverse.files.file.directory=" - -For a s3 store: - - ./asadmin create-jvm-options "\-Ddataverse.files.s3.type=s3" - ./asadmin create-jvm-options "\-Ddataverse.files.s3.label=s3" - ./asadmin delete-jvm-options "-Ddataverse.files.s3-bucket-name=" - ./asadmin create-jvm-options "-Ddataverse.files.s3.bucket-name=" - -Any additional S3 options you have set will need to be replaced as well, following the pattern in the last two lines above - delete the option including a '-' after 's3' and creating the same option with the '-' replaced by a '.', using the same value you currently have configured. - -Once these options are set, restarting the Glassfish service is all that is needed to complete the change. - -Note that the "\-Ddataverse.files.directory", if defined, continues to control where temporary files are stored (in the /temp subdir of that directory), independent of the location of any 'file' store defined above. - -Also note that the :MaxFileUploadSizeInBytes property has a new option to provide independent limits for each store instead of a single value for the whole installation. The default is to apply any existing limit defined by this property to all stores. - -### Direct S3 Upload Changes - -Direct upload to S3 is enabled per store by one new jvm option: - - ./asadmin create-jvm-options "\-Ddataverse.files..upload-redirect=true" - -The existing :MaxFileUploadSizeInBytes property and ```dataverse.files..url-expiration-minutes``` jvm option for the same store also apply to direct upload. - -Direct upload via the Dataverse web interface is transparent to the user and handled automatically by the browser. Some minor differences in file upload exist: directly uploaded files are not unzipped and Dataverse does not scan their content to help in assigning a MIME type. Ingest of tabular files and metadata extraction from FITS files will occur, but can be turned off for files above a specified size limit through the new dataverse.files..ingestsizelimit jvm option. - -API calls to support direct upload also exist, and, if direct upload is enabled for a store in Dataverse, the latest DVUploader (v1.0.8) provides a'-directupload' flag that enables its use. - -### Solr Update - -With this release we upgrade to the latest available stable release in the Solr 7.x branch. We recommend a fresh installation of Solr 7.7.2 (the index will be empty) -followed by an "index all". - -Before you start the "index all", Dataverse will appear to be empty because -the search results come from Solr. As indexing progresses, results will appear -until indexing is complete. - -### Dataverse Linking Fix - -The fix implemented for #6262 will display the datasets contained in linked dataverses in the linking dataverse. The full reindex described above will correct these counts. Going forward, this will happen automatically whenever a dataverse is linked. - -### Google Analytics Download Tracking Bug - -The button tracking capability discussed in the installation guide (http://guides.dataverse.org/en/4.20/installation/config.html#id88) relies on an analytics-code.html file that must be configured using the :WebAnalyticsCode setting. The example file provided in the installation guide is no longer compatible with recent Dataverse releases (>v4.16). Installations using this feature should update their analytics-code.html file by following the installation instructions using the updated example file. Alternately, sites can modify their existing files to include the one-line change made in the example file at line 120. - -### Run ReExportall - -We made changes to the JSON Export in this release (Issue 6650, PR #6669). If you'd like these changes to reflected in your JSON exports, you should run ReExportall as part of the upgrade process. We've included this in the step-by-step instructions below. - -### New JVM Options and Database Settings - -## New JVM Options for file storage drivers - -- The JVM option dataverse.files.file.directory= controls where temporary files are stored (in the /temp subdir of the defined directory), independent of the location of any 'file' store defined above. -- The JVM option dataverse.files..upload-redirect enables direct upload of files added to a dataset to the S3 bucket. (S3 stores only!) -- The JVM option dataverse.files..MaxFileUploadSizeInBytes controls the maximum size of file uploads allowed for the given file store. -- The JVM option dataverse.files..ingestsizelimit controls the maximum size of files for which ingest will be attempted, for the given file store. - -## New Database Settings for Shibboleth - -- The database setting :ShibAffiliationAttribute can now be set to prevent affiliations for Shibboleth users from being reset upon each log in. - -## Notes for Tool Developers and Integrators - -### Integration Test Coverage Reporting - -API-based integration tests are run every time a branch is merged to develop and the percentage of code covered by these integration tests is now shown on a badge at the bottom of the README.md file that serves as the homepage of Dataverse Github Repository. - -### Guestbook Column Changes - -Users of downloaded guestbooks should note that two new columns have been added: - -- Dataset PID -- File PID - -If you are expecting column in the CSV file to be in a particular order, you will need to make adjustments. - -Old columns: Guestbook, Dataset, Date, Type, File Name, File Id, User Name, Email, Institution, Position, Custom Questions - -New columns: Guestbook, Dataset, Dataset PID, Date, Type, File Name, File Id, File PID, User Name, Email, Institution, Position, Custom Questions - -### API Changes - -As reported in #6570, the affiliation for dataset contacts has been wrapped in parentheses in the JSON output from the Search API. These parentheses have now been removed. This is a backward incompatible change but it's expected that this will not cause issues for integrators. - -### Role Name Change - -The role alias provided in API responses has changed, so if anything was hard-coded to "editor" instead of "contributor" it will need to be updated. - -## Complete List of Changes - -For the complete list of code changes in this release, see the 4.20 milestone in Github. - -For help with upgrading, installing, or general questions please post to the Dataverse Google Group or email support@dataverse.org. - -## Installation - -If this is a new installation, please see our Installation Guide. - -## Upgrade - -1. Undeploy the previous version. - -- <glassfish install path>/glassfish4/bin/asadmin list-applications -- <glassfish install path>/glassfish4/bin/asadmin undeploy dataverse - -2. Stop glassfish and remove the generated directory, start. - -- service glassfish stop -- remove the generated directory: rm -rf <glassfish install path>glassfish4/glassfish/domains/domain1/generated -- service glassfish start - -3. Install and configure Solr v7.7.2 - -See http://guides.dataverse.org/en/4.20/installation/prerequisites.html#installing-solr - -4. Deploy this version. - -- <glassfish install path>/glassfish4/bin/asadmin deploy <path>dataverse-4.20.war - -5. The following set of commands to change the Glassfish JVM options will adapt an existing file or s3 store for this upgrade: -For a file store: - - ./asadmin create-jvm-options "\-Ddataverse.files.file.type=file" - ./asadmin create-jvm-options "\-Ddataverse.files.file.label=file" - ./asadmin create-jvm-options "\-Ddataverse.files.file.directory=" - -For a s3 store: - - ./asadmin create-jvm-options "\-Ddataverse.files.s3.type=s3" - ./asadmin create-jvm-options "\-Ddataverse.files.s3.label=s3" - ./asadmin delete-jvm-options "-Ddataverse.files.s3-bucket-name=" - ./asadmin create-jvm-options "-Ddataverse.files.s3.bucket-name=" - -Any additional S3 options you have set will need to be replaced as well, following the pattern in the last two lines above - delete the option including a '-' after 's3' and creating the same option with the '-' replaced by a '.', using the same value you currently have configured. - -6. Restart glassfish. - -7. Update Citation Metadata Block - -- `wget https://github.com/IQSS/dataverse/releases/download/4.20/citation.tsv` -- `curl http://localhost:8080/api/admin/datasetfield/load -X POST --data-binary @citation.tsv -H "Content-type: text/tab-separated-values"` - -8. Kick off full reindex - -http://guides.dataverse.org/en/4.20/admin/solr-search-index.html - -9. (Recommended) Run ReExportall to update JSON Exports - - diff --git a/doc/release-notes/5.0-release-notes.md b/doc/release-notes/5.0-release-notes.md deleted file mode 100644 index f87d428f30a..00000000000 --- a/doc/release-notes/5.0-release-notes.md +++ /dev/null @@ -1,353 +0,0 @@ -# Dataverse 5.0 - -This release brings new features, enhancements, and bug fixes to Dataverse. Thank you to all of the community members who contributed code, suggestions, bug reports, and other assistance across the project. - -Please note that this is a major release and these are long release notes. We offer no apologies. :) - -## Release Highlights - -### Continued Dataset and File Redesign: Dataset and File Button Redesign, Responsive Layout - -The buttons available on the Dataset and File pages have been redesigned. This change is to provide more scalability for future expanded options for data access and exploration, and to provide a consistent experience between the two pages. The dataset and file pages have also been redesigned to be more responsive and function better across multiple devices. - -This is an important step in the incremental process of the Dataset and File Redesign project, following the release of on-page previews, filtering and sorting options, tree view, and other enhancements. Additional features in support of these redesign efforts will follow in later 5.x releases. - -### Payara 5 - -A major upgrade of the application server provides security updates, access to new features like MicroProfile Config API, and will enable upgrades to other core technologies. - -Note that moving from Glassfish to Payara will be required as part of the move to Dataverse 5. - -### Download Dataset - -Users can now more easily download all files in Dataset through both the UI and API. If this causes server instability, it's suggested that Dataverse Installation Administrators take advantage of the new Standalone Zipper Service described below. - -#### Download All Option on the Dataset Page - -In previous versions of Dataverse, downloading all files from a dataset meant several clicks to select files and initiate the download. The Dataset Page now includes a Download All option for both the original and archival formats of the files in a dataset under the "Access Dataset" button. - -#### Download All Files in a Dataset by API - -In previous versions of Dataverse, downloading all files from a dataset via API was a two step process: - -- Find all the database ids of the files. -- Download all the files, using those ids (comma-separated). - -Now you can download all files from a dataset (assuming you have access to them) via API by passing the dataset persistent ID (PID such as DOI or Handle) or the dataset's database id. Versions are also supported, and you can pass :draft, :latest, :latest-published, or numbers (1.1, 2.0) similar to the "download metadata" API. - -### A Multi-File, Zipped Download Optimization - -In this release we are offering an experimental optimization for the multi-file, download-as-zip functionality. If this option is enabled, instead of enforcing size limits, we attempt to serve all the files that the user requested (that they are authorized to download), but the request is redirected to a standalone zipper service running as a cgi executable. Thus moving these potentially long-running jobs completely outside the Application Server (Payara); and preventing service threads from becoming locked serving them. Since zipping is also a CPU-intensive task, it is possible to have this service running on a different host system, thus freeing the cycles on the main Application Server. The system running the service needs to have access to the database as well as to the storage filesystem, and/or S3 bucket. - -Please consult the scripts/zipdownload/README.md in the Dataverse 5 source tree. - -The components of the standalone "zipper tool" can also be downloaded -here: - -https://github.com/IQSS/dataverse/releases/download/v5.0/zipper.zip - -### Updated File Handling - -Files without extensions can now be uploaded through the UI. This release also changes the way Dataverse handles duplicate (filename or checksum) files in a dataset. Specifically: - -- Files with the same checksum can be included in a dataset, even if the files are in the same directory. -- Files with the same filename can be included in a dataset as long as the files are in different directories. -- If a user uploads a file to a directory where a file already exists with that directory/filename combination, Dataverse will adjust the file path and names by adding "-1" or "-2" as applicable. This change will be visible in the list of files being uploaded. -- If the directory or name of an existing or newly uploaded file is edited in such a way that would create a directory/filename combination that already exists, Dataverse will display an error. -- If a user attempts to replace a file with another file that has the same checksum, an error message will be displayed and the file will not be able to be replaced. -- If a user attempts to replace a file with a file that has the same checksum as a different file in the dataset, a warning will be displayed. -- Files without extensions can now be uploaded through the UI. - -### Pre-Publish DOI Reservation with DataCite - -Dataverse installations using DataCite will be able to reserve the persistent identifiers for datasets with DataCite ahead of publishing time. This allows the DOI to be reserved earlier in the data sharing process and makes the step of publishing datasets simpler and less error-prone. - -### Primefaces 8 - -Primefaces, the open source UI framework upon which the Dataverse front end is built, has been updated to the most recent version. This provides security updates and bug fixes and will also allow Dataverse developers to take advantage of new features and enhancements. - -## Major Use Cases - -Newly-supported use cases in this release include: - -- Users will be presented with a new workflow around dataset and file access and exploration. (Issue #6684, PR #6909) -- Users will experience a UI appropriate across a variety of device sizes. (Issue #6684, PR #6909) -- Users will be able to download an entire dataset without needing to select all the files in that dataset. (Issue #6564, PR #6262) -- Users will be able to download all files in a dataset with a single API call. (Issue #4529, PR #7086) -- Users will have DOIs reserved for their datasets upon dataset create instead of at publish time. (Issue #5093, PR #6901) -- Users will be able to upload files without extensions. (Issue #6634, PR #6804) -- Users will be able to upload files with the same name in a dataset, as long as a those files are in different file paths. (Issue #4813, PR #6924) -- Users will be able to upload files with the same checksum in a dataset. (Issue #4813, PR #6924) -- Users will be less likely to encounter locks during the publishing process due to PID providers being unavailable. (Issue #6918, PR #7118) -- Users will now have their files validated during publish, and in the unlikely event that anything has happened to the files between deposit and publish, they will be able to take corrective action. (Issue #6558, PR #6790) -- Administrators will likely see more success with Harvesting, as many minor harvesting issues have been resolved. (Issues #7127, #7128, #4597, #7056, #7052, #7023, #7009, and #7003) -- Administrators can now enable an external zip service that frees up application server resources and allows the zip download limit to be increased. (Issue #6505, PR #6986) -- Administrators can now create groups based on users' email domains. (Issue #6936, PR #6974) -- Administrators can now set date facets to be organized chronologically. (Issue #4977, PR #6958) -- Administrators can now link harvested datasets using an API. (Issue #5886, PR #6935) -- Administrators can now destroy datasets with mapped shapefiles. (Issue #4093, PR #6860) - -## Notes for Dataverse Installation Administrators - -### Glassfish to Payara - -This upgrade requires a few extra steps. See the detailed upgrade instructions below. - -### Dataverse Installations Using DataCite: Upgrade Action Required - -If you are using DataCite as your DOI provider you must add a new JVM option called "doi.dataciterestapiurlstring" with a value of "https://api.datacite.org" for production environments and "https://api.test.datacite.org" for test environments. More information about this JVM option can be found in the [Installation Guide](http://guides.dataverse.org/en/5.0/installation/). - -"doi.mdcbaseurlstring" should be deleted if it was previously set. - -### Dataverse Installations Using DataCite: Upgrade Action Recommended - -For installations that are using DataCite, Dataverse v5.0 introduces a change in the process of registering the Persistent Identifier (DOI) for a dataset. Instead of registering it when the dataset is published for the first time, Dataverse will try to "reserve" the DOI when it's created (by registering it as a "draft", using DataCite terminology). When the user publishes the dataset, the DOI will be publicized as well (by switching the registration status to "findable"). This approach makes the process of publishing datasets simpler and less error-prone. - -New APIs have been provided for finding any unreserved DataCite-issued DOIs in your Dataverse, and for reserving them (see below). While not required - the user can still attempt to publish a dataset with an unreserved DOI - having all the identifiers reserved ahead of time is recommended. If you are upgrading an installation that uses DataCite, we specifically recommend that you reserve the DOIs for all your pre-existing unpublished drafts as soon as Dataverse v5.0 is deployed, since none of them were registered at create time. This can be done using the following API calls: - -- `/api/pids/unreserved` will report the ids of the datasets -- `/api/pids/:persistentId/reserve` reserves the assigned DOI with DataCite (will need to be run on every id reported by the the first API). - -See the [Native API Guide](http://guides.dataverse.org/en/5.0/api/native-api.html) for more information. - -Scripted, the whole process would look as follows (adjust as needed): - -``` - API_TOKEN='xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx' - - curl -s -H "X-Dataverse-key:$API_TOKEN" http://localhost:8080/api/pids/unreserved | - # the API outputs JSON; note the use of jq to parse it: - jq '.data.count[].pid' | tr -d '"' | - while read doi - do - curl -s -H "X-Dataverse-key:$API_TOKEN" -X POST http://localhost:8080/api/pids/:persistentId/reserve?persistentId=$doi - done -``` - -Going forward, once all the DOIs have been reserved for the legacy drafts, you may still get an occasional dataset with an unreserved identifier. DataCite service instability would be a potential cause. There is no reason to expect that to happen often, but it is not impossible. You may consider running the script above (perhaps with some extra diagnostics added) regularly, from a cron job or otherwise, to address this preemptively. - -### Terms of Use Display Updates - -In this release we’ve fixed an issue that would cause the Application Terms of Use to not display when the user's language is set to a language that does not match one of the languages for which terms were created and registered for that Dataverse installation. Instead of the expected Terms of Use, users signing up could receive the “There are no Terms of Use for this Dataverse installation” message. This could potentially result in some users signing up for an account without having the proper Terms of Use displayed. This will only affect installations that use the :ApplicationTermsOfUse setting. - -Please note that there is not currently a native workflow in Dataverse to display updated Terms of Use to a user or to force re-agreement. This would only potentially affect users that have signed up since the upgrade to 4.17 (or a following release if 4.17 was skipped). - -### Datafiles Validation when Publishing Datasets - -When a user requests to publish a dataset, Dataverse will now attempt to validate the physical files in the dataset, by recalculating the checksums and verifying them against the values in the database. The goal is to prevent any corrupted files in published datasets. Most of all the instances of actual damage to physical files that we've seen in the past happened while the datafiles were still in the Draft state. (Physical files become essentially read-only once published). So this is the logical place to catch any such issues. - -If any files in the dataset fail the validation, the dataset does not get published, and the user is notified that they need to contact their Dataverse support in order to address the issue before another attempt to publish can be made. See the "Troubleshooting" section of the Guide on how to fix such problems. - -This validation will be performed asynchronously, the same way as the registration of the file-level persistent ids. Similarly to the file PID registration, this validation process can be disabled on your system, with the setting `:FileValidationOnPublishEnabled`. (A Dataverse admin may choose to disable it if, for example, they are already running an external auditing system to monitor the integrity of the files in their Dataverse, and would prefer the publishing process to take less time). See the Configuration section of the [Installation Guide](http://guides.dataverse.org/en/5.0/installation/config.rst). - -Please note that we are not aware of any bugs in the current versions of Dataverse that would result in damage to users' files. But you may have some legacy files in your archive that were affected by some issue in the past, or perhaps affected by something outside Dataverse, so we are adding this feature out of abundance of caution. An example of a problem we've experienced in the early versions of Dataverse was a possible scenario where a user actually attempted to delete a Draft file from an unpublished version, where the database transaction would fail for whatever reason, but only after the physical file had already been deleted from the filesystem. Thus resulting in a datafile entry remaining in the dataset, but with the corresponding physical file missing. The fix for this case, since the user wanted to delete the file in the first place, is simply to confirm it and purge the datafile entity from the database. - -### The Setting :PIDAsynchRegFileCount is Deprecated as of 5.0 - -It used to specify the number of datafiles in the dataset to warrant adding a lock during publishing. As of v5.0 all datasets get locked for the duration of the publishing process. The setting will be ignored if present. - -### Location Changes for Related Projects - -The dataverse-ansible and dataverse-previewers repositories have been moved to the GDCC Organization on GitHub. If you have been referencing the dataverse-ansible repository from IQSS and the dataverse-previewers from QDR, please instead use them from their new locations: - - - - -### Harvesting Improvements - -Many updates have been made to address common Harvesting failures. You may see Harvests complete more often and have a higher success rate on a dataset-by-dataset basis. - -### New JVM Options and Database Settings - -Several new JVM options and DB Settings have been added in this release. More documentation about each of these settings can be found in the Configuration section of the [Installation Guide](http://guides.dataverse.org/en/5.0/installation/config.rst). - -#### New JVM Options - -- doi.dataciterestapiurlstring: Set with a value of "https://api.datacite.org" for production environments and "https://api.test.datacite.org" for test environments. Must be set if you are using DataCite as your DOI provider. -- dataverse.useripaddresssourceheader: If set, specifies an HTTP Header such as X-Forwarded-For to use to retrieve the user's IP address. This setting is useful in cases such as running Dataverse behind load balancers where the default option of getting the Remote Address from the servlet isn't correct (e.g. it would be the load balancer IP address). Note that unless your installation always sets the header you configure here, this could be used as a way to spoof the user's address. See the Configuration section of the [Installation Guide](http://guides.dataverse.org/en/5.0/installation/config.rst) for more information about proper use and security concerns. -- http.request-timeout-seconds: To facilitate large file upload and download, the Dataverse installer bumps the Payara **server-config.network-config.protocols.protocol.http-listener-1.http.request-timeout-seconds** setting from its default 900 seconds (15 minutes) to 1800 (30 minutes). - -#### New Database Settings - -- :CustomZipDownloadServiceUrl: If defined, this is the URL of the zipping service outside the main application server where zip downloads should be directed (instead of /api/access/datafiles/). -- :ShibAttributeCharacterSetConversionEnabled: By default, all attributes received from Shibboleth are converted from ISO-8859-1 to UTF-8. You can disable this behavior by setting to false. -- :ChronologicalDateFacets: Facets with Date/Year are sorted chronologically by default, with the most recent value first. To have them sorted by number of hits, e.g. with the year with the most results first, set this to false. -- :NavbarGuidesUrl: Set to a fully-qualified URL which will be used for the "User Guide" link in the navbar. -- :FileValidationOnPublishEnabled: Toggles validation of the physical files in the dataset when it's published, by recalculating the checksums and comparing against the values stored in the DataFile table. By default this setting is absent and Dataverse assumes it to be true. If enabled, the validation will be performed asynchronously, similarly to how we handle assigning persistent identifiers to datafiles, with the dataset locked for the duration of the publishing process. - -### Custom Analytics Code Changes - -You should update your custom analytics code to implement necessary changes for tracking updated dataset and file buttons. There was also a fix to the analytics code that will now properly track downloads for tabular files. - -For more information, see the documentation and sample analytics code snippet provided in [Installation Guide > Configuration > Web Analytics Code](http://guides.dataverse.org/en/5.0/installation/config.html#web-analytics-code) to reflect the changes implemented in this version (#6938/#6684). - -### Tracking Users' IP Addresses Behind an Address-Masking Proxy - -It is now possible to collect real user IP addresses in MDC logs and/or set up an IP group on a system running behind a proxy/load balancer that hides the addresses of incoming requests. See "Recording User IP Addresses" in the Configuration section of the [Installation Guide](http://guides.dataverse.org/en/5.0/installation/config.rst). - -### Reload Astrophysics Metadata Block (if used) - -Tooltips have been updated for the Astrophysics Metadata Block. If you'd like these updated Tooltips to be displayed to users of your installation, you should update the Astrophysics Metadata Block: - -`curl http://localhost:8080/api/admin/datasetfield/load -X POST --data-binary @astrophysics.tsv -H "Content-type: text/tab-separated-values"` - -We've included this in the step-by-step instructions below. - -### Run ReExportall - -We made changes to the JSON Export in this release. If you'd like these changes to reflected in your JSON exports, you should run ReExportall as part of the upgrade process following the steps in [Admin Guide](http://guides.dataverse.org/en/5.0/admin/metadataexport.html?highlight=export#batch-exports-through-the-api) - -We've included this in the step-by-step instructions below. - -## Notes for Tool Developers and Integrators - -## Complete List of Changes - -For the complete list of code changes in this release, see the [5.0 Milestone](https://github.com/IQSS/dataverse/milestone/89?closed=1) in Github. - -For help with upgrading, installing, or general questions please post to the [Dataverse Google Group](https://groups.google.com/forum/#!forum/dataverse-community) or email support@dataverse.org. - -## Installation - -If this is a new installation, please see our [Installation Guide](http://guides.dataverse.org/en/5.0/installation/) - -## Upgrade Instructions - -### Prerequisite: Retroactively store original file size - -Starting with release 4.10 the size of the saved original file (for an ingested tabular datafile) is stored in the database. We provided the following API that retrieve and permanently store the sizes for any already existing saved originals: - -`/api/admin/datafiles/integrity/fixmissingoriginalsizes` - -(See the documentation note in the Native API guide, under "[Datafile Integrity](https://guides.dataverse.org/en/5.0/api/native-api.html#datafile-integrity)"). - -To check your installation, issue this command: - - `curl http://localhost:8080/api/admin/datafiles/integrity/fixmissingoriginalsizes` - -### Upgrade from Glassfish 4.1 to Payara 5 - -The instructions below describe the upgrade procedure based on moving an existing glassfish4 domain directory under Payara. We recommend this method instead of setting up a brand-new Payara domain using the installer because it appears to be the easiest way to recreate your current configuration and preserve all your data. - -1. Download Payara, v5.2020.2 as of this writing: - - `curl -L -O https://github.com/payara/Payara/releases/download/payara-server-5.2020.2/payara-5.2020.2.zip` - `sha256sum payara-5.2020.2.zip` - 1f5f7ea30901b1b4c7bcdfa5591881a700c9b7e2022ae3894192ba97eb83cc3e - -2. Unzip it somewhere (/usr/local is a safe bet) - - `sudo unzip payara-5.2020.2.zip -d /usr/local/` - -3. Copy the Postgres driver to /usr/local/payara5/glassfish/lib - - `sudo cp /usr/local/glassfish4/glassfish/lib/postgresql-42.2.9.jar /usr/local/payara5/glassfish/lib/` - -4. Move payara5/glassfish/domains/domain1 out of the way - - `sudo mv /usr/local/payara5/glassfish/domains/domain1 /usr/local/payara5/glassfish/domains/domain1.orig` - -5. Undeploy the Dataverse web application (if deployed; version 4.20 is assumed in the example below) - - `sudo /usr/local/glassfish4/bin/asadmin list-applications` - `sudo /usr/local/glassfish4/bin/asadmin undeploy dataverse-4.20` - -6. Stop Glassfish; copy domain1 to Payara - - `sudo /usr/local/glassfish4/bin/asadmin stop-domain` - `sudo cp -ar /usr/local/glassfish4/glassfish/domains/domain1 /usr/local/payara5/glassfish/domains/` - -7. Remove the cache directories - - `sudo rm -rf /usr/local/payara5/glassfish/domains/domain1/generated/` - `sudo rm -rf /usr/local/payara5/glassfish/domains/domain1/osgi-cache/` - -8. Make the following changes in **domain.xml**: - - Replace the `-XX:PermSize` and `-XX:MaxPermSize` JVM options with `-XX:MetaspaceSize` and `-XX:MaxMetaspaceSize` - - ``` - -XX:MetaspaceSize=256m - -XX:MaxMetaspaceSize=512m - ``` - - Add the below JVM options beneath the -Ddataverse settings: - - ``` - -Dfish.payara.classloading.delegate=false - -XX:+UseG1GC - -XX:+UseStringDeduplication - -XX:+DisableExplicitGC - ``` - - Replace the following element: - - ``` - - - - - ``` - - with - - ``` - - - - ``` - -9. Change any full pathnames matching `/usr/local/glassfish4/...` to `/usr/local/payara5/...` or whatever it is in your case. Specifically check the `-Ddataverse.files.directory` and `-Ddataverse.files.file.directory` JVM options. - -10. In domain1/config/jhove.conf, change the hard-coded /usr/local/glassfish4 path, as above. - - (Optional): If you renamed your service account from glassfish to payara or appserver, update the ownership permissions. The Installation Guide recommends a service account of `dataverse`: - - `sudo chown -R dataverse /usr/local/payara5/glassfish/domains/domain1` - `sudo chown -R dataverse /usr/local/payara5/glassfish/lib` - -11. You will also need to check that the service account has write permission on the files directory, if they are located outside the old Glassfish domain. And/or make sure the service account has the correct AWS credentials, if you are using S3 for storage. - -12. Finally, start Payara: - - `sudo -u dataverse /usr/local/payara5/bin/asadmin start-domain` - -13. Deploy the Dataverse 5 warfile: - - `sudo -u dataverse /usr/local/payara5/bin/asadmin deploy /path/to/dataverse-5.0.war` - -14. Then restart Payara: - - `sudo -u dataverse /usr/local/payara5/bin/asadmin stop-domain` - `sudo -u dataverse /usr/local/payara5/bin/asadmin start-domain` - -### Additional Upgrade Steps - -1. Update Astrophysics Metadata Block (if used) - - `wget https://github.com/IQSS/dataverse/releases/download/v5.0/astrophysics.tsv` - `curl http://localhost:8080/api/admin/datasetfield/load -X POST --data-binary @astrophysics.tsv -H "Content-type: text/tab-separated-values"` - -2. (Recommended) Run ReExportall to update JSON Exports - - - -3. (Required for installations using DataCite) Add the JVM option doi.dataciterestapiurlstring - - For production environments: - - `/usr/local/payara5/bin/asadmin create-jvm-options "\-Ddoi.dataciterestapiurlstring=https\://api.datacite.org"` - - For test environments: - - `/usr/local/payara5/bin/asadmin create-jvm-options "\-Ddoi.dataciterestapiurlstring=https\://api.test.datacite.org"` - - The JVM option `doi.mdcbaseurlstring` should be deleted if it was previously set, for example: - - `/usr/local/payara5/bin/asadmin delete-jvm-options "\-Ddoi.mdcbaseurlstring=https\://api.test.datacite.org"` - -4. (Recommended for installations using DataCite) Pre-register DOIs - - Execute the script described in the section "Dataverse Installations Using DataCite: Upgrade Action Recommended" earlier in the Release Note. - - Please consult the earlier sections of the Release Note for any additional configuration options that may apply to your installation. diff --git a/doc/release-notes/5.1-release-notes.md b/doc/release-notes/5.1-release-notes.md deleted file mode 100644 index 3d106b2df7b..00000000000 --- a/doc/release-notes/5.1-release-notes.md +++ /dev/null @@ -1,99 +0,0 @@ -# Dataverse 5.1 - -This release brings new features, enhancements, and bug fixes to Dataverse. Thank you to all of the community members who contributed code, suggestions, bug reports, and other assistance across the project. - -## Release Highlights - -### Large File Upload for Installations Using AWS S3 - -The added support for multipart upload through the API and UI (Issue #6763) will allow files larger than 5 GB to be uploaded to Dataverse when an installation is running on AWS S3. Previously, only non-AWS S3 storage configurations would allow uploads larger than 5 GB. - -### Dataset-Specific Stores - -In previous releases, configuration options were added that allow each dataverse to have a specific store enabled. This release adds even more granularity, with the ability to set a dataset-level store. - -## Major Use Cases - -Newly-supported use cases in this release include: - -- Users can now upload files larger than 5 GB on installations running AWS S3 (Issue #6763, PR #6995) -- Administrators will now be able to specify a store at the dataset level in addition to the Dataverse level (Issue #6872, PR #7272) -- Users will have their dataset's directory structure retained when uploading a dataset with shapefiles (Issue #6873, PR #7279) -- Users will now be able to download zip files through the experimental Zipper service when the set of downloaded files have duplicate names (Issue [#80](https://github.com/IQSS/dataverse.harvard.edu/issues/80), PR #7276) -- Users will now be able to download zip files with the proper file structure through the experiment Zipper service (Issue #7255, PR #7258) -- Administrators will be able to use new APIs to keep the Solr index and the DB in sync, allowing easier resolution of an issue that would occasionally cause stale search results to not load. (Issue #4225, PR #7211) - -## Notes for Dataverse Installation Administrators - -### New API for setting a Dataset-level Store - -- This release adds a new API for setting a dataset-specific store. Learn more in the Managing Dataverse and Datasets section of the [Admin Guide](http://guides.dataverse.org/en/5.1/admin/solr-search-index.html). - -### Multipart Upload Storage Monitoring, Recommended Use for Multipart Upload - -Charges may be incurred for storage reserved for multipart uploads that are not completed or cancelled. Administrators may want to do periodic manual or automated checks for open multipart uploads. Learn more in the Big Data Support section of the [Developers Guide](http://guides.dataverse.org/en/5.1/developer/big-data-support.html). - -While multipart uploads can support much larger files, and can have advantages in terms of robust transfer and speed, they are more complex than single part direct uploads. Administrators should consider taking advantage of the options to limit use of multipart uploads to specific users by using multiple stores and configuring access to stores with high file size limits to specific Dataverses (added in 4.20) or Datasets (added in this release). - -### New APIs for keeping Solr records in sync - -This release adds new APIs to keep the Solr index and the DB in sync, allowing easier resolution of an issue that would occasionally cause search results to not load. Learn more in the Solr section of the [Admin Guide](http://guides.dataverse.org/en/5.1/admin/solr-search-index.html). - -### Documentation for Purging the Ingest Queue - -At times, it may be necessary to cancel long-running Ingest jobs in the interest of system stability. The Troubleshooting section of the [Admin Guide](http://guides.dataverse.org/en/5.1/admin/) now has specific steps. - -### Biomedical Metadata Block Updated - -The Life Science Metadata block (biomedical.tsv) was updated. "Other Design Type", "Other Factor Type", "Other Technology Type", "Other Technology Platform" boxes were added. See the "Additional Upgrade Steps" below if you use this in your installation. - -## Notes for Tool Developers and Integrators - -### Spaces in File Names - -Dataverse Installations using S3 storage will no longer replace spaces in file names of downloaded files with the + character. If your tool or integration has any special handling around this, you may need to make further adjustments to maintain backwards compatibility while also supporting Dataverse installations on 5.1+. - -## Complete List of Changes - -For the complete list of code changes in this release, see the [5.1 Milestone](https://github.com/IQSS/dataverse/milestone/90?closed=1) in Github. - -For help with upgrading, installing, or general questions please post to the [Dataverse Google Group](https://groups.google.com/forum/#!forum/dataverse-community) or email support@dataverse.org. - -## Installation - -If this is a new installation, please see our [Installation Guide](http://guides.dataverse.org/en/5.1/installation/) - -## Upgrade Instructions - -0. These instructions assume that you've already successfully upgraded from Dataverse 4.x to Dataverse 5 following the instructions in the [Dataverse 5 Release Notes](https://github.com/IQSS/dataverse/releases/tag/v5.0). - -1. Undeploy the previous version. - -/payara/bin/asadmin list-applications -/payara/bin/asadmin undeploy dataverse - -2. Stop payara and remove the generated directory, start. - -- service payara stop -- remove the generated directory: rm -rf payara/payara/domains/domain1/generated -- service payara start - -3. Deploy this version. -/payara/bin/asadmin deploy dataverse-5.1.war - -4. Restart payara - -### Additional Upgrade Steps - -1. Update Biomedical Metadata Block (if used), Reload Solr, ReExportAll - - `wget https://github.com/IQSS/dataverse/releases/download/v5.1/biomedical.tsv` - `curl http://localhost:8080/api/admin/datasetfield/load -X POST --data-binary @biomedical.tsv -H "Content-type: text/tab-separated-values"` - -- copy schema_dv_mdb_fields.xml and schema_dv_mdb_copies.xml to solr server, for example into /usr/local/solr/solr-7.7.2/server/solr/collection1/conf/ directory -- Restart Solr, or tell Solr to reload its configuration: - - `curl "http://localhost:8983/solr/admin/cores?action=RELOAD&core=collection1"` - -- Run ReExportall to update JSON Exports - diff --git a/doc/release-notes/5.1.1-release-notes.md b/doc/release-notes/5.1.1-release-notes.md deleted file mode 100644 index 739bd3da800..00000000000 --- a/doc/release-notes/5.1.1-release-notes.md +++ /dev/null @@ -1,63 +0,0 @@ -# Dataverse 5.1.1 - -This minor release adds important scaling improvements for installations running on AWS S3. It is recommended that 5.1.1 be used in production instead of 5.1. - -## Release Highlights - -### Connection Pool Size Configuration Option, Connection Optimizations - -Dataverse 5.1 improved the efficiency of making S3 connections through use of an http connection pool. This release adds optimizations around closing streams and channels that may hold S3 http connections open and exhaust the connection pool. In parallel, this release increases the default pool size from 50 to 256 and adds the ability to increase the size of the connection pool, so a larger pool can be configured if needed. - -## Major Use Cases - -Newly-supported use cases in this release include: - -- Administrators of installations using S3 will be able to define the connection pool size, allowing better resource scaling for larger installations (Issue #7309, PR #7313) - -## Notes for Dataverse Installation Administrators - -### 5.1.1 vs. 5.1 for Production Use - -As mentioned above, we encourage 5.1.1 instead of 5.1 for production use. - -### New JVM Option for Connection Pool Size - -Larger installations may want to increase the number of open S3 connections allowed (default is 256). For example, to set the value to 4096: - -`./asadmin create-jvm-options "-Ddataverse.files..connection-pool-size=4096"` -(where `` is the identifier of your S3 file store (likely `"s3"`). The JVM Options section of the [Configuration Guide](http://guides.dataverse.org/en/5.1.1/installation/config/) has more information. - -### New S3 Bucket CORS setting for Direct Upload/Download - -When using S3 storage with direct upload and/or download enabled, one must now expose the ETag header as documented in the [updated cors.json example](https://guides.dataverse.org/en/5.1.1/developers/big-data-support.html?highlight=etag#s3-direct-upload-and-download). - -## Complete List of Changes - -For the complete list of code changes in this release, see the [5.1.1 Milestone](https://github.com/IQSS/dataverse/milestone/91?closed=1) in Github. - -For help with upgrading, installing, or general questions please post to the [Dataverse Google Group](https://groups.google.com/forum/#!forum/dataverse-community) or email support@dataverse.org. - -## Installation - -If this is a new installation, please see our [Installation Guide](http://guides.dataverse.org/en/5.1.1/installation/) - -## Upgrade Instructions - -0. These instructions assume that you've already successfully upgraded to Dataverse 5.1 following the instructions in the [Dataverse 5.1 Release Notes](https://github.com/IQSS/dataverse/releases/tag/v5.1). - -1. Undeploy the previous version. - -`/bin/asadmin list-applications` -`/bin/asadmin undeploy dataverse<-version>` - -2. Stop payara and remove the generated directory, start. - -- `service payara stop` -- remove the generated directory: -`rm -rf /glassfish/domains/domain1/generated` -- `service payara start` - -3. Deploy this version. -`/bin/asadmin deploy dataverse-5.1.1.war` - -4. Restart payara diff --git a/doc/release-notes/5.2-release-notes.md b/doc/release-notes/5.2-release-notes.md deleted file mode 100644 index 5ed7afac902..00000000000 --- a/doc/release-notes/5.2-release-notes.md +++ /dev/null @@ -1,105 +0,0 @@ -# Dataverse 5.2 - -This release brings new features, enhancements, and bug fixes to Dataverse. Thank you to all of the community members who contributed code, suggestions, bug reports, and other assistance across the project. - -## Release Highlights - -### File Preview When Guestbooks or Terms Exist - -Previously, file preview was only available when files were publicly downloadable. Now if a guestbook or terms (or both) are configured for the dataset, they will be shown in the Preview tab and once they are agreed to, the file preview will appear (#6919). - -### Preview Only External Tools - -A new external tool type called "preview" has been added that prevents the tool from being displayed under "Explore Options" under the "Access File" button on the file landing page (#6919). - -### Dataset Page Edit Options Consolidation - -As part of the continued effort to redesign the Dataset and File pages, some of the edit options for a file on the dataset page are being moved to a "kebab" to allow for better consistency and future scalability. - -### Google Cloud Archiver - -Dataverse Bags can now be sent to a bucket in Google Cloud, including those in the "Coldline" storage class, which provides less expensive but slower access. - -## Major Use Cases - -Newly-supported use cases in this release include: - -- Users can now preview files that have a guestbook or terms. (Issue #6919, PR #7369) -- External tool developers can indicate that their tool is "preview only". (Issue #6919, PR #7369) -- Dataverse Administrators can set up a regular export to Google Cloud so that the installation's data is preserved (Issue #7140, PR #7292) -- Dataverse Administrators can use a regex when defining a group (Issue #7344, PR #7351) -- External Tool Developers can use a new API endpoint to retrieve a user's information (Issue #7307, PR #7345) - -## Notes for Dataverse Installation Administrators - -### Converting Explore External Tools to Preview Only - -When the war file is deployed, a SQL migration script will convert [dataverse-previewers][] to have both "explore" and "preview" types so that they will continue to be displayed in the Preview tab. - -If you would prefer that these tools be preview only, you can delete the tools, adjust the JSON manifests (changing "explore" to "preview"), and re-add them. - -[dataverse-previewers]: https://github.com/GlobalDataverseCommunityConsortium/dataverse-previewers - -### New Database Settings and JVM Options - -Installations integrating with Google Cloud Archiver will need to use two new database settings: - -- :GoogleCloudProject - the name of the project managing the bucket -- :GoogleCloudBucket - the name of the bucket to use - -For more information, see the Google Cloud Configuration section of the [Installation Guide](https://guides.dataverse.org/en/5.2/installation/) - -### Automation of Make Data Count Scripts - -Scripts have been added in order to automate Make Data Count processing. For more information, see the Make Data Count section of the [Admin Guide](https://guides.dataverse.org/en/5.2/admin/). - -## Notes for Tool Developers and Integrators - -### Preview Only External Tools, "hasPreviewMode" - -A new external tool type called "preview" has been added that prevents the tool from being displayed under "Explore Options" under the "Access File" button on the file landing page (#6919). This "preview" type replaces "hasPreviewMode", which has been removed. - -### Multiple Types for External Tools - -External tools now support multiple types. In practice, the types "explore" and "preview" are the only combination that makes a difference in the UI as opposed to only having only one or the other type (see "preview only" above). Multiple types are specified in the JSON manifest with an array in "types". The older, single "type" is still supported but should be considered deprecated. - -### User Information Endpoint - -New API endpoint to retrieve user info so that tools can email users if needed. - -## Complete List of Changes - -For the complete list of code changes in this release, see the [5.2 Milestone](https://github.com/IQSS/dataverse/milestone/92?closed=1) in Github. - -For help with upgrading, installing, or general questions please post to the [Dataverse Google Group](https://groups.google.com/forum/#!forum/dataverse-community) or email support@dataverse.org. - -## Installation - -If this is a new installation, please see our [Installation Guide](https://guides.dataverse.org/en/5.2/installation/). - -## Upgrade Instructions - -0\. These instructions assume that you've already successfully upgraded from Dataverse 4.x to Dataverse 5 following the instructions in the [Dataverse 5 Release Notes](https://github.com/IQSS/dataverse/releases/tag/v5.0). - -1\. Undeploy the previous version. - -- `/bin/asadmin list-applications` -- `/bin/asadmin undeploy dataverse<-version>` - -(where `` is where Payara 5 is installed, for example: `/usr/local/payara5`) - -2\. Stop payara and remove the generated directory, start. - -- `service payara stop` -- remove the generated directory: -`rm -rf /payara/domains/domain1/generated` -- `service payara start` - -3\. Deploy this version. - -- `/bin/asadmin deploy dataverse-5.2.war` - -4\. Restart payara - -- `service payara stop` -- `service payara start` diff --git a/doc/release-notes/5.3-release-notes.md b/doc/release-notes/5.3-release-notes.md deleted file mode 100644 index e62da980a3e..00000000000 --- a/doc/release-notes/5.3-release-notes.md +++ /dev/null @@ -1,164 +0,0 @@ -# Dataverse 5.3 - -This release brings new features, enhancements, and bug fixes to Dataverse. Thank you to all of the community members who contributed code, suggestions, bug reports, and other assistance across the project. - -## Release Highlights - -### Auxiliary Files (Experimental) - -Auxiliary files can now be added to datafiles and accessed using new experimental API endpoints. These endpoints allow additional, non-Dataverse generated metadata to be added alongside datafiles in dataverse. - -The support for auxiliary files in Dataverse is being driven by integration with the [Open Differential Privacy (DP) Project](https://opendp.io) and is designed to support the deposit and retrieval of differentially private metadata, but the endpoints are not specific to differential privacy use cases. - -### Additional Banner Functionality - -Banners in Dataverse can now be set to allow dismissal by a logged in user. Previously, banners would persist until they were removed by an administrator. This allows administrators to more easily communicate one-time messages to users. - -### File Tags Searchable from Advanced Search and Dataset Search - -File tags ("Documentation", "Data", "Code", etc.) now appear on the Advanced Search page. - -Performing a search for files on the dataset page now includes file tags. Previously, only file name and file description were searched. - -### Easier Configuration of Database Connections - -Previously, the configuration of the database connections has been quite static and not very easy to update. This has been an issue especially for cloud and container usage. Using new technologies provided by the move to Payara, you can now more easily configure the connection to your PostgreSQL DB. - -Using MicroProfile Config API (Issue #7000, Issue #7418), you can much more easily specify configuration -details. For an overview of supported options, please see the -[Installation Guide](https://guides.dataverse.org/en/5.3/installation/config.html#jvm-options). - -Note that some settings have been moved from domain.xml to code, such as min and max pool size. - -## Major Use Cases - -Newly-supported use cases in this release include: - -- Users can use an API to add auxiliary files to files in order to provide metadata representations for specific tools or integrations (Issue #7275, PR #7350) -- Administrators can use a new API to manage banner messages and take advantage of new banner display options (Issue #7263, PR #7434) -- Users replacing files will now have their files renamed when a file name conflict exists, making the behavior consistent with upload and edit (Issue #7335, PR #7336) -- Users will now be able to search on file tags on the advanced search and dataset pages (Issue #7194, PR #7385) - -## Notes for Dataverse Installation Administrators - -### Payara 5.2020.6 (or Higher) Required - -Some changes in this release require an upgrade to Payara 5.2020.6 or higher. - -Instructions on how to update can be found in the -[Payara documentation](https://docs.payara.fish/community/docs/5.2020.6/documentation/user-guides/upgrade-payara.html) - -### New Banner API, Obsolete DB Settings - -The functionality previously provided by the DB settings :StatusMessageHeader and ::StatusMessageText is no longer supported and is now provided through the Manage Banner Messages API. Learn more in the [API Guide](https://guides.dataverse.org/en/5.3/api/). - -### New Database Settings and JVM Options - -Several new JVM options have been added in this release: - -- dataverse.db.name -- dataverse.db.user -- dataverse.db.password -- dataverse.db.host -- dataverse.db.port - -For an overview of these new options, please see the -[Installation Guide](https://guides.dataverse.org/en/5.3/installation/config.html#jvm-options) - -See above note about obsolete DB options. - -### Introducing MicroProfile Config API - -With this Dataverse release, Dataverse Administrators can start to make use of the MicroProfile Config API. - -This will benefit both developers and sysadmins, but the codebase will have to be refactored to make use of it. As this will take time, we will always provide a backward compatible way of using it. - -For more details about these new options, please see the Consuming Configuration section of the [Developer Guide](https://guides.dataverse.org/en/5.3/developers/). - -### Java Message System Configuration - -The Ingest process uses the Java Message System to create ingest tasks in a queue. That queue had been configured from command line or domain.xml before. This has now changed to being done -in code. - -In the unlikely case you might want to change any of these settings, feel free to change and recompile or raise an issue on Github. See `IngestQueueProducer` for more details. - -If you want to clean up your existing installation, you can delete the old, unused queue like this: - -- `/bin/asadmin delete-connector-connection-pool --cascade=true jms IngestQueueConnectionFactoryPool` - -## Notes for Tool Developers and Integrators - -### Experimental Auxiliary File Support - -Experimental endpoints have been added to allow auxiliary files to be added to datafiles. These auxiliary files can be deposited and accessed via API. Later releases will include options for accessing these files through the UI. For more information, see the Auxiliary File Support section of the [Developer Guide](https://guides.dataverse.org/en/5.3/developers/). - -## Complete List of Changes - -For the complete list of code changes in this release, see the [5.3 Milestone](https://github.com/IQSS/dataverse/milestone/93?closed=1) in Github. - -For help with upgrading, installing, or general questions please post to the [Dataverse Google Group](https://groups.google.com/forum/#!forum/dataverse-community) or email support@dataverse.org. - -## Installation - -If this is a new installation, please see our [Installation Guide](https://guides.dataverse.org/en/5.3/installation/). - -## Upgrade Instructions - -0\. These instructions assume that you've already successfully upgraded from Dataverse 4.x to Dataverse 5 following the instructions in the [Dataverse 5 Release Notes](https://github.com/IQSS/dataverse/releases/tag/v5.0). - -1\. Upgrade to Payara 5.2020.6 or higher. - -Instructions on how to update can be found in the -[Payara documentation](https://docs.payara.fish/community/docs/5.2020.6/documentation/user-guides/upgrade-payara.html). - -It would likely be safer to upgrade Payara first, while still running Dataverse 5.2, and then proceed with the steps below. Upgrading from an earlier version of Payara should be a straightforward process: Undeploy Dataverse; stop Payara; move the current Payara directory out of the way; unzip the new Payara version in its place; replace **the brand new `payara/glassfish/domains/domain1` with your old, preserved `domain1`**; start Payara, deploy Dataverse 5.2. We still recommend that you read the detailed upgrade instructions above; and, if you run into any issues with this upgrade, it will help to be able to separate them from any problems with the upgrade of Dataverse proper. -If you are still using pre-5.0 version of Dataverse, and Glassfish version 4, please follow the upgrade instructions in the Dataverse 5.0 release notes; but use the latest version of Payara 5 (5.2020.7, as of this writing). - -2\. Undeploy the previous version. - -- `/bin/asadmin list-applications` -- `/bin/asadmin undeploy dataverse<-version>` - -(where `` is where Payara 5 is installed, for example: `/usr/local/payara5`) - -3\. Update your database connection. - -Please configure your connection details, replacing all the `${DB_...}`. - -- `/bin/asadmin create-system-properties "dataverse.db.user=${DB_USER}"` -- `/bin/asadmin create-system-properties "dataverse.db.host=${DB_HOST}"` -- `/bin/asadmin create-system-properties "dataverse.db.port=${DB_PORT}"` -- `/bin/asadmin create-system-properties "dataverse.db.name=${DB_NAME}"` -- `echo "AS_ADMIN_ALIASPASSWORD=${DB_PASS}" > /tmp/password.txt` - -- `/bin/asadmin create-password-alias --passwordfile /tmp/password.txt dataverse.db.password` -- `rm /tmp/password.txt` - -4\. In domain.xml, verify that the __TimerPool jdbc-connection-pool is using the H2 database, as follows (if you have the old Derby version from Glassfish 4, replace it): - - ` ` - -5\. Reset the EJB timer database back to default: - -- `/bin/asadmin set configs.config.server-config.ejb-container.ejb-timer-service.timer-datasource=jdbc/__TimerPool` - -6\. Delete the old password alias and DB pool: - -- `/bin/asadmin delete-jdbc-connection-pool --cascade=true dvnDbPool` -- `/bin/asadmin delete-password-alias db_password_alias` - -7\. Stop payara, remove the generated and ejbtimer database directories, then restart. - -- `service payara stop` -- `rm -rf /glassfish/domains/domain1/generated` -- `rm -rf /glassfish/domains/domain1/lib/databases/ejbtimer` -- `service payara start` - -8\. Deploy this version. - -- `/bin/asadmin deploy dataverse-5.3.war` - -9\. Restart payara - -- `service payara stop` -- `service payara start` diff --git a/doc/release-notes/5.4-release-notes.md b/doc/release-notes/5.4-release-notes.md deleted file mode 100644 index bf4b2b43d85..00000000000 --- a/doc/release-notes/5.4-release-notes.md +++ /dev/null @@ -1,344 +0,0 @@ -# Dataverse Software 5.4 - -This release brings new features, enhancements, and bug fixes to the Dataverse Software. Thank you to all of the community members who contributed code, suggestions, bug reports, and other assistance across the project. - -## Release Highlights - -### Deactivate Users API, Get User Traces API, Revoke Roles API - -A new API has been added to deactivate users to prevent them from logging in, receiving communications, or otherwise being active in the system. Deactivating a user is an alternative to deleting a user, especially when the latter is not possible due to the amount of interaction the user has had with the Dataverse installation. In order to learn more about a user before deleting, deactivating, or merging, a new "get user traces" API is available that will show objects created, roles, group memberships, and more. Finally, the "remove all roles" button available in the superuser dashboard is now also available via API. - -### New File Access API - -A new API offers *crawlable* access view of the folders and files within a dataset: - -`/api/datasets//dirindex/` - -will output a simple html listing, based on the standard Apache directory index, with Access API download links for individual files, and recursive calls to the API above for sub-folders. Please see the [Native API Guide](https://guides.dataverse.org/en/5.4/api/native-api.html) for more information. - -Using this API, ``wget --recursive`` (or similar crawling client) can be used to download all the files in a dataset, preserving the file names and folder structure; without having to use the download-as-zip API. In addition to being faster (zipping is a relatively resource-intensive operation on the server side), this process can be restarted if interrupted (with ``wget --continue`` or equivalent) - unlike zipped multi-file downloads that always have to start from the beginning. - -On a system that uses S3 with download redirects, the individual file downloads will be handled by S3 directly (with the exception of tabular files), without having to be proxied through the Dataverse application. - -### Restricted Files and DDI "dataDscr" Information (Summary Statistics, Variable Names, Variable Labels) - -In previous releases, DDI "dataDscr" information (summary statistics, variable names, and variable labels, sometimes known as "variable metadata") for tabular files that were ingested successfully were available even if files were restricted. This has been changed in the following ways: - -- At the dataset level, DDI exports no longer show "dataDscr" information for restricted files. There is only one version of this export and it is the version that's suitable for public consumption with the "dataDscr" information hidden for restricted files. -- Similarly, at the dataset level, the DDI HTML Codebook no longer shows "dataDscr" information for restricted files. -- At the file level, "dataDscr" information is no longer publicly available for restricted files. In practice, it was only possible to get this publicly via API (the download/access button was hidden). -- At the file level, "dataDscr" (variable metadata) information can still be downloaded for restricted files if you have access to download the file. - -### Search with Accented Characters - -Many languages include characters that have close analogs in ascii, e.g. (á, à, â, ç, é, è, ê, ë, í, ó, ö, ú, ù, û, ü…). This release changes the default Solr configuration to allow search to match words based on these associations, e.g. a search for Mercè would match the word Merce in a Dataset, and vice versa. This should generally be helpful, but can result in false positives, e.g. "canon" will be found searching for "cañon". - -### Java 11, PostgreSQL 13, and Solr 8 Support/Upgrades - -Several of the core components of the Dataverse Software have been upgraded. Specifically: - -- The Dataverse Software now runs on and requires Java 11. This will provide performance and security enhancements, allows developers to take advantage of new and updated Java features, and moves the project to a platform with better longer term support. This upgrade requires a few extra steps in the release process, outlined below. -- The Dataverse Software has now been tested with PostgreSQL versions up to 13. Versions 9.6+ will still work, but this update is necessary to support the software beyond PostgreSQL EOL later in 2021. -- The Dataverse Software now runs on Solr 8.8.1, the latest available stable release in the Solr 8.x series. - -### Saved Search Performance Improvements - -A refactoring has greatly improved Saved Search performance in the application. If your installation has multiple, potentially long-running Saved Searches in place, this greatly improves the probability that those search jobs will complete without timing out. - -### Worldmap/Geoconnect Integration Now Obsolete - -As of this release, the Geoconnect/Worldmap integration is no longer available. The Harvard University Worldmap is going through a migration process, and instead of updating this code to work with the new infrastructure, the decision was made to pursue future Geospatial exploration/analysis through other tools, following the External Tools Framework in the Dataverse Software. - -### Guides Updates - -The Dataverse Software Guides have been updated to follow recent changes to how different terms are used across the Dataverse Project. For more information, see Mercè's note to the community: - - - -### Conditionally Required Metadata Fields - -Prior to this release, when defining metadata for compound fields (via their dataset field types), fields could be either be optional or required, i.e. if required you must always have (at least one) value for that field. For example, Author Name being required means you must have at least one Author with an nonempty Author name. - -In order to support more robust metadata (and specifically to resolve #7551), we need to allow a third case: Conditionally Required, that is, the field is required if and only if any of its "sibling" fields are entered. For example, Producer Name is now conditionally required in the citation metadata block. A user does not have to enter a Producer, but if they do, they have to enter a Producer Name. - -## Major Use Cases - -Newly-supported major use cases in this release include: - -- Dataverse Installation Administrators can now deactivate users using a new API. (Issue #2419, PR #7629) -- Superusers can remove all of a user's assigned roles using a new API. (Issue #2419, PR #7629) -- Superusers can use an API to gather more information about actions a user has taken in the system in order to make an informed decisions about whether or not to deactivate or delete a user. (Issue #2419, PR #7629) -- Superusers will now be able to harvest from installations using ISO-639-3 language codes. (Issue #7638, PR #7690) -- Users interacting with the workflow system will receive status messages (Issue #7564, PR #7635) -- Users interacting with prepublication workflows will see speed improvements (Issue #7681, PR #7682) -- API Users will receive Dataverse collection API responses in a deterministic order. (Issue #7634, PR #7708) -- API Users will be able to access a list of crawlable URLs for file download, allowing for faster and easily resumable transfers. (Issue #7084, PR #7579) -- Users will no longer be able to access summary stats for restricted files. (Issue #7619, PR #7642) -- Users will now see truncated versions of long strings (primarily checksums) throughout the application (Issue #6685, PR #7312) -- Users will now be able to easily copy checksums, API tokens, and private URLs with a single click (Issue #6039, Issue #6685, PR #7539, PR #7312) -- Users uploading data through the Direct Upload API will now be able to use additional checksums (Issue #7600, PR #7602) -- Users searching for content will now be able to search using non-ascii characters. (Issue #820, PR #7378) -- Users can now replace files in draft datasets, a functionality previously only available on published datasets. (Issue #7149, PR #7337) -- Dataverse Installation Administrators can now set subfields of compound fields as **conditionally required**, that is, the field is required if and only if any of its "sibling" fields are entered. For example, Producer Name is now conditionally required in the citation metadata block. A user does not have to enter a Producer, but if they do, they have to enter a Producer Name. (Issue #7606, PR #7608) - -## Notes for Dataverse Installation Administrators - -### Java 11 Upgrade - -There are some things to note and keep in mind regarding the move to Java 11: - -- You should install the JDK/JRE following your usual methods, depending on your operating system. An example of this on a RHEL/CentOS 7 or RHEL/CentOS 8 system is: - - `$ sudo yum remove java-1.8.0-openjdk java-1.8.0-openjdk-devel java-1.8.0-openjdk-headless` - - `$ sudo yum install java-11-openjdk-devel` - - The `remove` command may provide an error message if -headless isn't installed. - - - Select the latest version of Java - - `$ sudo alternatives --config java` - - -- We targeted and tested Java 11, but 11+ will likely work. Java 11 was targeted because of its long term support. -- If you're moving from a Dataverse installation that was previously running Glassfish 4.x (typically this would be Dataverse Software 4.x), you will need to adjust some JVM options in domain.xml as part of the upgrade process. We've provided these optional steps below. These steps are not required if your first installed Dataverse Software version was running Payara 5.x (typically Dataverse Software 5.x). - -### PostgreSQL Versions Up To 13 Supported - -Up until this release our installation guide "strongly recommended" to install PostgreSQL v. 9.6. While that version is known to be very stable, it is nearing its end-of-life (in Nov. 2021). Dataverse Software has now been tested with versions up to 13. If you decide to upgrade PostgreSQL, the tested and recommended way of doing that is as follows: - -- Export your current database with ``pg_dumpall``; -- Install the new version of PostgreSQL; (make sure it's running on the same port, etc. so that no changes are needed in the Payara configuration) -- Re-import the database with ``psql``, as the postgres user. - -Consult the PostgreSQL upgrade documentation for more information, for example . - -### Solr Upgrade - -With this release we upgrade to the latest available stable release in the Solr 8.x branch. We recommend a fresh installation of Solr 8.8.1 (the index will be empty) followed by an "index all". - -Before you start the "index all", the Dataverse installation will appear to be empty because the search results come from Solr. As indexing progresses, partial results will appear until indexing is complete. - -See for more information. - -### Managing Conditionally Required Metadata Fields - -Prior to this release, when defining metadata for compound fields (via their dataset field types), fields could be either be optional or required, i.e. if required you must always have (at least one) value for that field. For example, Author Name being required means you must have at least one Author with an nonempty Author name. - -In order to support more robust metadata (and specifically to resolve #7551), we need to allow a third case: Conditionally Required, that is, the field is required if and only if any of its "sibling" fields are entered. For example, Producer Name is now conditionally required in the citation metadata block. A user does not have to enter a Producer, but if they do, they have to enter a Producer Name. - -This change required some modifications to how "required" is defined in the metadata .tsv files (for compound fields). - -Prior to this release, the value of required for the parent compound field did not matter and so was set to false. - -Going forward: - -- For optional, the parent compound field would be required = false and all children would be required = false. -- For required, the parent compound field would be required = true and at least one child would be required = true. -- For conditionally required, the parent compound field would be required = false and at least one child would be required = true. - -This release updates the citation .tsv file that is distributed with the software for the required parent compound fields (e.g. author), as well as sets Producer Name to be conditionally required. No other distributed .tsv files were updated, as they did not have any required compound values. - -**If you have created any custom metadata .tsv files**, you will need to make the same (type of) changes there. - -### Citation Metadata Block Update - -Due to the changes for Conditionally Required Metadata Fields, and a minor update in the citation metadata block to support extra ISO-639-3 language codes, a block upgrade is required. Instructions are provided below. - -### Retroactively Store Original File Size - -Beginning in Dataverse Software 4.10, the size of the saved original file (for an ingested tabular datafile) was stored in the database. For files added before this change, we provide an API that retrieves and permanently stores the sizes for any already existing saved originals. See [Datafile Integrity API](https://guides.dataverse.org/en/5.4/api/native-api.html#datafile-integrity) for more information. - -This was documented as a step in previous release notes, but we are noting it in these release notes to give it more visibility. - -### DB Cleanup for Saved Searches - -A previous version of the Dataverse Software changed the indexing logic so that when a user links a Dataverse collection, its children are also indexed as linked. This means that the children do not need to be separately linked, and in this version we removed the logic that creates a saved search to create those links when a Dataverse collection is linked. - -We recommend cleaning up the db to a) remove these saved searches and b) remove the links for the objects. We can do this via a few queries, which are available in the folder here: - - - -There are four sets of queries available, and they should be run in this order: - -- ss_for_deletion.txt to identify the Saved Searches to be deleted -- delete_ss.txt to delete the Saved Searches identified in the previous query -- dld_for_deletion.txt to identify the linked datasets and Dataverse collections to be deleted -- delete_dld.txt to delete the linked datasets and Dataverse collections identified in the previous queries - -Note: removing these saved searches and links should not affect what users will see as linked due to the aforementioned indexing change. Similarly, not removing these saved searches and links should not affect anything, but is a cleanup of unnecessary rows in the database. - -### DB Cleanup for Superusers Releasing without Version Updates - -In datasets where a superuser has run the Curate command and the update included a change to the fileaccessrequest flag, those changes would not be reflected appropriately in the published version. This should be a rare occurrence. - -Instead of an automated solution, we recommend inspecting the affected datasets and correcting the fileaccessrequest flag as appropriate. You can identify the affected datasets this via a query, which is available in the folder here: - - - -## New JVM Options and Database Settings - -For installations that were previously running on Dataverse Software 4.x, a number of new JVM options need to be added as part of the upgrade. The JVM Options are enumerated in the detailed upgrade instructions below. - -Two new Database settings were added: - -- :InstallationName -- :ExportInstallationAsDistributorOnlyWhenNotSet - -For an overview of these new options, please see the -[Installation Guide](https://guides.dataverse.org/en/5.4/installation/config.html#database-settings) - -## Notes for Tool Developers and Integrators - -### UTF-8 Characters and Spaces in File Names - -UTF-8 characters in filenames are now preserved when downloaded. - -Dataverse installations will no longer replace spaces in file names of downloaded files with the + character. If your tool or integration has any special handling around this, you may need to make further adjustments to maintain backwards compatibility while also supporting Dataverse installations on 5.4+. - -Note that this follows a change from 5.1 that only corrected this for installations running with S3 storage. This makes the behavior consistent across installations running all types of file storage. - -## Complete List of Changes - -For the complete list of code changes in this release, see the [5.4 Milestone](https://github.com/IQSS/dataverse/milestone/94?closed=1) in Github. - -For help with upgrading, installing, or general questions please post to the [Dataverse Community Google Group](https://groups.google.com/forum/#!forum/dataverse-community) or email support@dataverse.org. - -## Installation - -If this is a new installation, please see our [Installation Guide](https://guides.dataverse.org/en/5.4/installation/). - -## Upgrade Instructions - -0\. These instructions assume that you've already successfully upgraded from Dataverse Software 4.x to Dataverse Software 5 following the instructions in the [Dataverse Software 5 Release Notes](https://github.com/IQSS/dataverse/releases/tag/v5.0). After upgrading from the 4.x series to 5.0, you should progress through the other 5.x releases before attempting the upgrade to 5.4. - -1\. Upgrade to Java 11. - -2\. Upgrade to Solr 8.8.1. - -If you are running Payara as a non-root user (and you should be!), **remember not to execute the commands below as root**. Use `sudo` to change to that user first. For example, `sudo -i -u dataverse` if `dataverse` is your dedicated application user. - -In the following commands we assume that Payara 5 is installed in `/usr/local/payara5`. If not, adjust as needed. - -`export PAYARA=/usr/local/payara5` - -(or `setenv PAYARA /usr/local/payara5` if you are using a `csh`-like shell) - -3\. Undeploy the previous version. - -- `$PAYARA/bin/asadmin list-applications` -- `$PAYARA/bin/asadmin undeploy dataverse<-version>` - -4\. Stop Payara and remove the generated directory - -- `service payara stop` -- `rm -rf $PAYARA/glassfish/domains/domain1/generated` - -5\. **(only required for installations previously running Dataverse Software 4.x!)** In other words, if you have a domain.xml that **originated under Glassfish 4**, the below JVM Options need to be added. If your Dataverse installation was first installed on the 5.x series, these JVM options should already be present. - -In domain.xml: - -Remove the following JVM options from the `` section: - -`-Djava.endorsed.dirs=/usr/local/payara5/glassfish/modules/endorsed:/usr/local/payara5/glassfish/lib/endorsed` - -`-Djava.ext.dirs=${com.sun.aas.javaRoot}/lib/ext${path.separator}${com.sun.aas.javaRoot}/jre/lib/ext${path.separator}${com.sun.aas.instanceRoot}/lib/ext` - -Add the following JVM options to the `` section: - -`[9|]--add-opens=java.base/jdk.internal.loader=ALL-UNNAMED` - -`[9|]--add-opens=jdk.management/com.sun.management.internal=ALL-UNNAMED` - -`[9|]--add-exports=java.base/jdk.internal.ref=ALL-UNNAMED` - -`[9|]--add-opens=java.base/java.lang=ALL-UNNAMED` - -`[9|]--add-opens=java.base/java.net=ALL-UNNAMED` - -`[9|]--add-opens=java.base/java.nio=ALL-UNNAMED` - -`[9|]--add-opens=java.base/java.util=ALL-UNNAMED` - -`[9|]--add-opens=java.base/sun.nio.ch=ALL-UNNAMED` - -`[9|]--add-opens=java.management/sun.management=ALL-UNNAMED` - -`[9|]--add-opens=java.base/sun.net.www.protocol.jrt=ALL-UNNAMED` - -`[9|]--add-opens=java.base/sun.net.www.protocol.jar=ALL-UNNAMED` - -`[9|]--add-opens=java.naming/javax.naming.spi=ALL-UNNAMED` - -`[9|]--add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED` - -`[9|]--add-opens=java.logging/java.util.logging=ALL-UNNAMED` - -`[1.8.0u251|]-Xbootclasspath/a:${com.sun.aas.installRoot}/lib/grizzly-npn-api.jar` - -6\. Start Payara - -- `service payara start` - -7\. Deploy this version. - -- `$PAYARA/bin/asadmin deploy dataverse-5.4.war` - -8\. Restart payara - -- `service payara stop` -- `service payara start` - -9\. Reload Citation Metadata Block: - - `wget https://github.com/IQSS/dataverse/releases/download/v5.4/citation.tsv` - `curl http://localhost:8080/api/admin/datasetfield/load -X POST --data-binary @citation.tsv -H "Content-type: text/tab-separated-values"` - -## Additional Release Steps - -1\. Confirm that the schema.xml was updated with the new v5.4 version when you updated Solr. - -2\. Run the script updateSchemaMDB.sh to generate updated solr schema files and preserve any other custom fields in your Solr configuration. - -For example: (modify the path names as needed) - -cd /usr/local/solr-8.8.1/server/solr/collection1/conf -wget https://github.com/IQSS/dataverse/releases/download/v5.4/updateSchemaMDB.sh -chmod +x updateSchemaMDB.sh -./updateSchemaMDB.sh -t . - -See for more information. - -3\. Do a clean reindex by first clearing then indexing. Re-indexing is required to get full-functionality from this change. Please refer to the guides on how to clear and index if needed. - -4\. Upgrade Postgres. - -- Export your current database with ``pg_dumpall``; -- Install the new version of PostgreSQL; (make sure it's running on the same port, etc. so that no changes are needed in the Payara configuration) -- Re-import the database with ``psql``, as the postgres user. - -Consult the PostgreSQL upgrade documentation for more information, for example . - -5\. Retroactively store original file size - -Use the [Datafile Integrity API](https://guides.dataverse.org/en/5.4/api/native-api.html#datafile-integrity) to ensure that the sizes of all original files are stored in the database. - -6\. DB Cleanup for Superusers Releasing without Version Updates - -In datasets where a superuser has run the Curate command and the update included a change to the fileaccessrequest flag, those changes would not be reflected appropriately in the published version. This should be a rare occurrence. - -Instead of an automated solution, we recommend inspecting the affected datasets and correcting the fileaccessrequest flag as appropriate. You can identify the affected datasets this via a query, which is available in the folder here: - - - -7\. (Optional, but recommended) DB Cleanup for Saved Searches and Linked Objects - -Perform the DB Cleanup for Saved Searches and Linked Objects, summarized in the "Notes for Dataverse Installation Administrators" section above. - -8\. Take a backup of the Worldmap links, if any. - -9\. (Only required if custom metadata blocks are used in your Dataverse installation) Update any custom metadata blocks: - -In the .tsv for any custom metadata blocks, for any subfield that has a required value of TRUE, find the corresponding parent field and change its required value to TRUE. - -Note: As there is an accompanying Flyway script that updates the values directly in the database, you do not need to reload these metadata .tsv files via API, unless you make additional changes, e.g set some compound fields to be conditionally required. diff --git a/doc/release-notes/5.4.1-release-notes.md b/doc/release-notes/5.4.1-release-notes.md deleted file mode 100644 index 13433ac12d2..00000000000 --- a/doc/release-notes/5.4.1-release-notes.md +++ /dev/null @@ -1,46 +0,0 @@ -# Dataverse Software 5.4.1 - -This release provides a fix for a regression introduced in 5.4 and implements a few other small changes. Please use 5.4.1 for production deployments instead of 5.4. - -## Release Highlights - -### API Backwards Compatibility Maintained - -The syntax in the example in the [Basic File Access](https://guides.dataverse.org/en/5.4.1/api/dataaccess.html#basic-file-access) section of the Dataverse Software Guides will continue to work. - -## Complete List of Changes - -For the complete list of code changes in this release, see the [5.4.1 Milestone](https://github.com/IQSS/dataverse/milestone/95?closed=1) in Github. - -For help with upgrading, installing, or general questions please post to the [Dataverse Community Google Group](https://groups.google.com/forum/#!forum/dataverse-community) or email support@dataverse.org. - -## Installation - -If this is a new installation, please see our [Installation Guide](https://guides.dataverse.org/en/5.4.1/installation/). - -## Upgrade Instructions - -0\. These instructions assume that you've already successfully upgraded from Dataverse Software 4.x to Dataverse Software 5 following the instructions in the [Dataverse Software 5 Release Notes](https://github.com/IQSS/dataverse/releases/tag/v5.0). After upgrading from the 4.x series to 5.0, you should progress through the other 5.x releases before attempting the upgrade to 5.4.1. - -1\. Undeploy the previous version. - -- `$PAYARA/bin/asadmin list-applications` -- `$PAYARA/bin/asadmin undeploy dataverse<-version>` - -2\. Stop Payara and remove the generated directory - -- `service payara stop` -- `rm -rf $PAYARA/glassfish/domains/domain1/generated` - -3\. Start Payara - -- `service payara start` - -4\. Deploy this version. - -- `$PAYARA/bin/asadmin deploy dataverse-5.4.1.war` - -5\. Restart payara - -- `service payara stop` -- `service payara start` diff --git a/doc/release-notes/5.5-release-notes.md b/doc/release-notes/5.5-release-notes.md deleted file mode 100644 index 93383749578..00000000000 --- a/doc/release-notes/5.5-release-notes.md +++ /dev/null @@ -1,154 +0,0 @@ -# Dataverse Software 5.5 - -This release brings new features, enhancements, and bug fixes to the Dataverse Software. Thank you to all of the community members who contributed code, suggestions, bug reports, and other assistance across the project. - -## Release Highlights - -### Auxiliary Files Accessible Through the UI - -Auxiliary Files can now be downloaded from the web interface. Auxiliary files uploaded as type=DP appear under "Differentially Private Statistics" under file level download. The rest appear under "Other Auxiliary Files". - -Please note that the auxiliary files feature is experimental and is designed to support integration with tools from the [OpenDP Project](https://opendp.org). If the API endpoints are not needed they can be blocked. - -### Improved Workflow for Downloading Large Zip Files - -Users trying to download a zip file larger than the Dataverse installation's :ZipDownloadLimit will now receive messaging that the zip file is too large, and the user will be presented with alternate access options. Previously, the zip file would download and files above the :ZipDownloadLimit would be excluded and noted in a MANIFEST.TXT file. - -### Guidelines on Depositing Code - -The Software Metadata Working Group has created guidelines on depositing research code in a Dataverse installation. Learn more in the [Dataset Management section](https://guides.dataverse.org/en/latest/user/dataset-management.html#research-code) of the Dataverse Guides. - -### New Metrics API - -Users can retrieve new types of metrics and per-collection metrics. The new capabilities are [described](https://guides.dataverse.org/en/5.5/api/metrics.html) in the guides. A new version of the [Dataverse Metrics web app](https://github.com/IQSS/dataverse-metrics) adds interactive graphs to display these metrics. - -There is a backward incompatible change to the Metrics API whereby you must now send the header `Accept: application/json` if you want to continue receiving JSON output. As of this release, absence of this header results in CSV rather than JSON for time series metrics. For details, see [Return Formats](https://guides.dataverse.org/en/5.5/api/metrics.html#return-formats) in the Metrics API section. - -Because of this change from JSON to CSV, adjustments have been made to the software that makes use of the Metrics API: - -- Anyone running the existing Dataverse Metrics app will need to upgrade to 0.2.8 (or higher) or apply [a small patch](https://github.com/IQSS/dataverse-metrics/pull/60) to continue retrieving metrics from Dataverse instances upgrading to this release. -- Anyone who is using the Harvard Dataverse Repository custom homepage as a model might want to look at [changes](https://github.com/IQSS/dataverse.harvard.edu/pull/116) to get elements such as "browse by subject" working again. - -## Major Use Cases - -Newly-supported major use cases in this release include: - -- Users can now select and download auxiliary files through the UI. (Issue #7400, PR #7729) -- Users attempting to download zip files above the installation's size limit will receive better messaging and be directed to other download options. (Issue #7714, PR #7806) -- Superusers can now sort users on the Dashboard. (Issue #7814, PR #7815) -- Users can now access expanded and new metrics through a new API (Issue #7177, PR #7178) -- Dataverse collection administrators can now add a search facet on their collection pages for the Geospatial metadatablock's "Other" field, so that others can narrow searches in their collections using the values entered in that "Other" field (Issue #7399, PR #7813) -- Depositors can now receive guidance about depositing code into a Dataverse installation (PR #7717) - -## Notes for Dataverse Installation Administrators - -### Simple Search Fix for Solr Configuration - -The introduction in v4.17 of a schema_dv_mdb_copies.xml file as part of the Solr configuration accidentally removed the contents of most metadata fields from index used for simple searches in Dataverse (i.e. when one types a word without indicating which field to search in the normal search box). This was somewhat ameliorated/hidden by the fact that many common fields such as description were still included by other means. - -This release removes the schema_dv_mdb_copies.xml file and includes the updates needed in the schema.xml file. Installations with no custom metadata blocks can simply replace their current schema.xml file for Solr, restart Solr, and run a ['Reindex in Place' as described in the guides](https://guides.dataverse.org/en/latest/admin/solr-search-index.html#reindex-in-place). - -Installations using custom metadata blocks should manually copy the contents of their schema_dv_mdb_copies.xml file (excluding the enclosing `` element and only including the `` elements) into their schema.xml file, replacing the section between - -`` - -and - -``. - -In existing schema.xml files, this section currently includes only one line: - -``. - -In this release, that line has already been replaced with the default set of ``. -It doesn't matter whether schema_dv_mdb_copies.xml was originally created manually or via the recommended updateSchemaMDB.sh script and this fix will work with all prior versions of Dataverse from v4.17 on. If you make further changes to metadata blocks in your installation, you can repeat this process (i.e. run updateSchemaMDB.sh, copy the entries in schema_dv_mdb_copies.xml into the same section of schema.xml, restart solr, and reindex.) - -Once schema.xml is updated, Solr should be restarted and a ['Reindex in Place'](https://guides.dataverse.org/en/latest/admin/solr-search-index.html#reindex-in-place) will be required. (Future Dataverse Software versions will avoid this manual copy step.) - -### Geospatial Metadata Block Updated - -The Geospatial metadata block (geospatial.tsv) was updated. Dataverse collection administrators can now add a search facet on their collection pages for the metadata block's "Other" field, so that people searching in their collections can narrow searches using the values entered in that field. - -### Extended support for S3 Download Redirects ("Direct Downloads") - -If your installation uses S3 for storage and you have "direct downloads" enabled, please note that it will now cover the following download types that were not handled by redirects in the earlier versions: saved originals of tabular data files, cached RData frames, resized thumbnails for image files and other auxiliary files. In other words, all the forms of the file download API that take extra arguments, such as "format" or "imageThumb" - for example: - -`/api/access/datafile/12345?format=original` - -`/api/access/datafile/:persistentId?persistentId=doi:1234/ABCDE/FGHIJ&imageThumb=true` - -etc., that were previously excluded. - -Since browsers follow redirects automatically, this change should not in any way affect the web GUI users. However, some API users may experience problems, if they use it in a way that does not expect to receive a redirect response. For example, if a user has a script where they expect to download a saved original of an ingested tabular file with the following command: - -`curl https://yourhost.edu/api/access/datafile/12345?format=original > orig.dta` - -it will fail to save the file when it receives a 303 (redirect) response instead of 200. So they will need to add "-L" to the command line above, to instruct curl to follow redirects: - -`curl -L https://yourhost.edu/api/access/datafile/12345?format=original > orig.dta` - -Most of your API users have likely figured it out already, since you enabled S3 redirects for "straightforward" downloads in your installation. But we feel it was worth a heads up, just in case. - -### Authenticated User Deactivated Field Updated - -The "deactivated" field on the Authenticated User table has been updated to be a non-nullable field. When the field was added in version 5.3 it was set to 'false' in an update script. If for whatever reason that update failed in the 5.3 deploy you will need to re-run it before deploying 5.5. The update query you may need to run is: UPDATE authenticateduser SET deactivated = false WHERE deactivated IS NULL; - -### Metrics API Change - -See "New Metrics API" above for a backward incompatible change. - -## Notes for Tool Developers and Integrators - -### S3 Download Redirects - -See above note about download redirects. If your application integrates with the the Dataverse software using the APIs, you may need to change how redirects are handled in your tool or integration. - -### Metrics API Change - -See "New Metrics API" above for a backward incompatible change. - -## Complete List of Changes - -For the complete list of code changes in this release, see the [5.5 Milestone](https://github.com/IQSS/dataverse/milestone/96?closed=1) in Github. - -For help with upgrading, installing, or general questions please post to the [Dataverse Community Google Group](https://groups.google.com/forum/#!forum/dataverse-community) or email support@dataverse.org. - -## Installation - -If this is a new installation, please see our [Installation Guide](https://guides.dataverse.org/en/5.5/installation/). - -## Upgrade Instructions - -0\. These instructions assume that you've already successfully upgraded from Dataverse Software 4.x to Dataverse Software 5 following the instructions in the [Dataverse Software 5 Release Notes](https://github.com/IQSS/dataverse/releases/tag/v5.0). After upgrading from the 4.x series to 5.0, you should progress through the other 5.x releases before attempting the upgrade to 5.5. - -1\. Undeploy the previous version. - -- `$PAYARA/bin/asadmin list-applications` -- `$PAYARA/bin/asadmin undeploy dataverse<-version>` - -2\. Stop Payara and remove the generated directory - -- `service payara stop` -- `rm -rf $PAYARA/glassfish/domains/domain1/generated` - -3\. Start Payara - -- `service payara start` - -4\. Deploy this version. - -- `$PAYARA/bin/asadmin deploy dataverse-5.5.war` - -5\. Restart payara - -- `service payara stop` -- `service payara start` - -## Additional Release Steps - -1\. Follow the steps to update your Solr configuration, found in the "Notes for Dataverse Installation Administrators" section above. Note that there are different instructions for Dataverse installations running with custom metadata blocks and those without. - -2\. Update Geospatial Metadata Block (if used) - -- `wget https://github.com/IQSS/dataverse/releases/download/v5.5/geospatial.tsv` -- `curl http://localhost:8080/api/admin/datasetfield/load -X POST --data-binary @geospatial.tsv -H "Content-type: text/tab-separated-values"` diff --git a/doc/release-notes/5.6-release-notes.md b/doc/release-notes/5.6-release-notes.md deleted file mode 100644 index 813f93db278..00000000000 --- a/doc/release-notes/5.6-release-notes.md +++ /dev/null @@ -1,138 +0,0 @@ -# Dataverse Software 5.6 - -This release brings new features, enhancements, and bug fixes to the Dataverse Software. Thank you to all of the community members who contributed code, suggestions, bug reports, and other assistance across the project. - -## Release Highlights - -### Anonymized Access in Support of Double Blind Review - -Dataverse installations can select whether or not to allow users to create anonymized private URLs and can control which specific identifying fields are anonymized. If this is enabled, users can create private URLs that do not expose identifying information about dataset depositors, allowing for double blind reviews of datasets in the Dataverse installation. - -### Guestbook Responses API - -A new API to retrieve Guestbook responses has been added. This makes it easier to retrieve the records for large guestbooks and also makes it easier to integrate with external systems. - -### Dataset Semantic API (Experimental) - -Dataset metadata can be retrieved, set, and updated using a new, flatter JSON-LD format - following the format of an OAI-ORE export (RDA-conformant Bags), allowing for easier transfer of metadata to/from other systems (i.e. without needing to know Dataverse's metadata block and field storage architecture). This new API also allows for the update of terms metadata (#5899). - -This development was supported by the [Research Data Alliance](https://rd-alliance.org), DANS, and Sciences PO and follows the recommendations from the [Research Data Repository Interoperability Working Group](http://dx.doi.org/10.15497/RDA00025). - -### Dataset Migration API (Experimental) - -Datasets can now be imported following the format of an OAI-ORE export (RDA-conformant Bags), allowing for easier migration from one Dataverse installation to another, and migration from other systems. This experimental, superuser only, endpoint also allows keeping the existing persistent identifier (where the authority and shoulder match those for which the software is configured) and publication dates. - -This development was supported by DANS and the [Research Data Alliance](https://rd-alliance.org) and follows the recommendations from the [Research Data Repository Interoperability Working Group](http://dx.doi.org/10.15497/RDA00025). - -### Direct Upload API Now Available for Adding Multiple Files Metadata to the Dataset - -Users can now add metadata of multiple files to the dataset once the files exists in the s3 bucket using the direct upload API. This makes direct uploads more efficient and reduces server load by only updating the dataset once instead of once per file. For more information, see the [Direct DataFile Upload/Replace API section](https://guides.dataverse.org/en/5.6/developers/s3-direct-upload-api.html) of the Dataverse Software Guides. - -## Major Use Cases - -Newly-supported major use cases in this release include: - -- Users can create Private URLs that anonymize dataset metadata, allowing for double blind peer review. (Issue #1724, PR #7908) -- Users can download Guestbook records using a new API. (Issue #7767, PR #7931) -- Users can update terms metadata using the new semantic API. (Issue #5899, PR #7414) -- Users can retrieve, set, and update metadata using a new, flatter JSON-LD format. (Issue #6497, PR #7414) -- Administrators can use the Traces API to retrieve information about specific types of user activity (Issue #7952, PR #7953) - -## Notes for Dataverse Installation Administrators - -### New Database Constraint - -A new DB Constraint has been added in this release. Full instructions on how to identify whether or not your database needs any cleanup before the upgrade can be found in the [Dataverse software GitHub repository](https://github.com/IQSS/dataverse/blob/develop/scripts/issues/7451/PRE-RELEASE-INFO.txt). This information was also emailed out to Dataverse installation contacts. - -### Payara 5.2021.5 (or Higher) Required - -Some changes in this release require an upgrade to Payara 5.2021.5 or higher. (See the upgrade section). - -Instructions on how to update can be found in the [Payara documentation](https://docs.payara.fish/community/docs/5.2021.4/documentation/user-guides/upgrade-payara.html) We've included the necessary steps below, but we recommend that you review the Payara upgrade instructions as it could be helpful during any troubleshooting. - -Installations upgrading from a previous Payara version shouldn't encounter a logging configuration bug in Payara-5.2021.5, but if your server.log fills with repeated notes about logging configuration and WELD complaints about loading beans, see the paragraph on `logging.properties` in the [installation guide](https://guides.dataverse.org/en/5.6/installation/installation-main.html#running-the-dataverse-software-installer) - -### Enhancement to DDI Metadata Exports - -To increase support for internationalization and to improve compliance with CESSDA requirements, DDI exports now have a holdings element with a URI attribute whose value is the URL form of the dataset PID. - -## New JVM Options and DB Settings - -:AnonymizedFieldTypeNames can be used to enable creation of anonymized Private URLs and to specify which fields will be anonymized. - -## Notes for Tool Developers and Integrators - -### Semantic API - -The new Semantic API is especially helpful in data migrations and getting metadata into a Dataverse installation. Learn more in the [Developers Guide](https://guides.dataverse.org/en/5.6/developers/). - -## Complete List of Changes - -For the complete list of code changes in this release, see the [5.6 Milestone](https://github.com/IQSS/dataverse/milestone/97?closed=1) in Github. - -For help with upgrading, installing, or general questions please post to the [Dataverse Community Google Group](https://groups.google.com/forum/#!forum/dataverse-community) or email support@dataverse.org. - -## Installation - -If this is a new installation, please see our [Installation Guide](https://guides.dataverse.org/en/5.6/installation/). - -## Upgrade Instructions - -0\. These instructions assume that you've already successfully upgraded from Dataverse Software 4.x to Dataverse Software 5 following the instructions in the [Dataverse Software 5 Release Notes](https://github.com/IQSS/dataverse/releases/tag/v5.0). After upgrading from the 4.x series to 5.0, you should progress through the other 5.x releases before attempting the upgrade to 5.6. - -The steps below include a required upgrade to Payara 5.2021.5 or higher. (It is a simple matter of reusing your existing domain directory with the new distribution). But we also recommend that you review the Payara upgrade instructions as it could be helpful during any troubleshooting: [Payara documentation](https://docs.payara.fish/community/docs/5.2021.5/documentation/user-guides/upgrade-payara.html) - -If you are running Payara as a non-root user (and you should be!), **remember not to execute the commands below as root**. Use `sudo` to change to that user first. For example, `sudo -i -u dataverse` if `dataverse` is your dedicated application user. - -In the following commands we assume that Payara 5 is installed in `/usr/local/payara5`. If not, adjust as needed. - -`export PAYARA=/usr/local/payara5` - -(or `setenv PAYARA /usr/local/payara5` if you are using a `csh`-like shell) - -1\. Undeploy the previous version - -- `$PAYARA/bin/asadmin list-applications` -- `$PAYARA/bin/asadmin undeploy dataverse<-version>` - -2\. Stop Payara - -- `service payara stop` -- `rm -rf $PAYARA/glassfish/domains/domain1/generated` - -3\. Move the current Payara directory out of the way - -- `mv $PAYARA $PAYARA.MOVED` - -4\. Download the new Payara version (5.2021.5+), and unzip it in its place - -5\. Replace the brand new payara/glassfish/domains/domain1 with your old, preserved `domain1` - -6\. In domain.xml, add/confirm the existence of the following JVM option to/in the `` section: - -`[1.8.0u251|]-Xbootclasspath/a:${com.sun.aas.installRoot}/lib/grizzly-npn-api.jar` - -7\. Start Payara - -- `service payara start` - -8\. Deploy this version. - -- `$PAYARA/bin/asadmin deploy dataverse-5.6.war` - -9\. Restart payara - -- `service payara stop` -- `service payara start` - -10\. Run ReExportall to update JSON Exports - -## Additional Release Steps - -**If your installation relies on the database-side stored procedure for generating sequential numeric identifiers:** - -Note that you can skip this step if your installation uses the default-style, randomly-generated six alphanumeric character-long identifiers for your datasets! This is the case with most Dataverse installations. - -The underlying database framework has been modified in this release, to make it easier for installations to create custom procedures for generating identifier strings that suit their needs. Your current configuration will be automatically updated by the database upgrade (Flyway) script incorporated in the release. No manual configuration changes should be necessary. However, after the upgrade, we recommend that you confirm that your installation can still create new datasets, and that they are still assigned sequential numeric identifiers. In the unlikely chance that this is no longer working, please re-create the stored procedure following the steps described in the documentation for the `:IdentifierGenerationStyle` setting in the *Configuration* section of the Installation Guide for this release (v5.6). - -(Running the script supplied there will NOT overwrite the position on the sequence you are currently using!) diff --git a/doc/release-notes/5.7-release-notes.md b/doc/release-notes/5.7-release-notes.md deleted file mode 100644 index 041265410f4..00000000000 --- a/doc/release-notes/5.7-release-notes.md +++ /dev/null @@ -1,155 +0,0 @@ -# Dataverse Software 5.7 - -This release brings new features, enhancements, and bug fixes to the Dataverse Software. Thank you to all of the community members who contributed code, suggestions, bug reports, and other assistance across the project. - -## Release Highlights - -### Experimental Support for External Vocabulary Services - -Dataverse can now be configured to associate specific metadata fields with third-party vocabulary services to provide an easy way for users to select values from those vocabularies. The mapping involves use of external Javascripts. Two such scripts have been developed so far: one for vocabularies served via the [SKOSMOS protocol](https://skosmos.org) and one allowing people to be identified via their [ORCID](https://orcid.org). The guides contain info about the new [:CVocConf setting](https://guides.dataverse.org/en/5.7/installation/config.html#cvocconf) used for configuration and additional information about this functionality. Scripts, examples, and additional documentation are available at the [GDCC GitHub Repository](https://github.com/gdcc/dataverse-external-vocab-support). - -Please watch the [online presentation](https://youtu.be/mWMPxCwB2ys?t=1794), [read the document with requirements](https://docs.google.com/document/d/1txdcFuxskRx_tLsDQ7KKLFTMR_r9IBhorDu3V_r445w/edit?ts=5fdbca9a) and join the [Dataverse Working Group on Ontologies and Controlled Vocabularies](https://dataverse.org/community-calls) if you have some questions and want to contribute. - -This functionality was initially developed by [Data Archiving and Networked Services](http://dans.knaw.nl/en/) (DANS-KNAW), the Netherlands, and funded by [SSHOC](https://sshopencloud.eu), "Social Sciences and Humanities Open Cloud". SSHOC has received funding from the European Union’s Horizon 2020 project call H2020-INFRAEOSC-04-2018, grant agreement #823782. It was further improved by the Global Dataverse Community Consortium (GDCC) and extended with the support of semantic search. - -### Curation Status Labels - -A new :AllowedCurationLabels setting allows a sysadmins to define one or more sets of labels that can be applied to a draft Dataset version via the user interface or API to indicate the status of the dataset with respect to a defined curation process. - -Labels are completely customizable (alphanumeric or spaces, up to 32 characters, e.g. "Author contacted", "Privacy Review", "Awaiting paper publication"). Superusers can select a specific set of labels, or disable this functionality per collection. Anyone who can publish a draft dataset (e.g. curators) can set/change/remove labels (from the set specified for the collection containing the dataset) via the user interface or via an API. The API also would allow external tools to search for, read and set labels on Datasets, providing an integration mechanism. Labels are visible on the Dataset page and in Dataverse collection listings/search results. Internally, the labels have no effect, and at publication, any existing label will be removed. A reporting API call allows admins to get a list of datasets and their curation statuses. - -The Solr schema must be updated as part of installing the release of Dataverse containing this feature for it to work. - -## Major Use Cases - -Newly-supported major use cases in this release include: - -- Administrators will be able to set up integrations with external vocabulary services, allowing for autocomplete-assisted metadata entry, metadata standardization, and better integration with other systems (Issue #7711, PR #7946) -- Users viewing datasets in the root Dataverse collection will now see breadcrumbs that have have a link back to the root Dataverse collection (Issue #7527, PR #8078) -- Users will be able to more easily differentiate between datasets and files through new iconography (Issue #7991, PR #8021) -- Users retrieving large guestbooks over the API will experience fewer failures (Issue #8073, PR #8084) -- Dataverse collection administrators can specify which language will be used when entering metadata for new Datasets in a collection, based on a list of languages specified by the Dataverse installation administrator (Issue #7388, PR #7958) - - Users will see the language used for metadata entry indicated at the document or element level in metadata exports (Issue #7388, PR #7958) - - Administrators will now be able to specify the language(s) of controlled vocabulary entries, in addition to the installation's default language (Issue #6751, PR #7959) -- Administrators and curators can now receive notifications when a dataset is created (Issue #8069, PR #8070) -- Administrators with large files in their installation can disable the automatic checksum verification process at publish time (Issue #8043, PR #8074) - -## Notes for Dataverse Installation Administrators - -### Dataset Creation Notifications for Administrators - -A new :SendNotificationOnDatasetCreation setting has been added. When true, administrators and curators (those who can publish the dataset) will get a notification when a new dataset is created. This makes it easier to track activity in a Dataverse and, for example, allow admins to follow up when users do not publish a new dataset within some period of time. - -### Skip Checksum Validation at Publish Based on Size - -When a user requests to publish a dataset, the time taken to complete the publishing process varies based on the dataset/datafile size. - -With the additional settings of :DatasetChecksumValidationSizeLimit and :DataFileChecksumValidationSizeLimit, the checksum validation can be skipped while publishing. - -If the Dataverse administrator chooses to set these values, it's strongly recommended to have an external auditing system run periodically in order to monitor the integrity of the files in the Dataverse installation. - -### Guestbook Response API Update - -With this release the Retrieve Guestbook Responses for a Dataverse Collection API will no longer produce a file by default. You may specify an output file by adding a -o $YOURFILENAME to the curl command. - -### Dynamic JavaServer Faces Configuration Options - -This release includes a new way to easily change JSF settings via MicroProfile Config, especially useful during development. -See the [development guide on "Debugging"](https://guides.dataverse.org/en/5.7/developers/debugging.html) for more information. - -### Enhancements to DDI Metadata Exports - -Several changes have been made to the DDI exports to improve support for internationalization and to improve compliance with CESSDA requirements. These changes include: - -- Addition of xml:lang attributes specifying the dataset metadata language at the document level and for individual elements such as title and description -- Specification of controlled vocabulary terms in duplicate elements in multiple languages (in the installation default langauge and, if different, the dataset metadata language) - -While these changes are intended to improve harvesting and integration with external systems, they could break existing connections that make assumptions about the elements and attributes that have been changed. - -## New JVM Options and DB Settings - -- :SendNotificationOnDatasetCreation - A boolean setting that, if true will send an email and notification to additional users when a Dataset is created. Messages go to those, other than the dataset creator, who have the ability/permission necessary to publish the dataset. -- :DatasetChecksumValidationSizeLimit - disables the checksum validation while publishing for any dataset size greater than the limit. -- :DataFileChecksumValidationSizeLimit - Disables the checksum validation while publishing for any datafiles greater than the limit. -- :CVocConf - A JSON-structured setting that configures Dataverse to associate specific metadatablock fields with external vocabulary services and specific vocabularies/sub-vocabularies managed by that service. -- :MetadataLanguages - Sets which languages can be used when entering dataset metadata. -- :AllowedCurationLabels - A JSON Object containing lists of allowed labels (up to 32 characters, spaces allowed) that can be set, via API or UI by users with the permission to publish a dataset. The set of labels allowed for datasets can be selected by a superuser - via the Dataverse collection page (Edit/General Info) or set via API call. - -## Notes for Tool Developers and Integrators - -### Bags Now Support File Paths - -The original Bag generation code stored all dataset files directly under the /data directory. With the addition in Dataverse of a directory path for files and then a change to allow files with different paths to have the same name, archival Bags will now use the directory path from Dataverse to avoid name collisions within the /data directory. Prior to this update, Bags from Datasets with multiple files with the same name would have been created with only one of the files with that name (with warnings in the log, but still generating the Bag). - -## Complete List of Changes - -For the complete list of code changes in this release, see the [5.7 Milestone](https://github.com/IQSS/dataverse/milestone/98?closed=1) in Github. - -For help with upgrading, installing, or general questions please post to the [Dataverse Community Google Group](https://groups.google.com/forum/#!forum/dataverse-community) or email support@dataverse.org. - -## Installation - -If this is a new installation, please see our [Installation Guide](https://guides.dataverse.org/en/5.7/installation/). - -## Upgrade Instructions - -0\. These instructions assume that you've already successfully upgraded from Dataverse Software 4.x to Dataverse Software 5 following the instructions in the [Dataverse Software 5 Release Notes](https://github.com/IQSS/dataverse/releases/tag/v5.0). After upgrading from the 4.x series to 5.0, you should progress through the other 5.x releases before attempting the upgrade to 5.7. - -If you are running Payara as a non-root user (and you should be!), **remember not to execute the commands below as root**. Use `sudo` to change to that user first. For example, `sudo -i -u dataverse` if `dataverse` is your dedicated application user. - -In the following commands we assume that Payara 5 is installed in `/usr/local/payara5`. If not, adjust as needed. - -`export PAYARA=/usr/local/payara5` - -(or `setenv PAYARA /usr/local/payara5` if you are using a `csh`-like shell) - -1\. Undeploy the previous version. - -- `$PAYARA/bin/asadmin list-applications` -- `$PAYARA/bin/asadmin undeploy dataverse<-version>` - -2\. Stop Payara and remove the generated directory - -- `service payara stop` -- `rm -rf $PAYARA/glassfish/domains/domain1/generated` - -3\. Start Payara - -- `service payara start` - -4\. Deploy this version. - -- `$PAYARA/bin/asadmin deploy dataverse-5.7.war` - -5\. Restart payara - -- `service payara stop` -- `service payara start` - -## Additional Release Steps - -1\. Replace Solr schema.xml to allow Curation Labels to be used. See specific instructions below for those installations with custom metadata blocks (1a) and those without (1b). - -1a\. - -For installations with Custom Metadata Blocks: - --stop solr instance (usually service solr stop, depending on solr installation/OS, see the [Installation Guide](https://guides.dataverse.org/en/5.7/installation/prerequisites.html#solr-init-script) - -- add the following line to your schema.xml: - - `` - -- restart solr instance (usually service solr start, depending on solr/OS) - -1b\. - -For installations without Custom Metadata Blocks: - --stop solr instance (usually service solr stop, depending on solr installation/OS, see the [Installation Guide](https://guides.dataverse.org/en/5.7/installation/prerequisites.html#solr-init-script) - --replace schema.xml - -cp /tmp/dvinstall/schema.xml /usr/local/solr/solr-8.8.1/server/solr/collection1/conf - --start solr instance (usually service solr start, depending on solr/OS) \ No newline at end of file diff --git a/doc/release-notes/5.8-release-notes.md b/doc/release-notes/5.8-release-notes.md deleted file mode 100644 index b452739e161..00000000000 --- a/doc/release-notes/5.8-release-notes.md +++ /dev/null @@ -1,112 +0,0 @@ -# Dataverse Software 5.8 - -This release brings new features, enhancements, and bug fixes to the Dataverse Software. Thank you to all of the community members who contributed code, suggestions, bug reports, and other assistance across the project. - -## Release Highlights - -### Support for Data Embargoes - -The Dataverse Software now supports file-level embargoes. The ability to set embargoes, up to a maximum duration (in months), can be configured by a Dataverse installation administrator. For more information, see the [Embargoes section](https://guides.dataverse.org/en/5.8/user/dataset-management.html#embargoes) of the Dataverse Software Guides. - -- Users can configure a specific embargo, defined by an end date and a short reason, on a set of selected files or an individual file, by selecting the 'Embargo' menu item and entering information in a popup dialog. Embargoes can only be set, changed, or removed before a file has been published. After publication, only Dataverse installation administrators can make changes, using an API. - -- While embargoed, files cannot be previewed or downloaded (as if restricted, with no option to allow access requests). After the embargo expires, files become accessible. If the files were also restricted, they remain inaccessible and functionality is the same as for any restricted file. - -- By default, the citation date reported for the dataset and the datafiles in version 1.0 reflects the longest embargo period on any file in version 1.0, which is consistent with recommended practice from DataCite. Administrators can still specify an alternate date field to be used in the citation date via the [Set Citation Date Field Type for a Dataset API Call](https://guides.dataverse.org/en/5.8/api/native-api.html#set-citation-date-field-type-for-a-dataset). - -The work to add this functionality was initiated by Data Archiving and Networked Services (DANS-KNAW), the Netherlands. It was further developed by the Global Dataverse Community Consortium (GDCC) in cooperation with and with funding from DANS. - -## Major Use Cases and Infrastructure Enhancements - -Newly-supported major use cases in this release include: - -- Users can set file-level embargoes. (Issue #7743, #4052, #343, PR #8020) -- Improved accessibility of form labels on the advanced search page (Issue #8169, PR #8170) - -## Notes for Dataverse Installation Administrators - -### Mitigate Solr Schema Management Problems - -With [Release 5.5](https://github.com/IQSS/dataverse/releases/tag/v5.5), the `` definitions had been reincluded into `schema.xml` to fix searching for datasets. - -This release includes a final update to `schema.xml` and a new script `update-fields.sh` to manage your custom metadata fields, and to provide opportunities for other future improvements. The broken script `updateSchemaMDB.sh` has been removed. - -You will need to replace your schema.xml with the one provided in order to make sure that the new script can function. If you do not use any custom metadata blocks in your installation, this is the only change to be made. If you do use custom metadata blocks you will need to take a few extra steps, enumerated in the step-by-step instructions below. - -## New JVM Options and DB Settings - -- :MaxEmbargoDurationInMonths controls whether embargoes are allowed in a Dataverse instance and can limit the maximum duration users are allowed to specify. A value of 0 months or non-existent setting indicates embargoes are not supported. A value of -1 allows embargoes of any length. - -## Complete List of Changes - -For the complete list of code changes in this release, see the [5.8 Milestone](https://github.com/IQSS/dataverse/milestone/99?closed=1) in Github. - -For help with upgrading, installing, or general questions please post to the [Dataverse Community Google Group](https://groups.google.com/forum/#!forum/dataverse-community) or email support@dataverse.org. - -## Installation - -If this is a new installation, please see our [Installation Guide](https://guides.dataverse.org/en/5.8/installation/). Please also contact us to get added to the [Dataverse Project Map](https://guides.dataverse.org/en/5.8/installation/config.html#putting-your-dataverse-installation-on-the-map-at-dataverse-org) if you have not done so already. - -## Upgrade Instructions - -0\. These instructions assume that you've already successfully upgraded from Dataverse Software 4.x to Dataverse Software 5 following the instructions in the [Dataverse Software 5 Release Notes](https://github.com/IQSS/dataverse/releases/tag/v5.0). After upgrading from the 4.x series to 5.0, you should progress through the other 5.x releases before attempting the upgrade to 5.8. - -If you are running Payara as a non-root user (and you should be!), **remember not to execute the commands below as root**. Use `sudo` to change to that user first. For example, `sudo -i -u dataverse` if `dataverse` is your dedicated application user. - -In the following commands we assume that Payara 5 is installed in `/usr/local/payara5`. If not, adjust as needed. - -`export PAYARA=/usr/local/payara5` - -(or `setenv PAYARA /usr/local/payara5` if you are using a `csh`-like shell) - -1\. Undeploy the previous version. - -- `$PAYARA/bin/asadmin list-applications` -- `$PAYARA/bin/asadmin undeploy dataverse<-version>` - -2\. Stop Payara and remove the generated directory - -- `service payara stop` -- `rm -rf $PAYARA/glassfish/domains/domain1/generated` - -3\. Start Payara - -- `service payara start` - -4\. Deploy this version. - -- `$PAYARA/bin/asadmin deploy dataverse-5.8.war` - -5\. Restart payara - -- `service payara stop` -- `service payara start` - -6\. Update Solr schema.xml. - -`/usr/local/solr/solr-8.8.1/server/solr/collection1/conf` is used in the examples below as the location of your Solr schema. Please adapt it to the correct location, if different in your installation. Use `find / -name schema.xml` if in doubt. - -6a\. Replace `schema.xml` with the base version included in this release. - -``` - wget https://github.com/IQSS/dataverse/releases/download/v5.8/schema.xml - cp schema.xml /usr/local/solr/solr-8.8.1/server/solr/collection1/conf -``` - -For installations that are not using any Custom Metadata Blocks, **you can skip the next step**. - -6b\. For installations with Custom Metadata Blocks - -Use the script provided in the release to add the custom fields to the base `schema.xml` installed in the previous step. - -``` - wget https://github.com/IQSS/dataverse/releases/download/v5.8/update-fields.sh - chmod +x update-fields.sh - curl "http://localhost:8080/api/admin/index/solr/schema" | ./update-fields.sh /usr/local/solr/solr-8.8.1/server/solr/collection1/conf/schema.xml -``` - -(Note that the curl command above calls the admin api on `localhost` to obtain the list of the custom fields. In the unlikely case that you are running the main Dataverse Application and Solr on different servers, generate the `schema.xml` on the application node, then copy it onto the Solr server) - -7\. Restart Solr - -Usually `service solr stop; service solr start`, but may be different on your system. See the [Installation Guide](https://guides.dataverse.org/en/5.8/installation/prerequisites.html#solr-init-script) for more details. diff --git a/doc/release-notes/5.9-release-notes.md b/doc/release-notes/5.9-release-notes.md deleted file mode 100644 index 06005396908..00000000000 --- a/doc/release-notes/5.9-release-notes.md +++ /dev/null @@ -1,171 +0,0 @@ -# Dataverse Software 5.9 - -This release brings new features, enhancements, and bug fixes to the Dataverse Software. Thank you to all of the community members who contributed code, suggestions, bug reports, and other assistance across the project. - -## Release Highlights - -### Dataverse Collection Page Optimizations - -The Dataverse Collection page, which also serves as the search page and the homepage in most Dataverse installations, has been optimized, with a specific focus on reducing the number of queries for each page load. These optimizations will be more noticable on Dataverse installations with higher traffic. - -### Support for HTTP "Range" Header for Partial File Downloads - -Dataverse now supports the HTTP "Range" header, which allows users to download parts of a file. Here are some examples: - -- `bytes=0-9` gets the first 10 bytes. -- `bytes=10-19` gets 10 bytes from the middle. -- `bytes=-10` gets the last 10 bytes. -- `bytes=9-` gets all bytes except the first 10. - -Only a single range is supported. For more information, see the [Data Access API](https://guides.dataverse.org/en/5.9/api/dataaccess.html) section of the API Guide. - -### Support for Optional External Metadata Validation Scripts - -The Dataverse software now allows an installation administrator to provide custom scripts for additional metadata validation when datasets are being published and/or when Dataverse collections are being published or modified. The Harvard Dataverse Repository has been using this mechanism to combat content that violates our Terms of Use, specifically spam content. All the validation or verification logic is defined in these external scripts, thus making it possible for an installation to add checks custom-tailored to their needs. - -Please note that only the metadata are subject to these validation checks. This does not check the content of any uploaded files. - -For more information, see the [Database Settings](https://guides.dataverse.org/en/5.9/installation/config.html) section of the Guide. The new settings are listed below, in the "New JVM Options and DB Settings" section of these release notes. - -### Displaying Author's Identifier as Link - -In the dataset page's metadata tab the author's identifier is now displayed as a clickable link, which points to the profile page in the external service (ORCID, VIAF etc.) in cases where the identifier scheme provides a resolvable landing page. If the identifier does not match the expected scheme, a link is not shown. - -### Auxiliary File API Enhancements - -This release includes updates to the Auxiliary File API. These updates include: - -- Auxiliary files can now also be associated with non-tabular files -- Auxiliary files can now be deleted -- Duplicate Auxiliary files can no longer be created -- A new API has been added to list Auxiliary files by their origin -- Some auxiliary were being saved with the wrong content type (MIME type) but now the user can supply the content type on upload, overriding the type that would otherwise be assigned -- Improved error reporting -- A bugfix involving checksums for Auxiliary files - -Please note that the Auxiliary files feature is experimental and is designed to support integration with tools from the [OpenDP Project](https://opendp.org). If the API endpoints are not needed they can be blocked. - -## Major Use Cases and Infrastructure Enhancements - -Newly-supported major use cases in this release include: - -- The Dataverse collection page has been optimized, resulting in quicker load times on one of the most common pages in the application (Issue #7804, PR #8143) -- Users will now be able to specify a certain byte range in their downloads via API, allowing for downloads of file parts. (Issue #6397, PR #8087) -- A Dataverse installation administrator can now set up metadata validation for datasets and Dataverse collections, allowing for publish-time and create-time checks for all content. (Issue #8155, PR #8245) -- Users will be provided with clickable links to authors' ORCIDs and other IDs in the dataset metadata (Issue #7978, PR #7979) -- Users will now be able to associate Auxiliary files with non-tabular files (Issue #8235, PR #8237) -- Users will no longer be able to create duplicate Auxiliary files (Issue #8235, PR #8237) -- Users will be able to delete Auxiliary files (Issue #8235, PR #8237) -- Users can retrieve a list of Auxiliary files based on their origin (Issue #8235, PR #8237) -- Users will be able to supply the content type of Auxiliary files on upload (Issue #8241, PR #8282) -- The indexing process has been updated so that datasets with fewer files and indexed first, resulting in fewer failures and making it easier to identify problematically-large datasets. (Issue #8097, PR #8152) -- Users will no longer be able to create metadata records with problematic special characters, which would later require Dataverse installation administrator intervention and a database change (Issue #8018, PR #8242) -- The Dataverse software will now appropriately recognize files with the .geojson extension as GeoJSON files rather than "unknown" (Issue #8261, PR #8262) -- A Dataverse installation administrator can now retrieve more information about role deletion from the ActionLogRecord (Issue #2912, PR #8211) -- Users will be able to use a new role to allow a user to respond to file download requests without also giving them the power to manage the dataset (Issue #8109, PR #8174) -- Users will no longer be forced to update their passwords when moving from Dataverse 3.x to Dataverse 4.x (PR #7916) -- Improved accessibility of buttons on the Dataset and File pages (Issue #8247, PR #8257) - -## Notes for Dataverse Installation Administrators - -### Indexing Performance on Datasets with Large Numbers of Files - -We discovered that whenever a full reindexing needs to be performed, datasets with large numbers of files take an exceptionally long time to index. For example, in the Harvard Dataverse Repository, it takes several hours for a dataset that has 25,000 files. In situations where the Solr index needs to be erased and rebuilt from scratch (such as a Solr version upgrade, or a corrupt index, etc.) this can significantly delay the repopulation of the search catalog. - -We are still investigating the reasons behind this performance issue. For now, even though some improvements have been made, a dataset with thousands of files is still going to take a long time to index. In this release, we've made a simple change to the reindexing process, to index any such datasets at the very end of the batch, after all the datasets with fewer files have been reindexed. This does not improve the overall reindexing time, but will repopulate the bulk of the search index much faster for the users of the installation. - -### Custom Analytics Code Changes - -You should update your custom analytics code to capture a bug fix related to tracking within the dataset files table. This release restores that tracking. - -For more information, see the documentation and sample analytics code snippet provided in [Installation Guide](http://guides.dataverse.org/en/5.9/installation/config.html#web-analytics-code). This update can be used on any version 5.4+. - -### New ManageFilePermissions Permission - -Dataverse can now support a use case in which a Admin or Curator would like to delegate the ability to grant access to restricted files to other users. This can be implemented by creating a custom role (e.g. DownloadApprover) that has the new ManageFilePermissions permission. This release introduces the new permission, and a Flyway script adjusts the existing Admin and Curator roles so they continue to have the ability to grant file download requrests. - -### Thumbnail Defaults - -New *default* values have been added for the JVM settings `dataverse.dataAccess.thumbnail.image.limit` and `dataverse.dataAccess.thumbnail.pdf.limit`, of 3MB and 1MB respectively. This means that, *unless specified otherwise* by the JVM settings already in your domain configuration, the application will skip attempting to generate thumbnails for image files and PDFs that are above these size limits. -In previous versions, if these limits were not explicitly set, the application would try to create thumbnails for files of unlimited size. Which would occasionally cause problems with very large images. - -## New JVM Options and DB Settings - -The following DB settings allow configuration of the external metadata validator: - -- :DataverseMetadataValidatorScript -- :DataverseMetadataPublishValidationFailureMsg -- :DataverseMetadataUpdateValidationFailureMsg -- :DatasetMetadataValidatorScript -- :DatasetMetadataValidationFailureMsg -- :ExternalValidationAdminOverride - -See the [Database Settings](https://guides.dataverse.org/en/5.9/installation/config.html) section of the Guides for more information. - -## Notes for Developers and Integrators - -Two sections of the Developer Guide have been updated: - -- Instructions on how to sync a PR in progress with develop have been added in the version control section -- Guidance on avoiding ineffeciencies in JSF render logic has been added to the "Tips" section - -## Complete List of Changes - -For the complete list of code changes in this release, see the [5.9 Milestone](https://github.com/IQSS/dataverse/milestone/100?closed=1) in Github. - -For help with upgrading, installing, or general questions please post to the [Dataverse Community Google Group](https://groups.google.com/forum/#!forum/dataverse-community) or email support@dataverse.org. - -## Installation - -If this is a new installation, please see our [Installation Guide](https://guides.dataverse.org/en/5.9/installation/). Please also contact us to get added to the [Dataverse Project Map](https://guides.dataverse.org/en/5.9/installation/config.html#putting-your-dataverse-installation-on-the-map-at-dataverse-org) if you have not done so already. - -## Upgrade Instructions - -0\. These instructions assume that you've already successfully upgraded from Dataverse Software 4.x to Dataverse Software 5 following the instructions in the [Dataverse Software 5 Release Notes](https://github.com/IQSS/dataverse/releases/tag/v5.0). After upgrading from the 4.x series to 5.0, you should progress through the other 5.x releases before attempting the upgrade to 5.9. - -If you are running Payara as a non-root user (and you should be!), **remember not to execute the commands below as root**. Use `sudo` to change to that user first. For example, `sudo -i -u dataverse` if `dataverse` is your dedicated application user. - -In the following commands we assume that Payara 5 is installed in `/usr/local/payara5`. If not, adjust as needed. - -`export PAYARA=/usr/local/payara5` - -(or `setenv PAYARA /usr/local/payara5` if you are using a `csh`-like shell) - -1\. Undeploy the previous version. - -- `$PAYARA/bin/asadmin list-applications` -- `$PAYARA/bin/asadmin undeploy dataverse<-version>` - -2\. Stop Payara and remove the generated directory - -- `service payara stop` -- `rm -rf $PAYARA/glassfish/domains/domain1/generated` - -3\. Start Payara - -- `service payara start` - -4\. Deploy this version. - -- `$PAYARA/bin/asadmin deploy dataverse-5.9.war` - -5\. Restart payara - -- `service payara stop` -- `service payara start` - -6\. Run ReExportall to update JSON Exports - -Following the directions in the [Admin Guide](http://guides.dataverse.org/en/5.9/admin/metadataexport.html?highlight=export#batch-exports-through-the-api) - -## Additional Release Steps - -(for installations collecting web analytics) - -1\. Update custom analytics code per the [Installation Guide](http://guides.dataverse.org/en/5.9/installation/config.html#web-analytics-code). - -(for installations with GeoJSON files) - -1\. Redetect GeoJSON files to update the type from "Unknown" to GeoJSON, following the directions in the [API Guide](https://guides.dataverse.org/en/5.9/api/native-api.html#redetect-file-type) - -2\. Kick off full reindex following the directions in the [Admin Guide](http://guides.dataverse.org/en/5.9/admin/solr-search-index.html) diff --git a/doc/release-notes/README.md b/doc/release-notes/README.md index 00f80926ca2..f27a6118524 100644 --- a/doc/release-notes/README.md +++ b/doc/release-notes/README.md @@ -1,12 +1,14 @@ # Dataverse Release Notes -doc/sphinx-guides/source/developers/making-releases.rst documents the official process for making release notes but as indicated there, we are experimenting with a process with the following goals: +doc/sphinx-guides/source/developers/making-releases.rst documents the official process for making release notes. We have the following goals: - As a developer, I want to express in my pull request when an addition to the release notes will be necessary. -- As a developer, I want to be aware of changes that should be made to my dev environment after a pull request has been merged. These could be Solr schema changes or curl commands to reload metadata blocks, for example. +- As a developer, I want to be aware of changes that should be made to my development environment after a pull request has been merged (examples are Solr schema changes or curl commands to reload metadata blocks) +- As a developer or Dataverse administrator, I want a single source of truth for release notes. ## release-notes Directory Process - Create a Markdown file named after your branch (assuming your branch starts with an issue number as requested in doc/sphinx-guides/source/developers/version-control.rst) such as "5053-apis-custom-homepage.md". - In the file you created, give instructions for non-SQL upgrade steps that must be taken to run the branch in your pull request. Examples include Solr schema updates or reloading metadata blocks. (SQL updates are handled separately as indicated in doc/sphinx-guides/source/developers/sql-upgrade-scripts.rst.) -- At release time, gather all the files into final release notes and make a `git rm` commit to delete them to prevent clutter. +- At release time, gather all the files into final release notes and make a `git rm` commit (or equivalent process in Github Desktop) to delete them to prevent clutter. +- Copy the markdown from the final release notes into the releases page on Github and delete the .md file from this directory (to ensure that there is only a single copy of the notes) diff --git a/doc/sphinx-guides/source/developers/making-releases.rst b/doc/sphinx-guides/source/developers/making-releases.rst index cbd88b1a357..12fcf0dd0f4 100755 --- a/doc/sphinx-guides/source/developers/making-releases.rst +++ b/doc/sphinx-guides/source/developers/making-releases.rst @@ -58,7 +58,7 @@ At or near release time: - The "tag version" and "title" should be the number of the milestone with a "v" in front (i.e. v4.16). - Copy in the content from the .md file - For the description, follow post-4.16 examples at https://github.com/IQSS/dataverse/releases - +- Delete the .md file in the source code. This is to ensure that only one copy of the notes lives on in the future, as a single source of truth. Make Artifacts Available for Download -------------------------------------