Merge pull request #40 from IQSS/develop

update

.github/PULL_REQUEST_TEMPLATE.md

@@ -0,0 +1,15 @@
+**What this PR does / why we need it**:
+
+**Which issue(s) this PR closes**:
+
+Closes #
+
+**Special notes for your reviewer**:
+
+**Suggestions on how to test this**:
+
+**Does this PR introduce a user interface change?**:
+
+**Is there a release notes update needed for this change?**:
+
+**Additional documentation**:

conf/solr/7.3.1/updateSchemaMDB.sh

@@ -1,4 +1,4 @@
-#!/bin/sh
+#!/bin/bash
 set -euo pipefail
 
 # This script updates the <field> and <copyField> schema configuration necessary to properly
@@ -61,6 +61,12 @@ echo "Retrieve schema data from ${DATAVERSE_URL}/api/admin/index/solr/schema"
 TMPFILE=`mktemp`
 curl -f -sS "${DATAVERSE_URL}/api/admin/index/solr/schema${UNBLOCK_KEY}" > $TMPFILE
 
+### Fail gracefull if Dataverse is not ready yet.
+if [[ "`wc -l ${TMPFILE}`" < "3" ]]; then
+  echo "Dataverse responded with empty file. When running on K8s: did you bootstrap yet?"
+  exit 123
+fi
+
 ### Processing
 echo "Writing ${TARGET}/schema_dv_mdb_fields.xml"
 echo "<fields>" > ${TARGET}/schema_dv_mdb_fields.xml
@@ -76,4 +82,4 @@ rm ${TMPFILE}*
 
 ### Reloading
 echo "Triggering Solr RELOAD at ${SOLR_URL}/solr/admin/cores?action=RELOAD&core=collection1"
-curl -f -sS "${SOLR_URL}/solr/admin/cores?action=RELOAD&core=collection1"
+curl -f -sS "${SOLR_URL}/solr/admin/cores?action=RELOAD&core=collection1"

doc/mergeParty/readme.md

@@ -1,5 +1,5 @@
-# Merge Party Readme
-Welcome to the merge party! This document is intended to give a short overview of why we need this party, when was changed and how to change it. There's much work to do, so we'll keep it short. Hopefully.
+# Merge Party
+Welcome to the merge party! This document is intended to give a short overview of why we need this party, when was it changed and how to change it. There's much work to do, so we'll keep it short, hopefully.
 
 ## What Just Happened
 In order to allow users to log into Dataverse using credentials from other systems (e.g. institutional Shibboleth server), we had to refactor out the internal user management sub-system (formerly known as "DataverseUser") and introduce a new user system. The existing system was taken out of Dataverse but kept in the .war file, as we also need to support standalone instances.
@@ -16,7 +16,7 @@ From a merge standpoint, this means that code that previously referenced `Datave
 
 Most of these changes have been done by Michael/Phil - otherwise, the `auth` branch would not compile.
 
-Since the guest user does not live in the database, it does not have an id. Moreover, JPA classes cannot link directly to it\*. But have no fear - all users (and, really, all `RoleAssignee`s, which are users or groups) have an identifier. When you need to reference a user (and later, a group) just use the identifier (it's of type `String`). When needing to convert an identifier to a user, call `RoleAssigneeServiceBean.getRoleAssignee( identifier )` in the general case, or `AuthenticationServiceBean.getAuthenticatedUser(identifier)` if you're certain the identifier is of an authenticated user.
+The guest user does not live in the database so it does not have an id. Moreover, JPA classes cannot link directly to it\*. But have no fear - all users (and, really, all `RoleAssignee`s, which are users or groups) have an identifier. When you need to reference a user (and later, a group) just use the identifier (it's of type `String`). When needing to convert an identifier to a user, call `RoleAssigneeServiceBean.getRoleAssignee( identifier )` in the general case, or `AuthenticationServiceBean.getAuthenticatedUser(identifier)` if you're certain the identifier is of an authenticated user.
 
 
 \* We have debated this for a while, since we could have created a dummy record, like we've done so far. We went with this solution, as it is cleaner, can't be messed up by SQL scripts, and will make even more sense once groups arrive.
@@ -84,4 +84,4 @@ A new script that sets up the users and the dataverses, sets the system up for b
 
 ## Undoing the undoing the merge
 When merging back to master, we need to undo commit 8ae3e6a482b87b52a1745bb06f340875803d2c5b (a.k.a 8ae3e6a), which is the commit that undid the erroneous merge.
-More at http://www.christianengvall.se/undo-pushed-merge-git/
+More at http://www.christianengvall.se/undo-pushed-merge-git/

doc/release-notes/3937-new-installer-script.md

@@ -0,0 +1,12 @@
+## New Installer Script 
+
+We are introducing a new installer script, written in Python. It is
+intended to eventually replace the old installer (written in
+Perl). But for now it is being offered as an (experimental)
+alternative.
+
+See README_python.txt in scripts/installer and/or in the installer
+bundle for more information.
+
+
+

doc/release-notes/4.18-release-notes.md

@@ -0,0 +1,122 @@
+# Dataverse 4.18
+
+**Note: There is an issue in 4.18 with the display of validation messages on the dataset page (#6380) and we recommend using 4.18.1 for any production environments.**
+
+This release brings new features, enhancements, and bug fixes to Dataverse. Thank you to all of the community members who contributed code, suggestions, bug reports, and other assistance across the project.
+
+## Release Highlights
+
+### File Page Previews and Previewers
+
+File-level External Tools can now be configured to display in a "Preview Mode" designed for embedding within the file landing page.
+
+While not technically part of this release, previewers have been made available for several common file types. The previewers support for spreadsheet, image, text, document, audio, video, html files and more. These previewers can be found in the <a href="https://github.com/QualitativeDataRepository/dataverse-previewers">Qualitative Data Repository Github Repository</a>. The spreadsheet viewer was contributed by the [Dataverse SSHOC][] project.
+
+[Dataverse SSHOC]: https://www.sshopencloud.eu/news/developing-sshoc-dataverse
+
+### Microsoft Login
+
+Users can now create Dataverse accounts and login using self-provisioned Microsoft accounts such as live.com and outlook.com. Users can also use Microsoft accounts managed by their institutions. This new feature not only makes it easier to log in to Dataverse but will also streamline the interaction between any external tools that utilize Azure services that require login.
+
+### Add Data and Host Dataverse
+
+More workflows to add data have been added across the UI, including a new button on the My Data tab of the Account page, as well as a link in the Dataverse navbar, which will display on every page. This will provider users much easier access to start depositing data. By default, the Host Dataverse will be the installation root dataverse for these new Add Data workflows, but there is now a dropdown component allowing creators to select a dataverse you have proper permissions to create a new dataverse or dataset in.
+
+### Primefaces 7
+
+Primefaces, the open source UI framework upon which the Dataverse front end is built, has been updated to the most recent version. This provides security updates and bug fixes and will also allow Dataverse developers to take advantage of new features and enhancements.
+
+### Integration Test Pipeline and Test Health Reporting
+
+As part of the Dataverse Community's ongoing efforts to provide more robust automated testing infrastructure, and in support of the project's desire to have the develop branch constantly in a "release ready" state, API-based integration tests are now run every time a branch is merged to develop. The status of the last test run is available as a badge at the bottom of the README.md file that serves as the homepage of Dataverse Github Repository.
+
+### Make Data Count Metrics Updates
+
+A new configuration option has been added that allows Make Data Count metrics to be collected, but not reflected in the front end. This option was designed to allow installations to collect and verify metrics for a period before turning on the display to users.
+
+### Search API Enhancements
+
+The Dataverse Search API will now display unpublished content when an API token is passed (and appropriate permissions exist).
+
+### Additional Dataset Author Identifiers
+
+The following dataset author identifiers are now supported:
+
+- DAI: https://en.wikipedia.org/wiki/Digital_Author_Identifier
+- ResearcherID: http://researcherid.com
+- ScopusID: https://www.scopus.com
+
+## Major Use Cases
+
+Newly-supported use cases in this release include:
+
+- Users can view previews of several common file types, eliminating the need to download or explore a file just to get a quick look.
+- Users can log in using self-provisioned Microsoft accounts and also can log in using Microsoft accounts managed by an organization.
+- Dataverse administrators can now revoke and regenerate API tokens with an API call.
+- Users will receive notifications when their ingests complete, and will be informed if the ingest was a success or failure.
+- Dataverse developers will receive feedback about the health of the develop branch after their pull request was merged.
+- Dataverse tool developers will be able to query the Dataverse API for unpublished data as well as published data.
+- Dataverse administrators will be able to collect Make Data Count metrics without turning on the display for users.
+- Users with a DAI, ResearcherID, or ScopusID and use these author identifiers in their datasets.
+
+## Notes for Dataverse Installation Administrators
+
+### API Token Management
+
+- You can now delete a user's API token, recreate a user's API token, and find a token's expiration date. See the <a href="http://guides.dataverse.org/en/4.18/api/native-api.html">Native API guide</a> for more information.
+
+### New JVM Options
+
+[:mdcbaseurlstring](http://guides.dataverse.org/en/4.18/installation/config.html#mdcbaseurlstring) allows dataverse administrators to use a test base URL for Make Data Count.
+
+### New Database Settings
+
+[:DisplayMDCMetrics](http://guides.dataverse.org/en/4.18/installation/config.html#DisplayMDCMetrics) can be set to false to disable display of MDC metrics.
+
+## Notes for Tool Developers and Integrators
+
+### Preview Mode
+
+Tool Developers can now add the `hasPreviewMode` parameter to their file level external tools. This setting provides an embedded, simplified view of the tool on the file pages for any installation that installs the tool. See <a href="http://guides.dataverse.org/en/4.18/api/external-tools.html">Building External Tools</a> for more information.
+
+### API Token Management
+
+If your tool writes content back to Dataverse, you can now take advantage of administrative endpoints that delete and re-create API tokens. You can also use an endpoint that provides the expiration date of a specific API token. See the <a href="http://guides.dataverse.org/en/4.18/api/native-api.html">Native API guide</a> for more information.
+
+### View Unpublished Data Using Search API
+
+If you pass a token, the search API output will include unpublished content.
+
+## Complete List of Changes
+
+For the complete list of code changes in this release, see the <a href="https://github.com/IQSS/dataverse/milestone/85?closed=1">4.18 milestone</a> in Github.
+
+For help with upgrading, installing, or general questions please post to the <a href="https://groups.google.com/forum/#!forum/dataverse-community">Dataverse Google Group</a> or email [email protected].
+
+## Installation
+
+If this is a new installation, please see our <a href="http://guides.dataverse.org/en/4.18/installation/">Installation Guide</a>.
+
+## Upgrade
+
+1. Undeploy the previous version.
+
+- &lt;glassfish install path&gt;/glassfish4/bin/asadmin list-applications
+- &lt;glassfish install path&gt;/glassfish4/bin/asadmin undeploy dataverse
+
+2. Stop glassfish and remove the generated directory, start.
+
+- service glassfish stop
+- remove the generated directory: rm -rf &lt;glassfish install path&gt;glassfish4/glassfish/domains/domain1/generated
+- service glassfish start
+
+3. Deploy this version.
+
+- &lt;glassfish install path&gt;/glassfish4/bin/asadmin deploy &lt;path&gt;dataverse-4.18.war
+
+4. Restart glassfish.
+
+5. Update Citation Metadata Block
+
+- `wget https://github.com/IQSS/dataverse/releases/download/v4.18/citation.tsv`
+- `curl http://localhost:8080/api/admin/datasetfield/load -X POST --data-binary @citation.tsv -H "Content-type: text/tab-separated-values"`

doc/release-notes/4.18.1-release-notes.md

@@ -0,0 +1,45 @@
+# Dataverse 4.18.1
+
+This release provides a fix for a regression introduced in 4.18 and implements a few other small changes.  
+
+## Release Highlights
+
+### Proper Validation Messages
+
+When creating or editing dataset metadata, users were not receiving field-level indications about what entries failed validation and were only receiving a message at the top of the page. This fix restores field-level indications.
+
+## Major Use Cases
+
+Use cases in this release include:
+
+- Users will receive the proper messaging when dataset metadata entries are not valid.
+- Users can now view the expiration date of an API token and revoke a token on the API Token tab of the account page.
+
+## Complete List of Changes
+
+For the complete list of code changes in this release, see the <a href="https://github.com/IQSS/dataverse/milestone/86?closed=1">4.18.1 milestone</a> in Github.
+
+For help with upgrading, installing, or general questions please post to the <a href="https://groups.google.com/forum/#!forum/dataverse-community">Dataverse Google Group</a> or email [email protected].
+
+## Installation
+
+If this is a new installation, please see our <a href="http://guides.dataverse.org/en/4.18.1/installation/">Installation Guide</a>.
+
+## Upgrade
+
+1. Undeploy the previous version.
+
+- &lt;glassfish install path&gt;/glassfish4/bin/asadmin list-applications
+- &lt;glassfish install path&gt;/glassfish4/bin/asadmin undeploy dataverse
+
+2. Stop glassfish and remove the generated directory, start.
+
+- service glassfish stop
+- remove the generated directory: rm -rf &lt;glassfish install path&gt;glassfish4/glassfish/domains/domain1/generated
+- service glassfish start
+
+3. Deploy this version.
+
+- &lt;glassfish install path&gt;/glassfish4/bin/asadmin deploy &lt;path&gt;dataverse-4.18.1.war
+
+4. Restart glassfish.

doc/release-notes/4.19-release-notes.md

@@ -0,0 +1,8 @@
+# Dataverse 4.19
+
+Update Geospatial Metadata Block
+
+This update adds commas separating the values entered into Geographic Coverage.
+
+- `wget https://github.com/IQSS/dataverse/releases/download/v4.19/geospatial.tsv`
+- `curl http://localhost:8080/api/admin/datasetfield/load -X POST --data-binary @geospatial.tsv -H "Content-type: text/tab-separated-values"`

doc/release-notes/4714-binder.md

@@ -0,0 +1 @@
+https://mybinder.org now supports spinning up Jupyter Notebooks from Dataverse DOIs.

doc/release-notes/6300-6396-search-api.md

@@ -0,0 +1,13 @@
+## Major Use Cases
+
+Newly-supported use cases in this release include:
+
+- Search API users will see additional fields in the JSON output #6300 #6396
+
+## Notes for Tool Developers and Integrators
+
+### Search API
+
+The boolean parameter `query_entities` has been removed from the Search API.
+
+The former "true" behavior of "whether entities are queried via direct database calls (for developer use)" is now always true.

doc/release-notes/6426-reexport-all

@@ -0,0 +1,14 @@
+## Notes for Dataverse Installation Administrators
+
+### Run ReExportall
+
+We made changes to the JSON Export in this release (#6246). If you'd like these changes to reflected in your JSON exports, you should run ReExportall as part of the upgrade process. We've included this in the detailed instructions below.
+
+
+
+
+
+
+xxx. (Optional) Run ReExportall to update JSON Exports  
+
+   <http://guides.dataverse.org/en/4.19/admin/metadataexport.html?highlight=export#batch-exports-through-the-api>

doc/release-notes/6432-basic-oidc-support.md

@@ -0,0 +1,16 @@
+# Basic OpenID Connect support
+Working on epic #5974 brought us refactored code for our OAuth 2.0 based login options.
+This has now been extended to provide basic support for any OpenID Connect compliant
+authentication provider.
+
+While with our OAuth 2.0 login options you had to implement support for every provider
+by pull request, OpenID Connect provides a standardized way for authentication, user 
+details and more. You are able to use any provider just by loading a configuration file,
+without touching the codebase.
+
+While the usual prominent providers like Google et al feature OIDC support, there are
+plenty of options to easily attach your current user storage to a custom made provider,
+using enterprise grade software. See documentation for more details.
+
+This is to be extended with support for attribute mapping, group syncing and more in
+future versions of the code.  

doc/release-notes/dataverse-security-10-solr-vulnerability.md

@@ -0,0 +1,39 @@
+# Security vulnerability in Solr 
+
+A serious security issue has recently been identified in multiple
+versions of Solr search engine, including v.7.3 that Dataverse is
+currently using. Follow the instructions below to verify that your
+installation is safe from a potential attack. You can also consult the
+following link for a detailed description of the issue:
+<A HREF="https://github.com/veracode-research/solr-injection#7-cve-2019-xxxx-rce-via-velocity-template-by-_s00py">RCE in Solr via Velocity Template</A>.
+
+The vulnerability allows an intruder to execute arbitrary code on
+the system running Solr. Fortunately, it can only be exploited if Solr
+API access point is open to direct access from public networks (aka,
+"the outside world"), which is NOT needed in a Dataverse installation.
+
+We have always recommended having Solr (port 8983) firewalled off from
+public access in our installation guides. But we recommend that you
+double-check your firewall settings and verify that the port is not
+accessible from outside networks. The simplest quick test is to try
+the following URL in your browser:
+
+      `http://<your Solr server address>:8983`
+
+and confirm that you get "access denied" or that it times out, etc. 
+
+In most cases, when Solr runs on the same server as the Dataverse web
+application, you will only want the port accessible from localhost. We
+also recommend that you add the following arguments to the Solr
+startup command: `-j jetty.host=127.0.0.1`. This will make Solr accept
+connections from localhost only; adding redundancy, in case of the
+firewall failure.
+
+In a case where Solr needs to run on a different host, make sure that
+the firewall limits access to the port only to the Dataverse web
+host(s), by specific ip address(es).
+
+We would also like to reiterate that it is simply never a good idea to
+run Solr as root! Running the process as a non-privileged user would
+substantially minimize any potential damage even in the event that the
+instance is compromised.

doc/sphinx-guides/source/_static/installation/files/etc/init.d/solr

@@ -7,7 +7,7 @@
 
 SOLR_DIR="/usr/local/solr/solr-7.3.1"
 SOLR_COMMAND="bin/solr"
-SOLR_ARGS="-m 1g"
+SOLR_ARGS="-m 1g -j jetty.host=127.0.0.1"
 SOLR_USER=solr
 
 case $1 in
@@ -33,4 +33,3 @@ case $1 in
         exit 1
         ;;
 esac
-